Lead Software Engineer - Data

Lead Software Engineer – Data

Responsibilities

• Direct the design and rollout of Big Data solutions organization-wide
• Build and fine-tune distributed data pipelines utilizing Apache Spark
• Produce efficient and scalable Python code to support data engineering efforts
• Develop stream-processing systems using technologies like Apache Storm or Spark-Streaming
• Consolidate data originating from various sources, such as RDBMS, ERP systems, and file-based inputs
• Utilize ETL methodologies and frameworks to guarantee dependable data transformation and delivery
• Optimize Spark job performance to satisfy both business and technical needs
• Lead the team's adoption of native cloud data services on Azure or AWS using Databricks
• Oversee and coach a team of engineers to deliver top-quality data solutions effectively
• Champion Agile practices across the entire software development lifecycle

Requirements

• 8-14 years of hands-on experience with Big Data and associated technologies
• Strong command of distributed computing concepts and Apache Spark
• Skilled in Hadoop v2, MapReduce, HDFS, and Sqoop
• Hands-on experience with messaging platforms like Kafka or RabbitMQ
• Working knowledge of Big Data querying tools including Hive and Impala
• Solid grasp of SQL queries, joins, stored procedures, and relational schema design
• Practical experience with NoSQL databases like HBase, Cassandra, or MongoDB
• Familiarity with ETL methodologies and frameworks
• Hands-on background with native cloud data services on Azure or AWS via Databricks
• Proven ability to lead teams effectively
• Track record of designing and deploying Big Data solutions
• Hands-on experience applying Agile methodology