Lead Data Engineer

Key Responsibilities:

• Design, develop, and maintain scalable and secure

  ETL pipelines

  using

  PySpark

  ,

  Python

  , and

  SQL

  .
• Build and orchestrate data workflows leveraging

  AWS services

  such as

  Glue

  ,

  Lambda

  ,

  Step Functions

  ,

  S3

  ,

  Athena

  , and

  Redshift

  .
• Perform data profiling, cleansing, and transformation to ensure data quality and consistency.
  
• Optimize and tune data pipelines for performance and cost-efficiency.
  
• Work with data architects, analysts, and stakeholders to understand data requirements and translate them into technical solutions.
  
• Develop and maintain technical documentation for all ETL processes and data flows.
  
• Monitor data pipelines and troubleshoot any issues related to performance, failures, or data inconsistencies.
  
• Participate in Agile/Scrum development cycles and contribute to sprint planning and retrospectives.
  

Required Skills & Qualifications:

• 7+ years of hands-on experience in ETL development and data engineering.
  
• Strong programming skills in

  Python

  and

  PySpark

  .
• Expert-level knowledge of

  SQL

  for complex data manipulation and analysis.
• Experience with

  AWS services

  , including but not limited to:

• AWS Glue

• AWS Lambda

• Amazon S3

• Amazon Redshift

• AWS Step Functions

• Amazon Athena

• Solid understanding of data warehousing concepts, data modeling, and schema design.
  
• Experience working with large-scale distributed data systems and real-time/batch processing.
  
• Familiarity with version control tools (e.g., Git) and CI/CD pipelines.
  
• Excellent problem-solving and communication skills