AWS Data Engineer Lead

🚀 Job Summary

AWS Data Engineer Lead (PySpark Developer)

AWS Glue, Amazon EMR, and Lambda

🔑 Key Responsibilities

• Design, develop, and maintain robust data transformation pipelines utilizing

  PySpark

  on AWS Glue or Amazon EMR.
• Implement event-driven data architectures using services like

  AWS Lambda, S3 events, Amazon EventBridge, and AWS Step Functions

  to orchestrate scalable data workflows.
• Integrate and optimize data ingestion and reporting pipelines using

  Amazon Athena

  for query-based transformations and analysis.
• Develop logic to efficiently read, write, and process structured and unstructured data stored in

  DynamoDB, Amazon S3, and Athena

  .
• Proactively monitor, debug, and fine-tune data pipelines for optimal performance, scalability, and cost-efficiency using

  CloudWatch

  and platform-specific logging/metrics (Glue/EMR logs, Athena query metrics).
• Optimize PySpark code and configuration for execution on cloud-based data platforms.
• Collaborate effectively with cross-functional product and engineering teams to translate business and data requirements into robust technical implementations.

🛠️ Required Skills & Experience

• 8-10 years

  of total experience, with a focus on

  Python, PySpark,

  and distributed data processing technologies.

• Strong, hands-on expertise

  with core AWS data services, including

  AWS Glue, AWS Lambda, Amazon EMR, and S3

  .
• Proven experience integrating and working with

  Amazon Athena

  and

  DynamoDB

  for various data use cases.
• Proficiency in building and deploying serverless solutions and workflow orchestration using

  AWS Step Functions, EventBridge, and Lambda

  .
• Solid understanding of common data formats (e.g.,

  Parquet, Avro, JSON

  ) and advanced data transformation logic.
• Familiarity with data lake architecture principles.
• Experience working in the

  US Mountain Time Zone (MT)

  .

✨ Good to Have

• Exposure to

  CI/CD pipelines

  and Infrastructure as Code (e.g.,

  Terraform, AWS CDK

  ).
• Knowledge of data governance and security best practices within an AWS environment.
• Prior experience with on-premises to cloud data migration projects.
• Familiarity with advanced performance tuning techniques for Athena and leveraging

  DynamoDB Streams

  for event-driven ingestion.