AWS Glue Data Engineer

AWS Glue Data Engineer

Mandatory Skills:

We are seeking an experienced Senior Data Engineer with 6+ years of strong expertise in building and optimizing large-scale data pipelines using AWS Glue. The ideal candidate will have deep knowledge of AWS Glue Data Catalogue, AWS Glue Data Quality, and Data Quality Frameworks (AWS DQ), combined with hands-on experience in Python, PySpark, and PyDQ. This role requires a strategic thinker capable of leading data quality initiatives and ensuring robust data governance across complex data ecosystems.

Key Responsibilities:

• Lead the design, development, and optimization of scalable ETL pipelines using AWS Glue and PySpark in a cloud environment.
• Architect and manage AWS Glue Data Catalogue to enable efficient metadata management and data discovery.
• Drive the implementation and enhancement of data quality frameworks leveraging AWS Glue Data Quality and AWS DQ tools.
• Utilize PyDQ extensively to define, monitor, and enforce comprehensive data quality checks and validation rules.
• Mentor junior engineers and collaborate cross-functionally to establish best practices around data quality, governance, and pipeline automation.
• Identify bottlenecks, troubleshoot complex issues, and optimize workflows to ensure high availability and performance.
• Partner with data analytics, data science, and business teams to align data infrastructure with organizational goals.

Required Skills & Experience:

• 6+ years of experience in data engineering, with a focus on cloud-based ETL and data pipeline development.
• Expert-level proficiency with AWS Glue Data Catalogue and AWS Glue Data Quality services.
• Strong hands-on experience implementing Data Quality Frameworks (AWS DQ) in production environments.
• Deep understanding and practical expertise in PyDQ for data validation and quality assurance.
• Advanced programming skills in Python and PySpark for large-scale data processing.
• Proven ability to lead technical initiatives, mentor teams, and drive best practices.
• Experience working in agile environments and with modern data engineering tools and practices.