Data Engineer - Python/ETL

About The Role

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Python, PySpark, and SQL.
• Build and optimize scalable data pipelines leveraging AWS services (Glue, Lambda, S3, Athena, Step Functions).
• Implement and manage Data Warehousing solutions with strong knowledge of SCD (Type 1, Type 2) and Medallion Architecture.
• Develop efficient data models and ensure partitioning, indexing, and performance optimization in big data environments.
• Ensure high standards of data quality, governance, and security across pipelines and platforms.
• Collaborate with data scientists, analysts, and business stakeholders to translate business requirements into technical solutions.
• Monitor and troubleshoot production pipelines to ensure reliability and scalability.
• Contribute to automation, process improvements, and documentation for data engineering workflows.
  

Required Skillsets

• 5-8 years of proven experience in Data Engineering.
• Strong proficiency in Python, PySpark, and SQL for data processing and Solid understanding of ETL/ELT design principles and experience with AWS services (Glue, Lambda, S3, Athena, Step Functions).
• Hands-on experience with Data Warehousing concepts, SCD (Type 1, Type 2), and Medallion Architecture.
• Expertise in data modeling, partitioning strategies, and query performance tuning.
• Strong problem-solving and debugging skills in big data environments.
• Excellent communication skills to explain technical concepts to non-technical stakeholders.
  

Nice-to-Have Skills

• Experience with DBT for data transformation and testing.
• Exposure to Databricks and the Lakehouse architecture.
• Familiarity with CI/CD for data pipelines and infrastructure-as-code (Terraform, Knowledge of data security, compliance, and governance best practices.