Senior Data Engineer

Key Skills:

Roles and Responsibilities:

• Architect, build, and optimize scalable ETL pipelines using AWS Glue, Lambda, and EMR.
• Implement and maintain cloud-native data infrastructure leveraging AWS services (S3, Redshift, EMR).
• Develop and optimize data transformation workflows using Apache Spark for distributed processing.
• Design and implement real-time streaming solutions with AWS Kinesis or Apache Kafka.
• Orchestrate complex ETL workflows using Apache Airflow.
• Create and optimize data models and queries in Amazon Redshift to support analytics and reporting.
• Containerize data engineering applications using Docker for consistent deployments.
• Build and maintain automated monitoring and alerting systems for pipeline reliability.
• Collaborate with data scientists, analysts, and developers to design and deliver efficient data models.
• Ensure adherence to data governance, security, and compliance policies when handling sensitive data.

Skills Required:

• Strong expertise in ETL design and development, including batch and real-time data pipelines.
• Hands-on experience with AWS services (S3, Redshift, Glue, EMR, Lambda, Kinesis).
• Proficiency with Apache Airflow for scheduling and orchestration of data workflows.
• Solid understanding of data modeling, schema design, and query optimization in data warehouses like Redshift.
• Experience with Apache Spark or other distributed data processing frameworks.
• Familiarity with containerization (Docker) for deploying data engineering applications.
• Knowledge of streaming technologies such as AWS Kinesis or Apache Kafka.
• Strong skills in Python/SQL for data manipulation and automation.
• Experience in data governance, security, and compliance for large-scale systems.
• Excellent problem-solving, debugging, and collaboration skills.

Education: