Senior Data Engineer

Responsibilities

• Data Architecture & Pipeline Design

  : Design, develop, and maintain end-to-end data pipelines and ETL workflows on AWS, ensuring high performance, reliability, and scalability.

• Cloud Data Infrastructure

  : Leverage AWS services such as Amazon EMR, Glue, Redshift, and Kinesis to build and manage robust cloud-based data architectures.

• Real-Time Data Processing

  : Implement real-time data streaming pipelines using Kafka, Flink, and Spark Streaming to ingest and process large volumes of data.

• Data Lake Development

  : Design and manage scalable data lakes using AWS tools and frameworks such as Apache Hudi to ensure efficient storage, retrieval, and management of structured and unstructured data.

• Data Warehousing

  : Optimize and manage Snowflake for data warehousing, ensuring seamless integration with other data systems and high-performance query execution.

• Collaboration

  : Work closely with cross-functional teams, including Data Scientists, Analysts, and DevOps, to ensure alignment of data pipelines with business objectives and requirements.

• Automation & Optimization

  : Automate data processing workflows and improve pipeline performance through tuning and optimization of AWS-based services.

• Data Quality & Governance

  : Establish and maintain data quality and governance practices, ensuring compliance with security standards and best practices.

• Troubleshooting & Support

  : Provide ongoing support for data pipelines, troubleshooting performance issues and resolving data integrity concerns in a timely manner.

Required Qualifications

• Extensive experience (7+ years) in designing and developing scalable data architectures and pipelines, with a strong focus on AWS cloud services.
• Expertise in AWS services such as Amazon EMR, Glue, Redshift, Kinesis, and S3.
• Proficiency with distributed data processing frameworks, including Apache Spark, Apache Flink, and Kafka.
• In-depth knowledge of data lake technologies, specifically Apache Hudi, for large-scale data storage and processing.
• Experience with modern data warehousing solutions, particularly Snowflake, including data loading, querying, and optimization.
• Strong understanding of data modeling, schema design, and data management best practices.
• Experience with automation tools and CI/CD pipelines for data engineering workflows.
• Excellent problem-solving and troubleshooting skills, with a proven ability to optimize and scale data processing workflows.
• Solid understanding of data governance, security, and privacy practices in the cloud.

Preferred Qualifications

• Familiarity with additional AWS services such as SQS, Lambda, and Athena.
• Experience with machine learning platforms and integrating data pipelines for model training and deployment.
• Knowledge of containerization (Docker, Kubernetes) and Infrastructure as Code (Terraform, CloudFormation).
• Relevant certifications in AWS (e.g., AWS Certified Big Data - Specialty, AWS Certified Solutions Architect).