Data Engineer

Job Description:

Experience:

Location:

Designation: Sr. Software / Software Engineer

Job Summary

We are looking for a skilled Data Engineer with strong expertise in AWS cloud technologies and a solid understanding of data pipelines, ETL processes, and data modelling. The ideal candidate will design, build, and optimize scalable data infrastructure to support analytics, machine learning, and business intelligence needs.

Key Roles & Responsibilities

1. Data Pipeline Development

• Design, develop, and maintain end-to-end data pipelines on AWS using services such as Glue, Lambda, Step Functions, EMR, Kinesis, and S3.
• Implement ETL/ELT workflows to extract data from multiple sources, transform it efficiently, and load it into data warehouses or data lakes.
• Ensure pipelines are scalable, reliable, and cost-efficient.

2. Data Modelling & Warehousing

• Design and implement data models (dimensional & relational) optimized for analytics and reporting.
• Develop and maintain data warehouse/lake house solutions using Amazon Redshift, Snowflake, or Athena.
• Work closely with analysts and data scientists to make data easily accessible and usable.

3. Data Quality, Governance & Security

• Implement data validation, quality checks, and monitoring across pipelines.
• Apply best practices for data governance, lineage, and metadata management.
• Ensure data security, encryption, and compliance (IAM policies, KMS, VPC configurations, etc.).

4. Performance Optimization

• Optimize data workflows for speed, cost, and performance.
• Monitor data pipelines and troubleshoot issues related to scalability and latency.
• Tune queries and data storage configurations for better performance.

5. Collaboration & Cross-Functional Work

• Partner with data scientists, analysts, and software engineers to deliver reliable data solutions.
• Support business teams by providing clean, structured, and accessible data for insights and reporting.
• Contribute to code reviews, documentation, and data engineering best practices.

6. Automation & CI/CD

• Automate data pipeline deployments using Infrastructure as Code (IaC) tools like Terraform or CloudFormation.
• Implement CI/CD pipelines for data engineering workflows using tools such as Code Pipeline, Jenkins, or GitHub Actions.

Required Skills & Qualifications

• Bachelors or Master’s degree in Computer Science, Information Technology, or related field.
• 2.5–5 years of hands-on experience as a Data Engineer, preferably with AWS Cloud.
• Proficiency in Python and SQL for data manipulation and scripting.
• Strong experience with AWS data services:
• AWS Glue, S3, Athena, Lambda, Redshift, Kinesis, EMR, and CloudWatch.
• Knowledge of data modelling, ETL design patterns, and data architecture principles.
• Experience with version control (Git) and workflow orchestration tools (Airflow, Step Functions, etc.).

Preferred / Nice-to-Have Skills

• Experience with Snowflake, Databricks, or Apache Spark.
• Familiarity with Docker/Kubernetes for containerized data services.
• Understanding of streaming data and real-time analytics.
• Exposure to MLOps or data observability tools.

Soft Skills

• Strong analytical and problem-solving abilities.
• Excellent communication and collaboration skills.
• Ability to manage multiple priorities in a fast-paced environment.
• Passion for automation, scalability, and building reliable data systems