Data Engineer with AWS

Luxoft

8 - 13 years

10 - 15 Lacs

bengaluru

Posted:1 week ago| Platform:

Apply

Skills Required

data engineer rds data warehouse data pipelines data warehousing sql lambda automation postgresql iam redshift spark mysql etl data lake apache airflow programming s3 python airflow vpc elt data quality aws aws services

Work Mode

Work from Office

Job Type

Full Time

Job Description

Project description

We are looking for a skilled Data Engineer with strong experience in AWS and Apache Airflow to join a dynamic data team. You will be responsible for building and maintaining scalable data pipelines, orchestrating workflows, and ensuring data quality and availability across platforms.

Responsibilities

Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient ETL/ELT pipelines for ingesting, transforming, and loading data from various sources into our data lake and data warehouse.
AWS Expertise: Develop and manage data solutions using a wide range of AWS services, including but not limited to:
Storage: S3 (for data lakes), RDS (PostgreSQL, MySQL), Redshift (data warehousing), DynamoDB.
Compute/Processing: Glue, EMR (Spark), Lambda, Athena.
Orchestration/Workflow: AWS Step Functions, AWS Data Pipeline.
Networking Security: VPC, IAM, Security Groups.
Apache Airflow Orchestration: Develop, deploy, and manage complex Directed Acyclic Graphs (DAGs) in Apache Airflow to schedule, monitor, and automate data workflows. Implement best practices for Airflow DAG design, error handling, logging, and monitoring. Optimize Airflow performance and manage Airflow environments (e.g., using Amazon MWAA or self-hosted).

Skills

Must have

Overall, 8+ years of experience as a Data Engineer and over 5 years of relevant experience in AWS.
Programming: Strong proficiency in Python for data manipulation, scripting, and automation.
SQL: Expert-level SQL skills for querying, manipulating, and optimizing data in relational and analytical databases.
AWS Services: Proven hands-on experience designing, building, and operating data solutions on AWS (S3, Glue, Redshift, EMR, Lambda, RDS, Step Functions, etc.).
Apache Airflow: In-depth experience with Apache Airflow for data pipeline orchestration, including DAG development, custom operators, sensors, and managing Airflow environments.
ETL/ELT: Solid understanding and practical experience with ETL/ELT methodologies and tools.

More Jobs at Luxoft

Murex XVA Functional BA

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Fullstack Developer (React, JavaScript, TypeScript)

Pune, Maharashtra, India

3 - 3 yrs

Salary: Not disclosed

Data Engineer

Pune, Maharashtra, India

5 - 8 yrs

Salary: Not disclosed

Power Bi Developer

Gurugram

7 - 10 yrs

INR 15 - 20 Lacs

Credit Risk Consultant

Gurugram

4 - 8 yrs

INR 7 - 11 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Data Engineer Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.