UST logo

Lead I - Data Engineering- Python, Pyspark,lamda, Glue,Airflow, Athena

8 years

0 Lacs

chennai tamil nadu india

Posted:1 month ago| Platform: Linkedin logo

Apply

Skills Required

data python pyspark airflow pipeline coding testing etl informatica aws databricks apache terraform dynamodb sqs design storage processing code documentation query tuning software engineering optimization certifications development communication agile sql

Work Mode

On-site

Job Type

Full Time

Job Description

Role Description

Job Title:

Data Engineer

Hiring Location:

[Insert Location]

Experience Range

6 – 8 years

Role Proficiency

This role requires expertise in

data pipeline development

, including coding and testing data pipelines for ingesting, wrangling, transforming, and joining data from various sources. The ideal candidate will be skilled in

ETL tools

such as

Informatica, AWS Glue, Databricks, and DataProc

, with strong coding skills in

Python, PySpark, and SQL

.Candidates should have a

deep understanding of data warehousing

solutions like

Snowflake, BigQuery, Lakehouse, and Delta Lake

, and be capable of optimizing data solutions for cost and performance.

Must-Have Skills

  • AWS Lambda
  • AWS Glue
  • PySpark
  • Python
  • Apache Airflow
  • AWS Athena
  • Terraform (basic proficiency)

Good-to-Have Skills

  • AWS Step Functions
  • AWS DynamoDB
  • AWS ECS / EKS
  • AWS OpenSearch
  • AWS Kinesis
  • AWS SNS / SQS
  • AWS Lake Formation

Key Outcomes

Role Outcomes & Responsibilities:

  • Design, develop, and optimize scalable, high-performance data pipelines.
  • Ensure efficient, reliable, and cost-effective data storage and processing.
  • Maintain high standards for code quality, testing, and documentation.
  • Collaborate effectively with cross-functional teams to deliver end-to-end data solutions.

Responsibilities

  • Design, develop, and maintain data pipelines using AWS Glue, PySpark, and Lambda.
  • Build and manage data workflows using Apache Airflow.
  • Query and optimize large datasets using AWS Athena.
  • Implement infrastructure as code (IaC) using Terraform.
  • Conduct performance tuning for data processing workloads.
  • Participate in design reviews, ensuring alignment with best practices.
  • Collaborate with data scientists, analysts, and software engineers to deliver high-quality solutions.
  • Stay up to date with the latest cloud and data engineering trends.

Performance Measures

  • Adherence to coding and engineering standards.
  • On-time delivery and adherence to SLAs.
  • Reduced post-delivery defects.
  • Pipeline performance improvements (speed, resource optimization).
  • Quick turnaround on production issues.
  • Achievement of relevant technical/domain certifications.

Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
  • 6–8 years of experience as a Data Engineer or in a similar role.
  • Proven experience in data pipeline development and ETL design on AWS Cloud.
  • Strong problem-solving, analytical, and communication skills.
  • Ability to work in fast-paced, agile environments.

Technical Environment

Technologies:

Python, PySpark, SQL, AWS Glue, Lambda, Airflow, Athena, Terraform

Platforms:

AWS Cloud (Glue, Step Functions, DynamoDB, ECS, EKS, OpenSearch, Kinesis)

Data Warehousing:

Snowflake, BigQuery, Lakehouse, Delta Lake

Skills

python, pyspark,lamda, Glue,Airflow, Athena

More Jobs at UST

Architect II - Technical SAP ABAP

Bengaluru

12 - 15 yrs
|

INR 0 - 0 Lacs

Lead I - SAP FICO with PS (Project Systems) and RAR

Bengaluru

5 - 7 yrs
|

INR 0 - 0 Lacs

Lead II - SAP ABAP

Bengaluru

7 - 9 yrs
|

INR 0 - 0 Lacs

Procurement

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Manual & Python Automation

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
UST logo
UST

IT Services and IT Consulting

Aliso Viejo CA

RecommendedJobs for You

UST logo

Lead I - Data Engineering- Python, Pyspark,lamda, Glue,Airflow, Athena

UST

chennai, tamil nadu, india

UST logo

Lead I - Data Engineering- Python, Pyspark,lamda, Glue,Airflow, Athena

UST

chennai, tamil nadu, india