AWS Pyspark Lead

0 - 12 years

0 Lacs

Posted:2 days ago| Platform: Indeed logo

Apply

Work Mode

On-site

Job Type

Full Time

Job Description

Bangalore, Karnataka, India;Hyderabad, Telangana, India;Noida, Uttar Pradesh, India;Pune, Maharashtra, India;Indore, Madhya Pradesh, India


Qualification

:

Location: All locations in India
Experience: 8–12 Years Employment Type: Full-Time Department: Data Engineering / Cloud Data Solutions

Job Summary:

We are looking for a highly skilled and motivated Data Engineer with 8–12 years of hands-on experience to join our growing data engineering team. The ideal candidate will have a strong ground in AWS cloud services, Python programming, and big data technologies like Spark and SQL. You will play a key role in designing, building, and optimizing scalable data pipelines and analytics solutions to support business insights and decision-making.

Key Responsibilities:

  • Design, develop, and maintain robust and scalable data pipelines using AWS services such as Glue, Lambda, Kinesis, Step Functions, Athena, and DynamoDB.
  • Write efficient and reusable Python scripts for data transformation, automation, and orchestration.
  • Work with Spark to process large datasets and optimize data workflows.
  • Develop complex SQL queries for data extraction, validation, and reporting purposes.
  • Collaborate with data scientists, analysts, and cross-functional teams to understand data requirements and deliver end-to-end solutions.
  • Ensure best practices around IAM, S3 data management, and secure data exchange using SNS/SQS.
  • Monitor pipeline performance and troubleshoot data issues to ensure high availability and reliability.
  • Document technical solutions, data flows, and architectural decisions.

Required Skills & Qualifications:

  • 8–12 years of experience in data engineering or related field.
  • Strong hands-on expertise with AWS services, particularly:
    • Glue, Lambda, Kinesis, Step Functions, S3, DynamoDB, Athena, IAM, SNS, SQS.
  • Proficient in Python for scripting and automation.
  • Experience with Apache Spark for big data processing.
  • Strong knowledge of SQL and working with relational and non-relational databases.
  • Solid understanding of data architecture, data integration, and ETL best practices.
  • Ability to work in a fast-paced, collaborative environment and deliver high-quality solutions.

Preferred Qualifications:

  • AWS Certification (e.g., AWS Certified Data Analytics or Solutions Architect) is a plus.
  • Experience with CI/CD pipelines, infrastructure-as-code (Terraform/CloudFormation), or monitoring tools is an advantage.
  • Familiarity with data lake and real-time streaming architecture.

Experience

:

8 to 12 years

Job Reference Number

:

13520

Skills Required

:

AWS, Pyspark, Spark

Role

:

Location: All locations in India
Experience: 8–12 Years Employment Type: Full-Time Department: Data Engineering / Cloud Data Solutions

Job Summary:

We are looking for a highly skilled and motivated Data Engineer with 8–12 years of hands-on experience to join our growing data engineering team. The ideal candidate will have a strong ground in AWS cloud services, Python programming, and big data technologies like Spark and SQL. You will play a key role in designing, building, and optimizing scalable data pipelines and analytics solutions to support business insights and decision-making.

Key Responsibilities:

  • Design, develop, and maintain robust and scalable data pipelines using AWS services such as Glue, Lambda, Kinesis, Step Functions, Athena, and DynamoDB.
  • Write efficient and reusable Python scripts for data transformation, automation, and orchestration.
  • Work with Spark to process large datasets and optimize data workflows.
  • Develop complex SQL queries for data extraction, validation, and reporting purposes.
  • Collaborate with data scientists, analysts, and cross-functional teams to understand data requirements and deliver end-to-end solutions.
  • Ensure best practices around IAM, S3 data management, and secure data exchange using SNS/SQS.
  • Monitor pipeline performance and troubleshoot data issues to ensure high availability and reliability.
  • Document technical solutions, data flows, and architectural decisions.

Required Skills & Qualifications:

  • 8–12 years of experience in data engineering or related field.
  • Strong hands-on expertise with AWS services, particularly:
    • Glue, Lambda, Kinesis, Step Functions, S3, DynamoDB, Athena, IAM, SNS, SQS.
  • Proficient in Python for scripting and automation.
  • Experience with Apache Spark for big data processing.
  • Strong knowledge of SQL and working with relational and non-relational databases.
  • Solid understanding of data architecture, data integration, and ETL best practices.
  • Ability to work in a fast-paced, collaborative environment and deliver high-quality solutions.

Preferred Qualifications:

  • AWS Certification (e.g., AWS Certified Data Analytics or Solutions Architect) is a plus.
  • Experience with CI/CD pipelines, infrastructure-as-code (Terraform/CloudFormation), or monitoring tools is an advantage.
  • Familiarity with data lake and real-time streaming architecture.

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
Impetus Technologies logo
Impetus Technologies

Information Technology and Services

Bengaluru

RecommendedJobs for You

indore, madhya pradesh, india