SP Software logo

Big Data Engineer

SP Software

7 - 12 years

Bengaluru

Posted:4 hours ago

Job Description

Mandatory Skills

PySpark, Scala, Apache Spark, AWS, Amazon EMR, AWS Glue, AWS Step Functions, AWS Lambda, ETL Development and Big Data Pipeline Development.

Role & responsibilities

We are looking for an experienced Senior Big Data Engineer with strong expertise in PySpark, Scala, Apache Spark and AWS cloud services. The candidate will be responsible for designing, developing, implementing and optimizing scalable big data pipelines and ETL solutions. The role requires hands-on experience with AWS services such as Amazon EMR, AWS Glue, AWS Step Functions and AWS Lambda, along with expertise in data processing, pipeline orchestration, monitoring and performance optimization.

Key Responsibilities

  • Design, develop and maintain scalable big data pipelines using PySpark, Scala and Apache Spark.
  • Build and manage ETL/ELT workflows for processing large volumes of structured and unstructured data.
  • Develop and optimize data processing solutions using Amazon EMR and AWS Glue.
  • Implement workflow orchestration and automation using AWS Step Functions and AWS Lambda.
  • Monitor, troubleshoot and optimize data pipelines to ensure performance, reliability and data quality.
  • Work with Databricks and Snowflake for data processing, integration and analytics.
  • Leverage additional AWS services to build secure, scalable and efficient data engineering solutions.
  • Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.
  • Use Git for source code management, version control and collaborative development.
  • Explore and leverage AI-related technologies and capabilities to improve data engineering workflows.

Required Technical Skills

  • Strong hands-on experience in PySpark, Scala and Apache Spark.
  • Extensive experience with AWS cloud services.
  • Practical experience with Amazon EMR and AWS Glue.
  • Experience in AWS Step Functions and AWS Lambda.
  • Strong knowledge of ETL development and big data pipeline architecture.
  • Experience in data pipeline monitoring, troubleshooting and performance optimization.
  • Working knowledge of Git and version control.

Good-to-Have Skills

  • Experience with Databricks and Snowflake.
  • Knowledge of AI / Generative AI concepts and applications.
  • Exposure to additional AWS services such as S3, Redshift, Athena and CloudWatch.
  • Experience with data quality, workflow automation and distributed data processing.

Preferred Skills

Databricks, Snowflake, AI Knowledge, AWS Services, Data Pipeline Monitoring and Performance Optimization.

Skills Required

ETL Development GIT AWS Big Data SCALA ETL Pyspark Development

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Work Mode

Work from Office

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Big Data Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
SP Software logo
SP Software

RecommendedJobs for You

SP Software logo

Bengaluru

Bristlecone logo

Bengaluru

Athena Consultancy Services logo

Big Data Engineer

Athena Consultancy Services

Pune

Evoke HR logo

Chennai, Bengaluru

B

Big Data Engineer

BRISKWIN IT SOLUTIONS

Chennai

HNM Systems logo

Hyderabad

Ambiquest Hyderabad logo

Big Data Engineer

Ambiquest Hyderabad

Chennai, Bengaluru

dunnhumby logo

Gurugram