Data Engineer

4 - 6 years

Bengaluru

Posted:1 day ago

Job Description

Job Overview

We are seeking an experienced and results-driven Data Engineer with 4–6 years of relevant experience in designing, developing, and optimizing scalable data solutions using Azure Databricks and related technologies.

The ideal candidate should have strong hands-on expertise in Azure Databricks, PySpark, Apache Spark, SQL, Python, ETL/ELT development, Delta Lake, Azure cloud services, CI/CD, and performance optimization. The role requires practical experience in building reliable, secure, and production-grade data pipelines.


Key Responsibilities

  • Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python, and SQL.
  • Build and optimize batch and large-scale data-processing solutions using Apache Spark.
  • Develop data ingestion and transformation workflows for structured and semi-structured data.
  • Implement reliable data-processing frameworks using Delta Lake capabilities.
  • Perform data cleansing, validation, reconciliation, and quality checks to ensure accuracy and consistency.
  • Optimize Spark jobs, Databricks notebooks, SQL queries, clusters, and data pipelines for performance and cost efficiency.
  • Troubleshoot pipeline failures, data-quality issues, production incidents, and performance bottlenecks.
  • Implement incremental data processing, partitioning, caching, join optimization, and file-management techniques.
  • Develop reusable and configurable data-engineering components.
  • Implement version control and automated deployment processes using CI/CD and DevOps practices.
  • Collaborate with data architects, analysts, application teams, and business stakeholders to translate requirements into technical solutions.
  • Maintain technical documentation covering pipeline architecture, transformation logic, dependencies, and operational procedures.
  • Ensure solutions comply with organizational standards for security, governance, monitoring, and data protection.

Required Technical Skills

  • 4–6 years of relevant experience in data engineering and cloud-based data platforms.
  • Strong hands-on experience with Azure Databricks, PySpark, Apache Spark, Python, and advanced SQL.
  • Proven expertise in developing scalable ETL/ELT pipelines for batch and large-scale data processing.
  • Solid understanding of Delta Lake, including Delta tables, ACID transactions, schema evolution, merge/upsert operations, and table optimization.
  • Experience with Azure data services such as Azure Data Lake Storage and Azure Data Factory.
  • Strong knowledge of Spark and Databricks performance optimization, including partitioning, caching, join strategies, shuffle optimization, file sizing, and cluster configuration.
  • Experience with Git-based version control, CI/CD pipelines, and DevOps practices for automated deployment across environments.
  • Good understanding of data quality, security, monitoring, troubleshooting, and production support.

Skills Required

Azure Databricks Pyspark Apache Spark SQL Python ETL Azure Cloud Services Ci/cd Performance Optimization Data Processing Data Ingestion Data Cleansing

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
Concentrix logo
Concentrix

Business Process Outsourcing

Irving

RecommendedJobs for You

Hanuma Infotech logo

Data Engineer

Hanuma Infotech

India

Tibil Solutions logo

Data Engineer

Tibil Solutions

Bengaluru

CoffeeBeans logo

Data Engineer

CoffeeBeans

Bengaluru

Flexiple logo

Chennai

C

Data Engineer

Creencia Technologies

Bengaluru

D

Pune