Azure Databricks Developer

Pune

Posted:4 days ago

Job Description

Area(s) of responsibility
Key Responsibilities
  • Data Engineering Execution
  • Build and maintain ingestion frameworks (ADF / Databricks / Spark)
  • Implement Bronze -> Silver transformations aligned to architecture
  • Architect data quality checks, schema validation, and contract rules
  • Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
  • Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
  • Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
  • Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
  • Familiar with using Databricks Genie.
  • Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling
  • Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic
  • Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models
  • Databricks & PySpark Engineering
  • Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
  • Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
  • Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
  • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

Skills Required

Azure Databricks Data Engineering Spark Data Quality ETL Pipelines Third Party Pyspark Analytics Performance Tuning Ci/cd Microsoft Azure Spark and big data

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now

RecommendedJobs for You