Databricks Engineer _ Pan India _ Contract Role

Databricks Engineer _ Pan India

Role & responsibilities

• Develop and maintain a metadata driven generic ETL framework for automating ETL code Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS .
• Insure MO rating engine experience required.
• Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming).
• Develop and maintain robust data pipelines for batch and streaming data using Delta Lake and Spark Structured Streaming.
• Implement data quality checks, validations, and logging mechanisms.
• Optimize pipeline performance, cost, and reliability.
• Collaborate with data analysts, BI, and business teams to deliver fit for purpose datasets.
• Support data modelling efforts (star, snowflake schemas) de norm tables approach and assist with data warehousing initiatives.
• Work with orchestration tools Databricks Workflows to schedule and monitor pipelines.
• Follow best practices for version control, CI/CD, and collaborative development Skills Hands-on experience in ETL/Data Engineering roles.
• Strong expertise in Databricks (Pyspark, SQL, Delta Lake),
• Databricks Data Engineer Certification Preferred
• Experience with Spark optimization, partitioning, caching, and handling large-scale datasets.
• Proficiency in SQL and scripting in Python or Scala.
• Solid understanding of data lakehouse/medallion architectures and modern data platforms.
• Experience working with cloud storage systems like AWS S3 Familiarity with DevOps practices Git, CI/CD, Terraform, etc. Strong debugging, troubleshooting, and performance-tuning skills.