Senior Databricks Engineer

Job Description

Title: Senior Databricks Engineer Location: Baner, Pune Employment Type : Full-Time Availability : Immediate Joiners Only Work Mode : office Experience : 7+ Years Job Overview: We are seeking a highly skilled Senior Databricks Engineer to join our growing data engineering team. The ideal candidate will have deep expertise in building and optimizing scalable data pipelines on Databricks using PySpark, Spark SQL, and modern cloud-native tools. This role is crucial to delivering high-performance, production-ready solutions in collaboration with cross-functional teams including data scientists, architects, and business stakeholders. Key Responsibilities: Design, build, and maintain scalable data pipelines on Databricks using PySpark, Spark SQL, and optionally Scala. Work extensively with Databricks components such as Workspace, Jobs, DLT (Delta Live Tables), Repos, and Unity Catalog. Implement robust Delta Lake solutions adhering to Lakehouse and Medallion Architecture best practices. Collaborate closely with data architects, engineers, and business teams to translate requirements into technical solutions. Develop and manage CI/CD pipelines using tools like Azure DevOps, GitHub Actions, etc., for Databricks deployments. Ensure data quality, security, and governance using tools like Unity Catalog or Azure Purview. Utilize Airflow, Azure Data Factory, or Databricks Workflows for data orchestration and pipeline monitoring. Apply strong SQL and data modeling skills to support transformation and warehousing requirements. Communicate effectively with both technical and non-technical stakeholders. Required Skills & Qualifications: Minimum 4 years of hands-on experience in Data Engineering, with a focus on Databricks. Expertise in Databricks Workspace, Jobs, DLT, Repos, and Unity Catalog. Strong programming proficiency in PySpark and Spark SQL. Knowledge of Scala is a plus. Experience with one or more cloud platforms: Azure, AWS, or GCP. In-depth understanding of Delta Lake, Lakehouse architecture, and medallion architecture patterns. Proven ability to implement CI/CD pipelines using DevOps tools. Familiarity with orchestration tools like Apache Airflow, Azure Data Factory, or Databricks Workflows. Strong knowledge of data governance, metadata management, and lineage tracking. Excellent problem-solving, communication, and stakeholder management skills. If you're interested please share me your updated resume on jobs@newscapeconsulting.com contact number- +919702096664