Databricks Lead Engineer

Job Title: Databricks Lead
Location: Remote
Experience: 8+ Years
Apply: sanjai@vysystems.com

About the Role

We are seeking a skilled Databricks Developer with hands-on experience in Apache Spark, PySpark, AWS and data engineering on the Azure Databricks platform. The ideal candidate will design and implement scalable data pipelines, optimize data processing workflows, and contribute to building robust data solutions supporting analytics and business intelligence initiatives.

Key Responsibilities

Design, develop, and maintain data pipelines and ETL workflows using Databricks (PySpark, SQL, Scala, or Python).

Develop scalable and optimized data processing solutions for large datasets.

Work with Azure Data Lake, Delta Lake, and Azure Data Factory (ADF) to build end-to-end data solutions.

Implement data transformations, aggregations, and cleansing within Databricks.

Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical designs.

Optimize Spark jobs for performance and cost efficiency.

Monitor, debug, and troubleshoot Databricks jobs and clusters.

Ensure best practices for data governance, quality, and security.

Contribute to CI/CD pipelines for data workflows and infrastructure automation.

Document processes, workflows, and code to ensure maintainability.