Azure Databricks(ADB)

We are seeking a highly skilled and experienced Lead Data Engineer to design, build, and optimize scalable data pipelines and architectures using Azure Databricks, Azure Data Factory, and SQL/T-SQL. The ideal candidate will lead data engineering initiatives, mentor junior engineers, and collaborate with cross-functional teams to deliver high-performance data solutions. 

Job Responsibilities

Architecture & Design: Design and implement scalable, secure, and high-performance data architectures on Azure. Define data modeling standards and best practices for structured and unstructured data. Architect data lake and data warehouse solutions using Delta Lake and Synapse. Data Engineering: Develop and maintain ETL/ELT pipelines using Azure Data Factory and Databricks notebooks. Optimize Spark jobs for performance and cost efficiency. Implement data quality checks, lineage, and monitoring frameworks. SQL & T-SQL Development: Write complex SQL and T-SQL queries for data extraction, transformation, and reporting. Optimize stored procedures, views, and indexing strategies for performance. Collaborate with BI teams to support reporting and analytics needs. Leadership & Collaboration Lead a team of data engineers and provide technical guidance and mentorship. Collaborate with data scientists, analysts, and business stakeholders to understand requirements. Drive adoption of best practices in CI/CD, version control, and testing.

Work Experience

8+ years of experience in data engineering or architecture roles. Strong hands-on experience with Azure Databricks, ADF, Delta Lake, and Spark. Proficiency in SQL, T-SQL, and performance tuning. Experience with Azure Synapse, Azure Blob Storage, and Azure Key Vault. Familiarity with Git, CI/CD pipelines. 

Knowledge Technical

Strong hands-on experience with Azure Databricks, ADF, Delta Lake, and Spark. Proficiency in SQL, T-SQL, and performance tuning. Experience with Azure Synapse, Azure Blob Storage, and Azure Key Vault. Familiarity with Git, CI/CD pipelines. 

Knowledge Competency

Azure certifications (e.g., DP-203, AZ-305). Experience with data governance tools like Purview. Knowledge of Python or Scala for data processing. Experience with real-time data streaming (Kafka, Event Hubs)