Lead Data Engineer, Bangalore

Location:

Lead Data Engineer

Key Responsibilities:

• Design and develop scalable data pipelines

  using Azure Data Factory and Databricks.
• Lead the

  implementation of ETL/ELT workflows

  using PySpark and SparkSQL.
• Architect and optimize

  data lake and data warehouse solutions

  on Azure.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure

  data quality, governance, and security

  across all data platforms.
• Implement

  CI/CD pipelines

  for data engineering workflows.
• Monitor and troubleshoot performance issues in Spark jobs and ADF pipelines.
• Mentor junior data engineers and contribute to best practices and standards.

Mandatory Skills & Experience:

• Databricks (Mandatory):

• Extensive experience in building and managing notebooks, clusters, jobs, and workflows.
• Proficient in

  PySpark

  for distributed data processing.
• Experience with

  Delta Lake

  ,

  Unity Catalog

  , and

  Databricks SQL

  .

• Azure Data Factory (ADF):

• Strong exposure to building and orchestrating pipelines.
• Integration with various Azure services (Blob Storage, Synapse, Key Vault, etc.).

• PySpark:

• Advanced programming skills in PySpark for data transformation and aggregation.
• Performance tuning of Spark jobs.

• SQL & SparkSQL:

• Strong command over writing optimized SQL queries.
• Experience with SparkSQL for querying large datasets.

Preferred Skills:

• Experience with

  Azure Synapse Analytics

  ,

  Azure Functions

  , and

  Azure DevOps

  .
• Familiarity with

  data modeling

  ,

  data warehousing

  , and

  dimensional modeling

  .
• Knowledge of

  data governance frameworks

  and

  metadata management

  .
• Exposure to

  streaming data

  using Spark Structured Streaming or Azure Event Hub.

Qualifications:

• Bachelors or Masters degree in Computer Science, Engineering, or related field.
• 8+ years of experience in data engineering roles.
• Certifications in Azure or Databricks are a plus.

Regards

Manosh Happy

manosh.m@itcinfotech.com