Azure Data Engineer (Lead)

8 - 11 years

32 - 40 Lacs

Posted:1 week ago| Platform: Naukri logo

Apply

Work Mode

Work from Office

Job Type

Full Time

Job Description

Key Responsibilities

1. Data Quality Framework Design & Leadership

  • Define and implement enterprise-wide data quality frameworks and governance standards.

  • Databricks (Delta Lake)

  • Design DQ monitoring architecture profiling, lineage integration, and alerting mechanisms.

  • Establish KPIs and DQ scorecards to measure and communicate data trust metrics across domains.

2. Advanced Data Quality Development & Automation

  • Build and optimize complex validation, reconciliation, and anomaly detection workflows using PySpark and Python.

  • Implement rule-based and ML-based DQ checks, leveraging Ataccama workflows and open-source frameworks.

  • Integrate DQ rules into CI/CD and orchestration platforms (Airflow, ADF, or Databricks Workflows).

  • Partner with data engineers to embed DQ checks into ingestion and transformation pipelines.

3. Root Cause Analysis & Continuous Improvement

  • Lead root-cause investigations for recurring DQ issues and drive long-term remediation solutions.

  • Create and enforce best practices for rule versioning, DQ exception handling, and reporting.

  • Own the playbook for DQ incident response and continuous optimization.

4. Stakeholder Management & Governance

  • Act as the primary liaison between business data owners, IT, and governance teams.

  • Translate business DQ requirements into technical implementation strategies.

  • Drive executive-level reporting on DQ KPIs, SLAs, and issue trends.

  • Contribute to metadata management, lineage documentation, and master data alignment.

5. Mentorship & Leadership

  • Guide junior analysts and data engineers in developing robust DQ solutions.

  • Lead cross-functional squads to implement new data quality capabilities or upgrades.

  • Contribute to capability uplift training peers on DQ best practices, tools, and technologies.

Core Technical Skills

Category

Tools / Skills

Data Engineering & Quality

Databricks (Delta Lake), PySpark, SQL, Python

DQ Platforms

Ataccama ONE / Studio (rule authoring, workflow automation, profiling)

Orchestration & CI/CD

Apache Airflow, Azure Data Factory, Databricks Workflows, GitHub Actions

Data Warehouses

Databricks Lakehouse

Cloud & Infrastructure

Azure (preferred), AWS, or GCP; familiarity with Terraform or IaC concepts

Version Control / CI-CD

Git, GitHub Actions, Azure DevOps

Metadata & Governance

Collibra, Alation, Ataccama Catalog, OpenLineage

Monitoring & Observability

Grafana, Datadog, Prometheus for DQ metrics and alerts

Qualifications & Experience

  • Bachelor s or Master s in Computer Science, Information Systems, Statistics, or related field.

  • 9 12 years

  • Proven experience designing and deploying enterprise DQ frameworks and automated checks.

  • Databricks

  • SQL

  • Solid understanding of data models, lineage, reconciliation, and governance frameworks

  • Experience integrating DQ checks into CI/CD pipelines and orchestrated data flows.

Soft Skills & Leadership Attributes

  • Strong analytical thinking and systems-level problem solving.

  • Excellent communication and presentation skills for senior stakeholders.

  • Ability to balance detail orientation with strategic vision.

  • Influencer with a proactive, ownership-driven mindset.

  • Comfortable leading cross-functional teams in fast-paced, cloud-native environments.

Preferred / Nice to Have

  • Experience in financial, manufacturing, or large enterprise data environments.

  • MDM

  • machine learning-driven anomaly detection

  • Certifications: Databricks, Ataccama, or Cloud Data Engineering certifications (Azure/AWS).

Success Indicators

  • Increased DQ rule coverage and automation across key data domains.

  • Reduced manual DQ exceptions and faster remediation cycle times.

  • Measurable improvement in data trust metrics and reporting accuracy.

  • High stakeholder satisfaction with data availability and reliability.

EXPERIENCE

  • 8-11 Years


SKILLS

  • Primary Skill: Data Engineering

  • Sub Skill(s): Data Engineering

  • Additional Skill(s): Python, databricks, SQL, Azure Data Factory


ABOUT THE COMPANY

Infogain is a human-centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune 500 companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience-led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP).

Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Krak w, Noida, Gurgaon, Mumbai, Pune, and Bengaluru.



Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Infogain logo
Infogain

IT Services and IT Consulting

Los Gatos CA

RecommendedJobs for You