Chosen Talentz logo

Senior Data Engineer - Databricks / Fabric & Azure Data Factory

Chosen Talentz

6 - 10 years

12 - 18 Lacs

Coimbatore

Posted:3 days ago

Job Description

Job Summary

Data Engineer

The ideal candidate should have experience in building data pipelines, implementing ETL/ELT processes, transforming and integrating data from multiple sources, and ensuring data quality and reliability. Experience with SQL, Python, data modelling, and cloud-based data platforms is essential.

Position Details

  • Employment Type:

    Full time
  • Work Mode:

    Work from Office
  • Location:

    Coimbatore, India

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using

    Azure Data Factory (ADF)

    and either

    Databricks or Microsoft Fabric

    .
  • Build and maintain data ingestion pipelines using Azure Data Factory (ADF), integrating data from approved files, APIs, and other source systems. Manage incremental loads, schema changes, failed loads, error handling, and recovery.
  • Develop and maintain Bronze, Silver, and Gold data layers using Azure Databricks or Microsoft Fabric, implementing scalable data transformation logic. Experience with dbt for developing modular data models, reusable transformation logic, testing, and documenting dependencies is an added advantage.
  • Implement source-to-target mappings, reconciliation, incremental processing, historical data handling, and dimensional models.
  • Ensure pipeline data quality through validation rules, completeness and freshness checks, exception handling, reconciliation, and remediation tracking. Implement automated data quality tests using dbt or platform-native capabilities.
  • Follow engineering standards and best practices in pipeline development, participate in code and design reviews, and coordinate technical or architectural decisions with the onshore architect.
  • Coordinate data platform and Profisee data flows with the MDM engineer and trusted reporting datasets with the BI engineer.
  • Maintain technical metadata, transformation documentation, and lineage in coordination with the MDM and Data Governance Engineer.
  • Coordinate integration testing, defect resolution, deployment readiness, and production handover; provide evidence for release approval.
  • Identify delivery risks early and coordinate priorities with the Project Manager, CRM team, and client stakeholders.

Required Skills

  • 5+ years in data engineering and ETL development.
  • Strong hands-on SQL, Databricks or Microsoft Fabric.
  • Mandatory hands-on experience with Azure Data Factory (ADF) for data ingestion, orchestration, and pipeline development.
  • Experience building and maintaining ETL/ELT pipelines and scalable data integration solutions.
  • Strong understanding of Data Lakehouse architecture, Medallion architecture (Bronze, Silver, Gold), and data warehousing concepts.
  • Experience with source-to-target mapping, data reconciliation, incremental loading, historical data handling, and dimensional modeling.
  • Experience delivering at least two production data platform implementations, including source reconciliation and operational handover.
  • Working knowledge of MDM integration patterns, data governance controls, and Power BI data requirements.

Good to Have

  • Experience with Fivetran or other data ingestion / ELT tools.
  • Experience with dbt models, tests, documentation, incremental strategies, Git workflows, and performance tuning.
  • Financial services or investment data experience, including customer, account, advisor, or sales data.
  • Exposure to Profisee, Dynamics 365 integrations, Python, and Azure DevOps.
  • Databricks/Microsoft Fabric or Azure certifications.
  • Consulting or systems integrator delivery experience.

Additional Requirements

  • Bachelors degree in Computer Science, IT, Engineering, or a related field, or equivalent relevant experience.
  • Strong written and verbal communication, structured documentation and ability to take ownership of assigned tasks.
  • Availability for agreed overlap with US Central business hours and planned releases is expected.

Skills Required

SQL Senior Integration Microsoft Azure Databricks Azure Devops Microsoft Azure Factory

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Work Mode

Work from Office

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Azure DevOps Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Chosen Talentz logo
Chosen Talentz

RecommendedJobs for You