Job Summary
Data Engineer
The ideal candidate should have experience in building data pipelines, implementing ETL/ELT processes, transforming and integrating data from multiple sources, and ensuring data quality and reliability. Experience with SQL, Python, data modelling, and cloud-based data platforms is essential.
Position Details
Employment Type:
Full timeWork Mode:
Work from OfficeLocation:
Coimbatore, India
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using
Azure Data Factory (ADF)
and either Databricks or Microsoft Fabric
. - Build and maintain data ingestion pipelines using Azure Data Factory (ADF), integrating data from approved files, APIs, and other source systems. Manage incremental loads, schema changes, failed loads, error handling, and recovery.
- Develop and maintain Bronze, Silver, and Gold data layers using Azure Databricks or Microsoft Fabric, implementing scalable data transformation logic. Experience with dbt for developing modular data models, reusable transformation logic, testing, and documenting dependencies is an added advantage.
- Implement source-to-target mappings, reconciliation, incremental processing, historical data handling, and dimensional models.
- Ensure pipeline data quality through validation rules, completeness and freshness checks, exception handling, reconciliation, and remediation tracking. Implement automated data quality tests using dbt or platform-native capabilities.
- Follow engineering standards and best practices in pipeline development, participate in code and design reviews, and coordinate technical or architectural decisions with the onshore architect.
- Coordinate data platform and Profisee data flows with the MDM engineer and trusted reporting datasets with the BI engineer.
- Maintain technical metadata, transformation documentation, and lineage in coordination with the MDM and Data Governance Engineer.
- Coordinate integration testing, defect resolution, deployment readiness, and production handover; provide evidence for release approval.
- Identify delivery risks early and coordinate priorities with the Project Manager, CRM team, and client stakeholders.
Required Skills
- 5+ years in data engineering and ETL development.
- Strong hands-on SQL, Databricks or Microsoft Fabric.
- Mandatory hands-on experience with Azure Data Factory (ADF) for data ingestion, orchestration, and pipeline development.
- Experience building and maintaining ETL/ELT pipelines and scalable data integration solutions.
- Strong understanding of Data Lakehouse architecture, Medallion architecture (Bronze, Silver, Gold), and data warehousing concepts.
- Experience with source-to-target mapping, data reconciliation, incremental loading, historical data handling, and dimensional modeling.
- Experience delivering at least two production data platform implementations, including source reconciliation and operational handover.
- Working knowledge of MDM integration patterns, data governance controls, and Power BI data requirements.
Good to Have
- Experience with Fivetran or other data ingestion / ELT tools.
- Experience with dbt models, tests, documentation, incremental strategies, Git workflows, and performance tuning.
- Financial services or investment data experience, including customer, account, advisor, or sales data.
- Exposure to Profisee, Dynamics 365 integrations, Python, and Azure DevOps.
- Databricks/Microsoft Fabric or Azure certifications.
- Consulting or systems integrator delivery experience.
Additional Requirements
- Bachelors degree in Computer Science, IT, Engineering, or a related field, or equivalent relevant experience.
- Strong written and verbal communication, structured documentation and ability to take ownership of assigned tasks.
- Availability for agreed overlap with US Central business hours and planned releases is expected.