Data Engineer

We are looking for an experienced Data Engineer to drive the development of scalable, secure, and high-performance data solutions. This role requires deep technical expertise in Python, Apache Spark, Delta Lake, and orchestration tools like Databricks Workflows or Azure Data Factory. The ideal candidate will also bring a strong understanding of data governance, metadata management, and regulatory compliance in the insurance and financial services domains. Proficient in developing Python Applications, Spark-based workflows, leveraging Delta Lake, and orchestrating jobs using Databricks Workflows or Azure Data Factory. Good to understand retention metadata, business rules, and data governance policies into reusable pipelines. Strong understanding of data privacy, security, and regulatory needs in insurance and financial domains.

Key Responsibilities:

• Develop robust data pipelines and ETL workflows using Python and Apache Spark. 
• Orchestrate complex workflows using Databricks Workflows or Azure Data Factory. 
• Translate business rules, retention metadata, and data governance policies into reusable, modular, and scalable pipeline components. 
• Ensure adherence to data privacy, security, and compliance standards (e.g., GDPR, HIPAA, etc.). 
• Collaborate with cross-functional teams including data architects, analysts, and business stakeholders to align data solutions with business goals.

Required Skills & Qualifications:

• 4 to 7 years of experience in data engineering 
• Expert-level proficiency in Python, Apache Spark, and Delta Lake. 
• Strong experience with Databricks Workflows and/or Azure Data Factory. 
• Deep understanding of data governance, metadata management, and business rule integration. 
• Strong communication skills. 
• Experience with cloud platforms such as Azure. 

Preferred Qualifications:

• Experience with CI/CD pipelines and DevOps practices in data engineering. 
• Familiarity with data cataloging and data quality tools. 
• Certifications in Azure Data Engineering or related technologies. 
• Exposure to enterprise data architecture and modern data stack tools.