data engineer

Job Description

Role & responsibilities 1. Pipeline Development and Support Design, build, and optimize scalable ETL pipelines on Databricks using PySpark, SQL, and Delta Lake. Work with structured and semi-structured insurance data (policy, claims, actuarial, risk, customer data) from multiple sources. Implement data quality checks, governance, and monitoring across pipelines. Collaborate with data scientists, actuaries, and business stakeholders to translate analytics requirements into data models. Develop and deliver compelling visualizations and dashboards using Databricks SQL, Power BI, Tableau, or similar tools. Monitor and troubleshoot pipeline issues, ensuring data integrity and resolving bottlenecks or failures. Optimize Databricks clusters for performance and cost efficiency. Support ML model deployment pipelines in collaboration with data science teams. Document pipelines, workflows, and architecture following best practices. 2. SQL Write complex SQL queries to extract, transform, and load (ETL) data for reporting, analytics, or downstream applications. Optimize SQL queries for performance, especially when working with large datasets in Snowflake or other relational databases. Create and maintain database schemas, tables, views, and stored procedures to support business requirements. 3. Data Integration Integrate data from diverse sources (e.g., on-premises databases, cloud storage like S3 or Azure Blob, or third-party APIs) into a unified system. Ensure data consistency, quality, and availability by implementing data validation and cleansing processes. 4. Good to have skills Insurance domain experience P&C or L&A domain experience candidate will be preferred Team player / strong communication skills Experience with MLflow, feature stores, and model monitoring. Hands-on experience with data governance tools (e.g., Unity Catalog, Collibra). Familiarity with regulatory and compliance requirements in insurance data. Skills Typically Required 5+ years of experience in data engineering, with at least 2+ years hands-on with Databricks and Spark. Strong proficiency in PySpark, SQL, Delta Lake, and data modeling. Solid understanding of cloud platforms (Azure, AWS, or GCP) and data lake architectures. Experience integrating Databricks with BI tools (Power BI, Tableau, Looker) for business-facing dashboards. Knowledge of insurance data (L&A, P&C) and industry metrics is highly preferred. Familiarity with DevOps tools (Git, CI/CD pipelines) and orchestration tools (Airflow, Databricks Jobs). Strong communication skills to explain technical concepts to business stakeholders