Data Engineer

Our client, is looking for an experienced Data Engineer to design, develop, and maintain scalable data solutions, build efficient data pipelines, manage data integration processes, and ensure data quality and performance.

Experience 3-7 years

RESPONSIBILITIES

• Pipelines & Ingestion: Independently build database, file, and API ingestion into ADLS Gen2 and Azure Databricks; implement incremental loads, change data capture where appropriate, and safe reprocessing.

• Transformations & Modeling: Develop SQL and Python/PySpark transformations and Delta Lake tables; build bronze, silver, and gold layers and analytics-ready models with clear data contracts.

• Reliability & Orchestration: Schedule dependencies through Databricks Jobs or Azure Data Factory; implement validation, monitoring, alerts, retries, and recovery procedures to meet agreed data freshness targets.

• Security & Delivery: Apply Unity Catalog access controls and lineage, use Azure Key Vault or approved secret management, and deliver automated tests and CI/CD releases through Git-based review.

• Performance & Partnership: Tune Spark jobs, queries, and table layout for performance and cost; gather requirements with U.S.-based stakeholders and document mappings, quality rules, and operational runbooks.

• AI-Assisted Engineering: Use approved AI tools to assist with SQL/Python development, testing, troubleshooting, and documentation; validate generated code and outputs and protect client data.

REQUIREMENTS

Experience ranges are indicative; equivalent demonstrated capability will be considered.

• Required - Experience: Typical range: 3-5 years in data engineering, including 2+ years delivering Azure Databricks solutions. Demonstrate independent pipeline delivery from requirements through deployment and production support.

• Required - Core Skills: Strong SQL, Python/PySpark, Delta Lake, ADLS Gen2, ETL/ELT, and data modeling; hands-on incremental ingestion, orchestration, automated testing, and Spark performance tuning.

• Required - Delivery: Practical Unity Catalog, secure Azure access, Git/CI/CD, monitoring, and recovery experience; ability to manage commitments and explain data quality issues to U.S.-based stakeholders.

• Preferred - Additional Platforms: Experience with one or more: Azure Data Factory, Event Hubs, Structured Streaming, Lakeflow pipelines, Terraform, Microsoft Fabric, or Azure Synapse Analytics. Depth across all tools is not expected.

• Preferred - Integration & Certification: Power BI integration or analytics data modeling is a plus. Relevant Databricks data engineering certification is preferred; equivalent practical experience is welcomed.