Job
Description
As a technology consulting firm specializing in Cloud Data Engineering and Analytics, you will be responsible for designing, building, and maintaining scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). Your key responsibilities will include: - Developing and optimizing Spark/PySpark jobs for performance, cost-efficiency, and reliability - Tuning cluster sizing and autoscaling policies - Implementing data quality, observability, lineage, and monitoring to ensure production SLAs - Collaborating with Data Engineers, Data Scientists, and Architects to define schemas, partitioning strategies, and data models - Building CI/CD and release automation for Databricks assets - Managing Git-based source control and code review processes - Providing production support by troubleshooting job failures, performing root-cause analysis, and producing improvement plans In order to excel in this role, you must have strong skills in Databricks, Delta Live Tables (DLT), Apache Spark, PySpark, Delta Lake, and SQL. Additionally, experience with Apache Airflow, Azure Data Factory, and Git/CI-CD tooling (such as Azure DevOps, GitHub Actions, Jenkins) would be preferred. Qualifications required for this position include a minimum of 5 years of professional data engineering experience with hands-on Databricks development. A strong understanding of cloud data platforms (Azure/AWS/GCP) and best practices for data governance, security, and cost optimization is essential. A degree in Computer Science, Engineering, or equivalent practical experience is desirable. As a technology consulting firm specializing in Cloud Data Engineering and Analytics, you will be responsible for designing, building, and maintaining scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). Your key responsibilities will include: - Developing and optimizing Spark/PySpark jobs for performance, cost-efficiency, and reliability - Tuning cluster sizing and autoscaling policies - Implementing data quality, observability, lineage, and monitoring to ensure production SLAs - Collaborating with Data Engineers, Data Scientists, and Architects to define schemas, partitioning strategies, and data models - Building CI/CD and release automation for Databricks assets - Managing Git-based source control and code review processes - Providing production support by troubleshooting job failures, performing root-cause analysis, and producing improvement plans In order to excel in this role, you must have strong skills in Databricks, Delta Live Tables (DLT), Apache Spark, PySpark, Delta Lake, and SQL. Additionally, experience with Apache Airflow, Azure Data Factory, and Git/CI-CD tooling (such as Azure DevOps, GitHub Actions, Jenkins) would be preferred. Qualifications required for this position include a minimum of 5 years of professional data engineering experience with hands-on Databricks development. A strong understanding of cloud data platforms (Azure/AWS/GCP) and best practices for data governance, security, and cost optimization is essential. A degree in Computer Science, Engineering, or equivalent practical experience is desirable.
