Data Engineer Databricks Specialist

As a skilled Data Engineer with 3 to 5 years of experience, you will be responsible for designing, building, and maintaining robust ETL/ELT pipelines using Databricks (PySpark/SQL). You will develop and optimize data workflows and pipelines on Delta Lake and Databricks Lakehouse architecture. Your role will involve integrating data from multiple sources to ensure data quality, reliability, and performance. Collaboration with data scientists, analysts, and business stakeholders will be essential to translate requirements into scalable data solutions. Monitoring and troubleshooting production data pipelines for performance and cost optimization will also be part of your responsibilities. Working with DevOps teams for CI/CD integration and automation of Databricks jobs and notebooks will be crucial, along with maintaining metadata, documentation, and versioning for data pipelines and assets. The ideal candidate for this position should possess 3 to 5 years of experience in data engineering or big data development. Strong hands-on experience with Databricks (Notebook, Jobs, Workflows) is required, along with proficiency in PySpark, Spark SQL, and Delta Lake. Experience with Azure or AWS, particularly Azure Data Lake, Blob Storage, Synapse, etc., is preferred. Strong SQL skills for data manipulation and analysis are necessary, as well as familiarity with Git, CI/CD pipelines, and job orchestration tools such as Airflow and Databricks Workflows. An understanding of data modeling, data warehousing, and data governance best practices is also expected.,