DataBrick

We are seeking an experienced Databricks Developer with 5 to 7 years of expertise in big data engineering and analytics platforms. The ideal candidate will have deep experience in Apache Spark Databricks PySpark and cloud platforms such as Azure or AWS and a strong background in data pipeline development performance optimization and data governance.Key ResponsibilitiesDesign develop and maintain scalable data pipelines using Apache Spark on DatabricksWrite efficient and production ready PySpark or Scala code for data transformation and ETL processesIntegrate data from various structured and unstructured sources into a unified platformImplement Delta Lake and manage data versioning updates and schema evolutionOptimize data processing workflows for performance scalability and cost efficiencyCollaborate with data scientists analysts and business stakeholders to deliver high quality datasetsImplement data quality checks validation routines and logging mechanismsMonitor and debug production jobs using Databricks jobs notebooks and clustersEnsure security privacy and compliance standards are met throughout the data lifecycleProvide guidance and mentorship to junior team members

Required Skills And Qualifications

5 to 7 years of experience in Big Data developmentHands on experience with Databricks including Workflows Notebooks Delta Live Tables Unity CatalogStrong programming skills in PySpark and or ScalaSolid understanding of Delta Lake architectureProficient in SQL for data analysis and transformationExperience with cloud platforms such as Azure Azure Data Lake Data Factory Synapse or AWS S3 Glue RedshiftFamiliarity with CICD for Databricks deployments using GitHub Actions Azure DevOpsKnowledge of data governance cataloguing and security best practicesExperience working in an Agile Scrum environment

Preferred Skills

Experience with Databricks Unity Catalog and Delta Live TablesExposure to machine learning workflows in DatabricksExperience with Apache Airflow Kafka or other orchestration messaging toolsCertifications Databricks Certified Data Engineer Associate Professional Azure AWS certification