Databricks Developer

Role Description

Job Summary

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark to ingest, process, and transform large data sets from diverse sources.
• Implement ETL (Extract, Transform, Load) workflows to support analytics, reporting, and machine learning initiatives.
• Optimize data jobs for performance, scalability, and cost efficiency within the Databricks environment.
• Collaborate with data engineers, data scientists, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
• Develop and maintain notebooks in Databricks using Python, Scala, SQL, or R.
• Ensure data quality by implementing data validation, testing, and monitoring procedures.
• Integrate Databricks with cloud storage services (such as Azure Data Lake, AWS S3, or Google Cloud Storage) and other data sources.
• Automate data workflows and job schedules using Databricks Jobs, Workflows, or orchestration tools such as Apache Airflow or Azure Data Factory.
• Participate in code reviews, support best practices in software development, and contribute to process improvements.
• Monitor, troubleshoot, and resolve issues related to data pipelines and Databricks environment.
• Contribute to the design and implementation of data architectures, including data lakes, data warehouses, and streaming data solutions.
• Document technical designs, data flows, and process workflows for knowledge sharing and compliance requirements.
• Stay current with Databricks platform updates, industry trends, and best practices for big data, analytics, and cloud technologies.
  

Skills

Mandatory Skills : Databricks, Spark, Azure/AWS