Databricks Data Engineer

Position Overview:

The Databricks Data Engineer role is ideal for a skilled and motivated Databricks Data Engineer to design, build, and optimize data pipelines and analytics solutions on the Databricks Lakehouse Platform. This role is ideal for candidates who have experience with big data technologies and a passion for building robust data systems.

Key Responsibilities:

• Develop and maintain scalable data pipelines using Databricks and Apache Spark (primarily with PySpark).
• Build and manage Delta Lake tables and optimize them for analytics and machine learning workloads.
• Integrate data from diverse sources including APIs, cloud storage, databases, and streaming platforms like Kafka.
• Transform raw data into clean, structured formats using best practices in ETL/ELT.
• Implement data quality checks and ensure consistency, completeness, and accuracy of data sets.
• Collaborate with data scientists, analysts, and business stakeholders to meet data requirements for reporting and analytics.
• Automate workflows using Databricks Workflows, job clusters, and scheduling tools.
• Monitor and optimize Spark jobs for performance and cost efficiency.
• Document and maintain data models, pipeline designs, and technical workflows.

Qualifications:

• Education: Bachelor’s degree in Computer Science, Engineering, or a related field.
• Experience:
  • 3+ years of experience in data engineering or software development.
  • 1+ years of hands-on experience working with Databricks and Apache Spark in production environments
• Skills:
  • Proficient in PySpark and SQL.
  • Solid understanding of data warehousing, ETL/ELT design, and big data best practices.
  • Experience with cloud platforms such as Azure, AWS, or GCP.
  • Experience with Delta Lake, Databricks Workflows, and MLflow.
  • Exposure to data governance tools like Unity Catalog or Purview.
  • Experience with real-time data processing (e.g., using Kafka, Structured Streaming).
  • Familiarity with version control systems (e.g., Git) and CI/CD practices.
  • Familiarity with data visualization tools (e.g., Power BI, Tableau) and BI integration
• Certifications :
  • Databricks Certified Data Engineer Associate or Professional (a plus).