Databricks - Lead

Key Responsibilities

• Architect and Develop Data Pipelines: Design and implement data pipelines that can handle the 4Vs of data (Volume, Variety, Velocity, Veracity) from diverse sources.
• Implement CDC: Utilize Change Data Capture (CDC) to efficiently and reliably synchronize data between source and target systems.
• Build a Medallion Architecture: Design and implement a data platform following the Medallion Architecture (Raw, Refined, Insights) to ensure data quality, manage transformations, and support analytical needs.
• Orchestrate and Manage Workflows: Develop and manage data orchestration processes to automate data ingestion, transformation, and delivery.
• Data Strategy & Business Value: Articulate how data solutions create tangible business value, and collaborate with stakeholders to define data-driven strategies.
• Technical Proficiency: Develop and maintain data solutions using Python and PySpark.
• Modern Data Formats: Work with open data formats such as Parquet and Iceberg.
• Resilient Architecture: Implement resilient architecture patterns to ensure high availability and fault tolerance of data systems.
• Reporting & Analytics: Integrate with various reporting tools to provide business insights. (will be a plus)

Soft Skills

• Communication & Collaboration: Excellent communication skills with the ability to articulate technical concepts and business value to both technical and non-technical audiences.
• Personal Attributes: Curious and inquisitive mindset with a passion for continuous learning and problem-solving.