Data Engineer - Azure Databricks/PySpark

Description

Key Responsibilities

• Migrate Palantir data pipelines to Databricks Notebooks, leveraging PySpark for complex transformations.
• Replace proprietary Palantir libraries with open source or custom Pyspark implementations
• Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency.
• Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems.
• Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices.
• Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling.
• Understand business and technical requirements, translating them into scalable Databricks solutions.
• Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment.
• Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks.
• Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies.
  

Required Qualifications

• Bachelors degree in computer science, Information Systems, or a related discipline.
• 6+ years of hands-on experience with Databricks, including expertise in PySpark.
• Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO).
• Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying.
• Solid knowledge of Azure Cloud Services, particularly Azure Databricks and Azure Data Lake Storage.
• Familiarity with version control systems (e.g., Git) and CI/CD pipelines for data engineering workflows.
• Excellent analytical and problem-solving skills with a focus on detail-oriented development.
  

Preferred Qualifications

• Proficiency in Palantir and experience in migrating Palantir data pipelines to Databricks.
• Expertise in Airflow integration for workflow orchestration, including designing and managing DAGs.
• Familiarity with advanced Airflow features, such as SLA monitoring and external task dependencies.
• Advanced knowledge of Delta Lake optimizations, such as compaction, Z-ordering, and vacuuming.
• Experience with real-time streaming data pipelines using tools like Kafka or Azure Event Hubs.
• Experience with building, updating, deploying, finetuning ML models
• Certifications such as Databricks Certified Associate Developer for Apache Spark or equivalent.
• Experience in Agile development methodologies.