Senior Data Engineer

Responsibilities

• ETL Development: Build and maintain ETL/ELT pipelines using Databricks & Apache Spark (PySpark/Scala).
• Data Warehousing: Design and implement Star Schema, Snowflake Schema, and Data Vault models.
• Databricks Optimization: Optimize Delta Lake storage, caching, and job execution to improve query performance.
• Databricks Optimization: Optimize Delta Lake storage, caching, and job execution to improve query performance.
• Data Ingestion & Processing: Develop scalable ingestion pipelines using Structured Streaming, Auto Loader, and Databricks Workflows.
• Data Governance & Security: Implement Unity Catalog, role-based access controls (RBAC), and compliance standards.
• Performance Tuning: Optimize Spark jobs, partitioning strategies, and query execution plans in Databricks.
• Collaboration: Work closely with Data Scientists, BI Analysts, and DevOps teams to enable seamless data workflows.
  

Requirements

• Education: Bachelor's/Master's degree in Computer Science, Data Engineering, or a related field.
• Programming: Strong expertise in Python (PySpark), SQL, and optionally Scala.
• Databricks Platform: Experience with Databricks SQL, Delta Live Tables (DLT), and MLflow integration.
• ETL & Orchestration: Hands-on experience with Apache Airflow, dbt (Data Build Tool), and Databricks Workflows.
• Data Warehousing: Strong understanding ofDimensional Modeling, Snowflake Schema, and Data Vault methodologies.
• Data Governance: Experience with Unity Catalog, Delta Sharing, and data lineage tracking.
• CI/CD & Version Control: Familiarity with Git, CI/CD pipelines for Databricks (Terraform, Databricks Repos).
• Real-time data streaming experience(Kafka, Event Hubs, or Pulsar).
• Databricks Certification (Data Engineer Associate or Professional)is a plus.
• Experience with monitoring and logging(Datadog, Prometheus, or Databricks Metrics).