Data Engineer

Job Summary:

Data Engineer

Job Responsibility:

• Data Engineering & Architecture
• Design and implement scalable and optimized data pipelines on Databricks using Delta Lake, PySpark, and SQL.
• Develop ETL/ELT frameworks for batch and streaming data processing.
• Ensure data quality, governance, and observability using Unity Catalog, Great Expectations, or custom validations.
• Optimize Spark jobs for performance, cost, and scalability.
• Cloud & Infrastructure (Azure/AWS/GCP)
• Deploy and manage Databricks clusters, workspaces, and Jobs.
• Work with Terraform or ARM templates for infrastructure automation.
• Integrate cloud-native services like Azure Data Factory, AWS Glue, or GCP Cloud Composer.
• MLOps & CI/CD Automation
• Implement CI/CD pipelines for Databricks notebooks, workflows, and ML models.
• Work with MLflow for model tracking and lifecycle management.
• Automate data pipelines using Azure DevOps, GitHub Actions, or Jenkins.
• Leadership & Collaboration
• Lead a team of data engineers, ensuring best practices and code quality.
• Collaborate with data scientists, analysts, and business teams to understand requirements.
• Conduct performance reviews, technical mentoring, and upskilling sessions.

Skills:

• Strong hands-on experience in Databricks, Apache Spark (PySpark/Scala), and Delta Lake. Expertise in SQL, ETL/ELT pipelines, and data modelling.
• Experience with Azure, AWS, or GCP cloud platforms. Knowledge of MLOps, MLflow, and CI/CD best practices.
• Experience in workflow orchestration using Databricks Workflows, Airflow, or Prefect. Understanding of cost optimization, cluster tuning, and performance monitoring in Databricks.
• Strong leadership, stakeholder management, and mentoring skills.
• Experience with data lakehouse architectures and Unity Catalog.
• Hands-on with Terraform, Infrastructure-as-Code (IaC), or Kubernetes. 
• Familiarity with data governance, security, and privacy frameworks.