Data Engineer

Role Overview

Databricks

Location:

Key Responsibilities

Technical Design

• Develop, and maintain data pipelines and ETL workflows using Databricks and Airflow.
• Good understanding of Databricks Delta Live tables, and Unity Catalog
• Build frameworks and templates for ETL/ELT, for telemetry pipelines.
• Ensure cloud-native designs aligned with AWS platform best practices.

Governance, Security & Optimization

• Ensure data quality, integrity, and governance across data platforms.
• Optimize SQL queries and data models for performance and scalability.
• Implement monitoring and observability for data workflows.

Cross-Functional Collaboration

• Work closely with architects and leads to understand data requirements and build technical solutions.
• Collaborate with analytics and business teams to deliver actionable insights through BI dashboards and reports

AI/LLM Enablement (Advantage)

• Familiarity with Agentic AI concepts for automation, data quality validation, and metadata enrichment will be an added advantage.

Required Skills & Experience

• 6-9 years of experience in data engineering, with at least 1 years in Databricks.
• Hands-on expertise in:
• Databricks (PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live tables)
• SQL for data modeling and query optimization
• AWS services (S3, Glue, Kinesis, Redshift, API Gateway, SNS, Lambda)
• Airflow for workflow orchestration
• Added advantage – DBT, Splunk, Atlan, Java, API, Power BI
• Strong analytical and problem-solving skills with attention to detail.

Preferred Qualifications

• Knowledge of Agentic AI concepts and frameworks.
• Experience with Python and Spark for data engineering tasks.
• AWS and Databricks certification in data analytics or related specialization.
• Familiarity with big data technologies for data pipelines.