Data Engineer

Job Title: Senior Data Engineer Azure Databricks

Experience: 7+ Years

Location: Navi Mumbai (On-site)

Job Overview: We are looking for a highly skilled Senior Data Engineer with expertise in Azure Databricks, big data technologies, and cloud-based data architectures. The ideal candidate will have hands-on experience building scalable data pipelines and platforms for high-volume batch and real-time processing, preferably in the manufacturing or energy sector.

Key Responsibilities: • Design and implement robust data pipelines using Azure Databricks, Spark, and other Azure data services. • Collaborate with architects, data scientists, and business teams to understand requirements and translate them into scalable data solutions. • Build and optimize batch and streaming data workflows for ingestion, transformation, and processing. • Ensure data quality, reliability, and governance across the data lifecycle. • Develop and maintain data models and integration frameworks to support reporting and advanced analytics use cases. • Participate in code reviews, deployment automation, and continuous integration processes. • Document data flows, pipeline logic, and key architecture components. • Perform performance tuning and troubleshooting of data jobs and workflows.

Required Skills & Experience: • 7+ years of hands-on experience in data engineering with strong command of Azure Databricks and Apache Spark. • Proficiency in Python or Scala for data pipeline development. • Experience with streaming platforms like Kafka, Event Hub, or Structured Streaming. • Strong knowledge of data modeling, data lake/lakehouse architectures, and data warehouse concepts. • Experience building ETL/ELT workflows using orchestration tools such as ADF, Airflow, or similar. • Familiarity with CI/CD for data engineering and use of tools like Git, Azure DevOps, or Jenkins. • Understanding of data governance, metadata management, and access control mechanisms. • Solid understanding of structured and unstructured data processing at scale.

Preferred Qualifications: • Experience with Delta Lake and lakehouse patterns. • Knowledge of IoT, time-series data, or domain-specific energy/manufacturing data systems. • Exposure to data visualization tools (Power BI, Tableau) and BI integration. • Awareness of regulatory standards and data security practices (GDPR, HIPAA, etc.).