Senior Data Engineer

Job Title: Senior Data Engineer Azure Databricks & Azure Stack

Location: [Onsite - Hyderabad]

Experience: 6-8 years

Employment Type: Full-Time

Job Summary:

TechNavitas seeking a highly skilled Senior Data Engineer with 6-8 years of experience in designing and implementing modern data engineering solutions on Azure Cloud. The ideal candidate will have deep expertise in Azure Databricks, Azure Stack, and building data dashboards using Databricks. You will play a critical role in developing scalable, secure, and high-performance data pipelines that power advanced analytics and machine learning workloads.

Key Responsibilities:

• Design and Develop Data Pipelines: Build and optimize robust ETL/ELT workflows using Azure Databricks to process large-scale datasets from diverse sources.

• Azure Stack Integration: Implement and manage data workflows within Azure Stack environments for hybrid cloud scenarios.

• Dashboards & Visualization: Develop interactive dashboards and visualizations in Databricks for business and technical stakeholders.

• Performance Optimization: Tune Spark jobs for performance and cost efficiency, leveraging Delta Lake, Parquet, and advanced caching strategies.

• Data Modeling: Design and maintain logical and physical data models that support structured and unstructured data needs.

• Collaboration: Work closely with data scientists, analysts, and business teams to understand requirements and deliver data solutions that enable insights.

• Security & Compliance: Ensure adherence to enterprise data security, privacy, and governance standards, especially in hybrid Azure environments.

• Automation & CI/CD: Implement CI/CD pipelines for Databricks workflows using Azure DevOps or similar tools.

Required Skills and Experience:

Technical Skills:

• Min 6-8 years of data engineering experience with strong focus on Azure ecosystem.

• Deep expertise in Azure Databricks (PySpark/Scala/SparkSQL) for big data processing.

• Solid understanding of Azure Stack Hub/Edge for hybrid cloud architecture.

• Hands-on experience with Delta Lake, data lakes, and data lakehouse architectures.

• Proficiency in developing dashboards within Databricks SQL and integrating with BI tools like Power BI or Tableau

• Strong knowledge of data modeling, data warehousing (e.g., Synapse Analytics), and ELT/ETL best practices.

• Experience with event-driven architectures and streaming data pipelines using Azure Event Hubs, Kafka, or Databricks Structured Streaming.

• Familiarity with Git, Azure DevOps, and CI/CD automation for data workflows.

Soft Skills:

• Strong problem-solving and analytical thinking.

• Ability to communicate technical concepts effectively to non-technical stakeholders.

• Proven track record of working in Agile/Scrum teams.

Preferred Qualifications:

• Experience working with hybrid or multi-cloud environments (Azure Stack + Azure Public Cloud).

• Knowledge of ML lifecycle and MLOps practices for data pipelines feeding ML models.

• Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect Expert.

Why Join Us?

• Work on cutting-edge data engineering projects across hybrid cloud environments.

• Be part of a dynamic team driving innovation in big data and advanced analytics.

• Competitive compensation and professional growth opportunities.