Senior Data Engineer

Whetstone Cyber Solutions Pvt. Ltd.

8 years

0 Lacs

india

Posted:2 days ago| Platform:

Apply

Skills Required

data azure design analytics adf databricks power engineering pipeline etl pyspark spark storage tuning optimization model reporting support security automate scheduling latency resolve governance architecture code integration reports sql analysis devops github versioning kafka python

Work Mode

Remote

Job Type

Contractual

Job Description

Job Title:

Experience:

Location:

Employment:

We are seeking an experienced Azure Data Engineer to design, build, and optimize scalable data pipelines and analytics solutions on the Microsoft Azure platform. The ideal candidate will have strong hands-on expertise in Azure Data Factory (ADF), Azure Databricks, and Power BI, and will work closely with business stakeholders, data analysts, and data scientists to enable data-driven decision-making.

Responsibilities:

Data Engineering & Pipeline Development
Design, develop, and maintain end-to-end ETL/ELT pipelines using Azure Data Factory (ADF).
Ingest data from multiple sources (databases, APIs, flat files, streaming sources) into Azure data platforms.
Implement data transformations using Azure Databricks (PySpark / Spark SQL).
Build and optimize data solutions using Azure Data Lake Storage (ADLS Gen2) and/or Azure Synapse Analytics.
Ensure data quality, data validation, and schema enforcement.
Implement partitioning, performance tuning, and cost optimization strategies.
Collaborate with BI teams to model data for reporting and analytics.
Support and optimize datasets consumed by Power BI dashboards and reports.
Implement incremental refresh, row-level security (RLS), and performance best practices in Power BI.
Automate workflows and scheduling using ADF triggers and pipelines.
Monitor pipeline performance, failures, and data latency; troubleshoot and resolve issues.
Implement data security, access control, and governance aligned with Azure best practices.
Work closely with business analysts, data scientists, and application teams.
Document data pipelines, architecture, and operational procedures.
Participate in design reviews, code reviews, and continuous improvement initiatives.

Requirements:

Strong experience with Azure Data Factory (ADF) – pipelines, triggers, integration runtimes.
Hands-on expertise in Azure Databricks with PySpark / Spark SQL.
Experience building datasets and reports for Power BI.
Solid understanding of data warehousing concepts (star schema, slowly changing dimensions).
Proficiency in SQL for data analysis and transformations.
Experience with Azure Data Lake Storage (ADLS Gen2).
Experience with Azure Synapse Analytics.
Knowledge of CI/CD pipelines using Azure DevOps or GitHub Actions.
Exposure to Delta Lake, data versioning, and optimization techniques.
Experience with streaming data (Event Hub / Kafka) is a plus.
Familiarity with Python beyond Spark (for utilities, automation).

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

Whetstone Cyber Solutions Pvt. Ltd.

Login to

Please Verify Your Phone or Email

Confirm Action

Senior Data Engineer