Azure Data Engineer

Infogain

5 - 8 years

10 - 11 Lacs

gurugram

Posted:-1 days ago| Platform:

Apply

Skills Required

hive continuous integration python scala pyspark microsoft azure apache pig code versioning tools machine learning data engineering application development sql dataproc gen git automation spark gcp jenkins terraform hadoop bigquery sqoop big data

Work Mode

Work from Office

Job Type

Full Time

Job Description

Analyze existing
Hadoop, Pig, and Spark
scripts from Dataproc and refactor them into Databricks-native PySpark.
Implement
data ingestion and transformation pipelines
using Delta Lake best practices.
Apply
conversion rules and templates
for automated code migration and testing.
Conduct
data validation
between legacy and migrated environments (schema, count, and data-level checks).
Collaborate on developing
AI-driven tools
for code conversion, dependency extraction, and error remediation.
Ensure best practices for code versioning, error handling, and performance optimization.
Participate in UAT, troubleshooting, and post-migration validation activities.

Technical Skills

Core:
Python, PySpark, SQL
Databricks:
Delta Lake, Unity Catalog, Databricks Workflows, MLflow (basic understanding)
GCP:
Dataproc, BigQuery, GCS, Composer/Airflow, Cloud Functions
Data Engineering:
Hadoop, Hive, Pig, Spark SQL
Automation:
Experience with migration utilities or AI-assisted code transformation tools
CI/CD:
Git, Jenkins, Terraform (preferred)
Validation:
Data comparison utilities (Delta-to-Delta, DataFrame diffing, schema validation)

Preferred Experience

5-8 years in data engineering or big data application development.
Hands-on experience migrating Spark or Hadoop workloads to Databricks.
Familiarity with
Delta architecture
,
data quality frameworks
, and
GCP cloud integration
.
Exposure to
GenAI-based tools
for automation or code refactoring is a plus.

EXPERIENCE

6-8 Years

More Jobs at Infogain

Aws Data Engineer

Pune, Delhi NCR, Bengaluru

4 - 6 yrs

INR 13 - 23 Lacs

SAP Developer (Senior)

Bengaluru

5 - 8 yrs

INR 6 - 11 Lacs

Dot NET Fullstack (React) Developer (Senior)

Bengaluru

6 - 8 yrs

INR 13 - 15 Lacs

Application Support Engineer (Senior)

Bengaluru

6 - 8 yrs

INR 8 - 10 Lacs

Dot NET Core Developer (Senior)

Bengaluru

6 - 8 yrs

INR 13 - 15 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.