Posted:21 hours ago|
Platform:
Hybrid
Full Time
Key Responsibilities
Analyze existing Hadoop, Pig, and Spark scripts from Dataproc and refactor them into Databricks-native PySpark.
Implement data ingestion and transformation pipelines using Delta Lake best practices.
Apply conversion rules and templates for automated code migration and testing.
Conduct data validation between legacy and migrated environments (schema, count, and data-level checks).
Collaborate on developing AI-driven tools for code conversion, dependency extraction, and error remediation.
Ensure best practices for code versioning, error handling, and performance optimization.
Participate in UAT, troubleshooting, and post-migration validation activities.
Technical Skills Core: Python, PySpark, SQLDatabricks: Delta Lake, Unity Catalog, Databricks Workflows, MLflow (basic understanding)GCP: Dataproc, BigQuery, GCS, Composer/Airflow, Cloud Functions
Data Engineering: Hadoop, Hive, Pig, Spark SQLAutomation: Experience with migration utilities or AI-assisted code transformation tools CI/CD: Git, Jenkins, Terraform (preferred)Validation: Data comparison utilities (Delta-to-Delta, DataFrame diffing, schema validation)
Preferred Experience58 years in data engineering or big data application development.Hands-on experience migrating Spark or Hadoop workloads to Databricks.Familiarity with Delta architecture, data quality frameworks, and GCP cloud integration.Exposure to GenAI-based tools for automation or code refactoring is a plus.
Infogain
Upload Resume
Drag or click to upload
Your data is secure with us, protected by advanced encryption.
Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.
We have sent an OTP to your contact. Please enter it below to verify.
Instantly access job listings, apply easily, and track applications.
Practice Python coding challenges to boost your skills
Start Practicing Python Now
hyderabad, chennai, bengaluru
17.0 - 22.5 Lacs P.A.
bengaluru
14.0 - 24.0 Lacs P.A.
bengaluru
24.0 - 48.0 Lacs P.A.
pune, chennai, bengaluru
4.5 - 9.5 Lacs P.A.
noida, pune, bengaluru
20.0 - 32.5 Lacs P.A.
bengaluru
24.0 - 48.0 Lacs P.A.
gurugram, bengaluru
8.5 - 18.5 Lacs P.A.
kochi, mumbai, chennai
10.0 - 20.0 Lacs P.A.
chennai
20.0 - 25.0 Lacs P.A.
chennai
15.0 - 20.0 Lacs P.A.
