Posted:2 days ago|
Platform:
On-site
Part Time
ROLES & RESPONSIBILITIES
Key Responsibilities
Lead design and execution of Dataproc ? Databricks PySpark migration roadmap.
Define modernization strategy, including data ingestion, transformation, orchestration, and governance.
Architect scalable Delta Lake and Unity Catalog–based solutions.
Manage and guide teams on code conversion, dependency mapping, and data validation.
Collaborate with platform, infra, and DevOps teams to optimize compute costs and performance.
Own the automation & GenAI acceleration layer, integrating code parsers, lineage tools, and validation utilities.
Conduct performance benchmarking, cost optimization, and platform tuning (Photon, Auto-scaling, Delta Caching).
Mentor senior and mid-level developers, ensuring quality standards, documentation, and delivery timelines.
Technical Skills
Languages: Python, PySpark, SQL
Platforms: Databricks (Jobs, Workflows, Delta Live Tables, Unity Catalog), GCP Dataproc
Data Tools: Hadoop, Hive, Pig, Spark (RDD & DataFrame APIs), Delta Lake
Cloud & Integration: GCS, BigQuery, Pub/Sub, Cloud Composer, Airflow
Automation: GenAI-powered migration tools, custom Python utilities for code conversion
Version Control & DevOps: Git, Terraform, Jenkins, CI/CD pipelines
Other: Performance tuning, cost optimization, and lineage tracking with Unity Catalog
Preferred Experience
10–14 years of data engineering experience with at least 3 years leading Databricks or Spark modernization programs.
Proven success in migration or replatforming projects from Hadoop or Dataproc to Databricks.
Exposure to AI/GenAI in code transformation or data engineering automation.
Strong stakeholder management and technical leadership skills.
EXPERIENCE
SKILLS
ABOUT THE COMPANY
Infogain is a human-centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune 500 companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience-led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP).
Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Kraków, Noida, Gurgaon, Mumbai, Pune, and Bengaluru.
Infogain
Upload Resume
Drag or click to upload
Your data is secure with us, protected by advanced encryption.
Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.
We have sent an OTP to your contact. Please enter it below to verify.
Instantly access job listings, apply easily, and track applications.
Practice Python coding challenges to boost your skills
Start Practicing Python Now
gurgaon
3.575 - 9.425 Lacs P.A.
gurugram, haryana, india
Salary: Not disclosed
bengaluru
32.5 - 40.0 Lacs P.A.
bengaluru
7.0 - 9.0 Lacs P.A.
maharashtra
Salary: Not disclosed
Bengaluru
32.5 - 40.0 Lacs P.A.
Bengaluru
32.5 - 40.0 Lacs P.A.
gurgaon
3.575 - 9.425 Lacs P.A.
gurugram, haryana, india
Salary: Not disclosed
bengaluru
7.0 - 9.0 Lacs P.A.
