Lead Data Engineer – Transfer Solutions

10 - 15 years

10 - 15 Lacs

Posted:6 days ago| Platform: Foundit logo

Apply

Skills Required

Work Mode

On-site

Job Type

Full Time

Job Description

The Role

We are looking for an experienced Data Engineer to design and develop advanced data migration pipelines from traditional OLTP databases (e.g., Oracle) to modern big data platforms such as Cloudera and Databricks. The ideal candidate will possess expertise in technologies such as Python, Java, Spark, and NiFi, along with a proven track record in managing data pipelines for tasks including initial snapshot loading, building Change Data Capture (CDC) pipelines, exception management, reconciliation, data security, and retention. This role also demands proficiency in data modeling, cataloging, taxonomy creation, and ensuring robust data provenance and lineage to support governance and compliance requirements.

Key Responsibilities

  • Design, develop, and optimize data migration pipelines from OLTP databases like Oracle to big data platforms, including Cloudera CDP/CDH and Databricks.
  • Build scalable ETL workflows using tools like Python, Scala, Apache Spark, and Apache NiFi to support initial snapshots, CDC, exception handling, and reconciliation processes.
  • Implement data security measures, such as encryption, access controls, and compliance with data retention policies, across all migration pipelines.
  • Develop and maintain data models, taxonomy structures, and cataloging systems to ensure logical organization and easy accessibility of data.
  • Establish data lineage and provenance to ensure traceability and compliance with governance frameworks.
  • Collaborate with cross-functional teams to understand data migration requirements, ensuring high-quality and timely delivery of solutions.
  • Monitor and troubleshoot data pipelines to ensure performance, scalability, and reliability.
  • Stay updated on emerging technologies in data engineering and big data ecosystems, proposing improvements to existing systems and processes.

Required Skills And Qualifications

  • 10+ years of experience in data engineering, with at least 2 years in a leadership or technical lead role.
  • Proficiency in OLTP databases, particularly Oracle, and data egress techniques.
  • Strong programming skills in Python, Scala and Java.
  • Expertise in Apache Spark, Flink, Kafka and data integration tools like Apache NiFi.
  • Hands-on experience with Cloudera Data Platform CDP/CDH, Apache Ozone
  • Familiarity with cloud-based big data ecosystems such as AWS Databrick, S3, Glue etc
  • Familiarity with patterns such as Medallion, data layers, datalake, datawarehouse, experience in building scalable ETL pipeline, optimizing data workflows, leveraging platforms to integrate transform, and store large datasets.
  • Knowledge of data security best practices, including encryption, data masking, and role-based access control.
  • Exceptional problem-solving and analytical abilities
  • Strong communication and leadership skills, with the ability to navigate ambiguity and collaborate effectively across diverse teams.Optional Awareness on regulatory compliance requirements for data handling and privacy
  • Education: Bachelor's or Master's degree in Computer Science

Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Mastercard logo
Mastercard

IT Services and IT Consulting

Purchase NY

RecommendedJobs for You

Mumbai City, Maharashtra, India

Bengaluru, Karnataka, India