Data engineer lead

Very Urgent position for Data Engineer Lead

Job Title: Data Engineer Lead

Experience: 5-10 Years

 Budget :10-14LPA(based on experience)

Location: Pune -hybrid

Notice Period: Immediate to 15days

Mandatory skills: Python,GCP,Spark,SQL(EXPERT)

Tech Stack Table

Skills

Experience

Rating out of 10

Python

GCP

Spark

SQL-Expert

Lead

Responsibilities

Lead and mentor a team of data engineers,

Design, develop, and architect scalable ETL processes to extract, transform, and load petabytes of data from on-premises SQL Server to GCP Cloud SQL PostgreSQL.

Oversee the comprehensive analysis of existing SQL Server schemas, data types, stored procedures, and complex data models, defining strategies for their optimal conversion and refactoring for PostgreSQL.

Establish and enforce rigorous data validation, quality, and integrity frameworks throughout the migration lifecycle, ensuring accuracy and consistency.

Collaborate strategically with Database Administrators, application architects, business stakeholders, and security teams to define migration scope, requirements, and cutover plans.

Lead the development and maintenance of advanced scripts (primarily Python) for automating large-scale migration tasks, complex data transformations, and reconciliation processes.

Proactively identify, troubleshoot, and lead the resolution of complex data discrepancies, performance bottlenecks, and technical challenges during migration.

Define and maintain comprehensive documentation standards for migration strategies, data mapping, transformation rules, and post-migration validation procedures.

Ensure data governance, security, and compliance standards are meticulously applied throughout the migration process, including data encryption and access controls within GCP.

Implement Schema conversion or custom schema mapping strategy for SQL Server to PostgreSQL shift

Refactor and translate complex stored procedures and T-SQL logic to PostgreSQL-compatible constructs while preserving functional equivalence.

Develop and execute comprehensive data reconciliation strategies to ensure consistency and parity between legacy and migrated datasets post-cutover.

Design fallback procedures and lead post-migration verification and support to ensure business continuity.

Ensuring metadata cataloging and data lineage tracking using GCP-native or integrated tools.

Must-Have Skills

Google Cloud Platform (GCP).

elational database architecture, advanced schema design, data modeling, and performance tuning.

T-SQL (SQL Server) and PostgreSQL.

Hands-on experience with data migration processes, including moving datasets from on-premises databases to cloud storage solutions.

Proficiency in designing, implementing, and optimizing complex ETL/ELT pipelines for high-volume data movement, leveraging tools and custom scripting.

Strong knowledge of GCP services: Cloud SQL, Dataflow, Pub/Sub, Cloud Storage, Dataproc, Cloud Composer, Cloud Functions, and Bigquery.

Solid understanding of data governance, security, and compliance practices in the cloud, including the management of sensitive data during migration.

Strong programming skills in Python or Java for building data pipelines and automating processes.

Experience with real-time data processing using Pub/Sub, Dataflow, or similar GCP services.

Experience with CI/CD practices and tools like Jenkins, GitLab, or Cloud Build for automating the data engineering pipeline.

Knowledge of data modeling and best practices for structuring cloud data storage for optimal query performance and analytics in GCP.

Familiarity with observability and monitoring tools in GCP (e.g., Stackdriver, Prometheus) for real-time data pipeline visibility and alerting.

Good-to-Have Skills

Direct experience with GCP Database Migration Service, Storage Transfer Service, or similar cloud-native migration tools.

Familiarity with data orchestration using tools like Cloud Composer (based on Apache Airflow) for managing workflows.

Experience with containerization tools like Docker and Kubernetes for deploying data pipelines in a scalable manner.

Exposure to DataOps tools and methodologies for managing data workflows.

Experience with machine learning platforms like AI Platform in GCP to integrate with data pipelines.

Familiarity with data lake architecture and the integration of BigQuery with Google Cloud Storage or Dataproc.

Kindly share a profile only in this tracker format ,attach the tracker to the body of the mail.

Without this tracker format profile will not be considered.

Sl. No

Date

Position

Names of the Candidate

Mobile Number

Email id

Total Experience

Relevant Experience

CUrrent CTC

Expected CTC

Notice Period / On Paper

Current Organisation

Current Location

Address with Pin code

Reason of leaving

DOB

Offer in hand

VENDOR NAME

Comments