Job
Description
Title: Data Engineer Location: Baner, Pune (Hybrid) 6 to 12 Months contract Responsibilities: Design, develop, and execute robust scalable data pipelines to extract, transform, and load data from on-premises SQL Server databases to GCP Cloud SQL PostgreSQL. Analyze existing SQL Server schemas, data types, and stored procedures, and plan for their conversion and optimization for the PostgreSQL environment. Implement and support data migration strategies from on-premise or legacy systems to cloud environments, primarily GCP. Implement rigorous data validation and quality checks before, during, and after migration to ensure data integrity and consistency. Collaborate closely with Database Administrators, application developers, and business analysts to understand source data structures and target requirements. Develop and maintain scripts (primarily Python or Java) for automating migration tasks, data validation, and post-migration data reconciliation. Identify and resolve data discrepancies, performance bottlenecks, and technical challenges encountered during the migration process. Document migration strategies, data mapping, transformation rules, and post-migration validation procedures. Support cutover activities and ensure minimal downtime during the transition phase. Apply data governance, security, and privacy standards across data assets in the cloud. Refactor SQL Server stored procedures and business logic for implementation in PostgreSQL or application layer where applicable. Leverage schema conversion tools (e.g., pgLoader, custom scripts) to automate and validate schema translation from SQL Server to PostgreSQL. Develop automated data validation and reconciliation scripts to ensure row-level parity and business logic integrity post-migration. Implement robust monitoring, logging, and alerting mechanisms to ensure pipeline reliability and quick failure resolution using GCP-native tools (e.g., Stackdriver/Cloud Monitoring). Must-Have Skills: Expert-level SQL proficiency across T-SQL (SQL Server) and PostgreSQL with strong hands-on experience in data transformation, query optimization, and relational database design. Solid understanding and hands-on experience working with Relational Databases. Strong experience in data engineering, with hands-on work on cloud, preferrably GCP. Experience with data migration techniques and strategies between different relational database platforms. Hands-on experience on any Cloud Data and Monitoring services such as Relational Database services, Data Pipeline services, Logging and monitoring services, - with one of the cloud providers - GCP, AWS or Azure. Experience with Python or Java for building and managing data pipelines with proficiency in data manipulation, scripting, and automation of data processes. Familiarity with ETL/ELT processes and orchestration tools like Cloud Composer (Airflow). Understanding of data modeling and schema design. Strong analytical and problem-solving skills, with a keen eye for data quality and integrity Experience with version control systems like Git. Good-to-Have Skills Exposure to database migration tools or services (e.g., AWS DMS, GCP Database Migration Service, or similar). Experience with real-time data processing using Pub/Sub. Experience with shell scripting. Exposure to CI/CD pipelines for deploying and maintaining data workflows. Familiarity with NoSQL databases and other GCP data services (e.g., Firestore, Bigtable). Show more Show less
