Job
Description
As a GCP Data Engineer specializing in Data Migration & Transformation, your role will involve designing and building robust, scalable data pipelines and architectures on GCP, with a focus on BigQuery. Your responsibilities will include migrating and transforming large-scale data systems to GCP while ensuring performance, scalability, and reliability. You will automate data lineage extraction, maintain data integrity across platforms, and collaborate with stakeholders to implement GCP-native and 3rd-party tools for data ingestion and integration. Additionally, you will optimize complex SQL queries in BigQuery for data analysis, operationalize data pipelines using tools like Apache Airflow, DataFlow, and Pub/Sub, and enable machine learning capabilities through well-structured data pipelines. Your participation in Agile processes and technical design discussions will be crucial. Key Responsibilities: - Design and build robust, scalable data pipelines and architectures on GCP, especially BigQuery. - Migrate and transform large-scale data systems and datasets to GCP with a focus on performance, scalability, and reliability. - Automate data lineage extraction and ensure data integrity across multiple systems and platforms. - Collaborate with architects and stakeholders to implement GCP-native and 3rd-party tools for data ingestion, integration, and transformation. - Develop and optimize complex SQL queries in BigQuery for data analysis and transformation. - Operationalize data pipelines using tools like Apache Airflow (Cloud Composer), DataFlow, and Pub/Sub. - Enable machine learning capabilities through well-structured, ML-friendly data pipelines. - Participate in Agile processes and contribute to technical design discussions, code reviews, and documentation. Qualifications Required: - 5+ years of experience in Data Warehousing, Data Engineering, or similar roles. - Minimum 2 years of hands-on experience working with GCP BigQuery. - Proficiency in Python, SQL, Apache Airflow, and GCP services including BigQuery, DataFlow, Cloud Composer, Pub/Sub, and Cloud Functions. - Experience with data pipeline automation, data modeling, and building reusable data products. - Solid understanding of data lineage, metadata integration, and data cataloging (preferably GCP Data Catalog and Informatica EDC). - Proven ability to analyze complex datasets and derive actionable insights. - Demonstrated experience building and deploying analytics platforms on cloud environments (GCP preferred). In this role, you will also benefit from having strong analytical and problem-solving capabilities, exposure to machine learning pipeline architecture, excellent communication skills, familiarity with Agile methodologies, and a self-driven, innovative mindset committed to delivering high-quality solutions. Additionally, experience with documenting complex data engineering systems and developing test plans would be advantageous. As a GCP Data Engineer specializing in Data Migration & Transformation, your role will involve designing and building robust, scalable data pipelines and architectures on GCP, with a focus on BigQuery. Your responsibilities will include migrating and transforming large-scale data systems to GCP while ensuring performance, scalability, and reliability. You will automate data lineage extraction, maintain data integrity across platforms, and collaborate with stakeholders to implement GCP-native and 3rd-party tools for data ingestion and integration. Additionally, you will optimize complex SQL queries in BigQuery for data analysis, operationalize data pipelines using tools like Apache Airflow, DataFlow, and Pub/Sub, and enable machine learning capabilities through well-structured data pipelines. Your participation in Agile processes and technical design discussions will be crucial. Key Responsibilities: - Design and build robust, scalable data pipelines and architectures on GCP, especially BigQuery. - Migrate and transform large-scale data systems and datasets to GCP with a focus on performance, scalability, and reliability. - Automate data lineage extraction and ensure data integrity across multiple systems and platforms. - Collaborate with architects and stakeholders to implement GCP-native and 3rd-party tools for data ingestion, integration, and transformation. - Develop and optimize complex SQL queries in BigQuery for data analysis and transformation. - Operationalize data pipelines using tools like Apache Airflow (Cloud Composer), DataFlow, and Pub/Sub. - Enable machine learning capabilities through well-structured, ML-friendly data pipelines. - Participate in Agile processes and contribute to technical design discussions, code reviews, and documentation. Qualifications Required: - 5+ years of experience in Data Warehousing, Data Engineering, or similar roles. - Minimum 2 years of hands-on experience working with GCP BigQuery. - Proficiency in Python, SQL, Apache Airflow, and GCP services including BigQuery, Data
