Senior Data Engineer

As a Senior Data Engineer at the company, your role involves designing, developing, and maintaining scalable ETL pipelines and data systems using Python and Google Cloud Platform (GCP). Your responsibilities include: - Designing, developing, testing, and maintaining robust ETL data pipelines using Python. - Working extensively with GCP services such as Dataflow, BigQuery, Cloud Functions, Cloud Composer (Airflow), IAM, Cloud Run, and Google Cloud Storage. - Implementing data ingestion, transformation, and validation logic to maintain data quality and consistency. - Collaborating with cross-functional teams, including data scientists and analysts, to deliver reliable data solutions. - Managing version control through GitHub and contributing to CI/CD pipelines for data projects. - Writing and optimizing complex SQL queries across databases like SQL Server, Oracle, and PostgreSQL. - Creating and maintaining documentation, including data flow diagrams and process documentation. Your technical expertise should include: - Strong proficiency in Python for backend or data engineering projects. - Deep working knowledge of GCP services, especially Dataflow, BigQuery, Cloud Functions, and Cloud Composer. - Experience with data orchestration and workflow tools like Airflow. - Proficiency in Apache Spark and Kafka for data processing and streaming. - Hands-on experience with FastAPI, MongoDB, Redis/Bigtable. - Sound understanding of CI/CD practices and version control systems like GitHub. - Advanced SQL skills and experience with enterprise-grade relational databases. - Solid experience in cloud migration and large-scale data integration. It would be nice to have experience with Snowflake or Databricks for big data analytics and familiarity with GKE, Cloud Run deployments, or Azure Data Factory. As a Senior Data Engineer at the company, your role involves designing, developing, and maintaining scalable ETL pipelines and data systems using Python and Google Cloud Platform (GCP). Your responsibilities include: - Designing, developing, testing, and maintaining robust ETL data pipelines using Python. - Working extensively with GCP services such as Dataflow, BigQuery, Cloud Functions, Cloud Composer (Airflow), IAM, Cloud Run, and Google Cloud Storage. - Implementing data ingestion, transformation, and validation logic to maintain data quality and consistency. - Collaborating with cross-functional teams, including data scientists and analysts, to deliver reliable data solutions. - Managing version control through GitHub and contributing to CI/CD pipelines for data projects. - Writing and optimizing complex SQL queries across databases like SQL Server, Oracle, and PostgreSQL. - Creating and maintaining documentation, including data flow diagrams and process documentation. Your technical expertise should include: - Strong proficiency in Python for backend or data engineering projects. - Deep working knowledge of GCP services, especially Dataflow, BigQuery, Cloud Functions, and Cloud Composer. - Experience with data orchestration and workflow tools like Airflow. - Proficiency in Apache Spark and Kafka for data processing and streaming. - Hands-on experience with FastAPI, MongoDB, Redis/Bigtable. - Sound understanding of CI/CD practices and version control systems like GitHub. - Advanced SQL skills and experience with enterprise-grade relational databases. - Solid experience in cloud migration and large-scale data integration. It would be nice to have experience with Snowflake or Databricks for big data analytics and familiarity with GKE, Cloud Run deployments, or Azure Data Factory.