Consultant – Cloud Data Engineer – GCP

Key Responsibilities:

• Design, develop, and manage scalable, secure, and efficient data pipelines on Google Cloud Platform (GCP) to process and transform large datasets from multiple sources.
• Implement and optimize GCP data services such as BigQuery, Cloud Storage, Cloud SQL, Dataflow, Dataproc, Pub/Sub, and Cloud Functions.
• Architect and maintain ETL/ELT processes to ensure efficient data ingestion, transformation, and storage using Cloud Data Fusion, Dataflow (Apache Beam), or Composer (Airflow).
• Architect and implement data models on GCP to support efficient data storage and retrieval in BigQuery, Cloud Spanner, or Firestore.
• Collaborate with data architects to design and implement data lakes, data warehouses, and data marts on GCP.
• Build and maintain data integration workflows using tools like Cloud Composer (Apache Airflow), Cloud Functions, or Dataflow.
• Utilize GCP DevOps tools for source control, build automation, release management, and infrastructure as code (IaC) using Terraform or Deployment Manager.
• Implement CI/CD pipelines within Google Cloud Build to automate the build, test, and deployment of data pipelines and infrastructure changes, ensuring rapid and reliable delivery of data solutions.
• Utilize big data technologies such as Dataproc (Apache Spark, Hadoop, etc.) to handle large volumes of data and perform complex analytics tasks.
• Implement real-time data processing solutions using streaming technologies like Pub/Sub and Dataflow (Apache Beam) to enable timely insights and actions.
• Implement data governance policies and security controls within GCP environments to protect sensitive data and ensure compliance with regulatory requirements such as GDPR, HIPAA, or PCI DSS.
• Optimize data pipelines and processing workflows for performance, scalability, and cost-effectiveness on GCP, leveraging BigQuery optimizations, Autoscaling, and cost management techniques.
• Monitor, troubleshoot, and optimize the performance of data pipelines and cloud-based systems to ensure high availability, low latency, and scalability.
• Work closely with cross-functional teams to understand business requirements and translate them into scalable GCP data solutions.
• Ensure data security and compliance with industry best practices, including IAM roles, VPC configurations, encryption, and access controls.
• Conduct regular code reviews, provide feedback, and ensure adherence to best practices and standards.
• Stay up to date with the latest GCP services and cloud technologies to drive innovation within the team.

Job Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 2-3 years of experience as a Cloud Data Engineer or similar role, with a strong focus on GCP cloud services.
• Proficiency in designing and building scalable data architectures using GCP data services such as BigQuery, Cloud Storage, Dataflow, Cloud SQL, and Dataproc.
• Strong experience with ETL/ELT frameworks and tools like Cloud Composer (Apache Airflow), Cloud Data Fusion, or Dataflow (Apache Beam).
• Expertise in SQL, Python, or other programming languages used in data engineering.
• Hands-on experience with data lakes, data warehouses, and data pipeline orchestration on GCP.
• Familiarity with CI/CD pipelines and infrastructure-as-code (IaC) tools like Terraform, Deployment Manager, or Cloud Build.
• Understanding data governance, security, and compliance standards, including encryption, IAM policies, and role-based access control (RBAC).
• Experience in data modeling, data normalization, and performance optimization for cloud-based data solutions.
• GCP Certifications such as Google Cloud Professional Data Engineer or Google Cloud Professional Cloud Architect.
• Experience with Apache Spark or Databricks on GCP.
• Knowledge of machine learning workflows and working with data science teams is a plus.
• Familiarity with DevOps practices and tools such as Docker and Kubernetes.
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
• Strong problem-solving and troubleshooting skills, with a proactive approach to identifying and resolving issues.
• Ability to adapt to a fast-paced, agile environment and manage multiple priorities effectively.