Google Data Engineer

Google Cloud Platform (GCP) Data Engineer

Key Responsibilities:

Data Pipeline Development:

• Design and implement ETL/ELT workflows using GCP services such as Dataflow, BigQuery, and Cloud Composer (Apache Airflow).
• Develop real-time and batch data processing pipelines.

Data Integration and Management:

• Integrate data from various structured and unstructured sources using APIs, Pub/Sub, and Cloud Storage.
• Perform data modeling and database design for optimal performance.

Data Analytics and Optimization:

• Utilize BigQuery for building and optimizing data warehouses.
• Design solutions to improve data quality, reliability, and efficiency.

Cloud Infrastructure:

• Set up and manage GCP environments including IAM, Cloud Storage, and networking.
• Optimize compute resources using services like Cloud Run, Kubernetes, and DataProc.

Collaboration and Reporting:

• Work with cross-functional teams to understand data requirements.
• Provide dashboards and insights using BI tools like Looker or third-party platforms.

Documentation and Best Practices:

• Maintain technical documentation for data workflows.
• Ensure compliance with security and privacy standards.

Required Skills:

Technical Proficiency in GCP Services:

• BigQuery, Dataflow, Pub/Sub, Cloud Storage, DataProc, Cloud Composer, and AI/ML tools.

Programming and Querying:

• Proficient in SQL, Python, and Java/Scala (for Dataflow or Spark).
• Experience with BigQuery SQL optimizations.

Data Processing and Analytics:

• Strong experience in ETL/ELT workflows.
• Knowledge of real-time streaming technologies and tools.

Cloud Infrastructure:

• Experience in setting up GCP environments, networking, IAM policies, and CI/CD pipelines.

Tools and Frameworks:

• Hands-on experience with Airflow, Terraform, and CI/CD tools (e.g., Jenkins, GitHub Actions).
• Familiarity with containerization using Docker and Kubernetes.

Data Management:

• Expertise in data warehousing, data modeling, and database performance tuning.

Soft Skills:

• Strong problem-solving and analytical skills.
• Ability to communicate complex ideas to technical and non-technical stakeholders.
• Team collaboration and adaptability.

Preferred Qualifications:

• GCP Professional Data Engineer certification.
• Experience with AI/ML tools in GCP (e.g., AutoML, Vertex AI).
• Familiarity with data governance and security standards (e.g., GDPR, HIPAA).