Data Engineer - Lead

Job Description

We are looking for 10+years experienced candidates for this role. Responsibilities include: Develop and maintain scalable ETL data pipelines using GCP services. Integrate structured and unstructured data into centralized data platforms. Build data pipelines leveraging BigQuery, Cloud Storage, Dataflow, and Pub/Sub. Collaborate with data scientists and analysts to ensure data availability for analytics. Manage and optimize data warehouse environments, particularly BigQuery. Design and implement real-time data streaming using Cloud Pub/Sub and Dataflow. Implement data governance and ensure high-quality data standards. Use Cloud Functions and Cloud Composer for task automation and orchestration. Develop data transformation scripts using Python, Java, or Scala. Optimize database performance for cost and scalability. Work with cross-functional teams to meet business data needs. Participate in code reviews, troubleshooting, and applying best practices. Use Apache Airflow or Cloud Composer for pipeline orchestration. Implement Terraform or similar IaC tools for managing cloud infrastructure. Use SQL to query and manage large datasets in BigQuery. Leverage Docker and Kubernetes for containerized environments. Collaborate with DevOps teams to build CI/CD pipelines for data workflows. Communicate complex technical concepts to non-technical stakeholders. Identify and resolve performance bottlenecks in data pipelines. Stay updated with emerging GCP technologies and data engineering best practices. Primary Skills : 7+ years of data engineering experience, 4+ years with GCP data services. Proficiency with BigQuery, Cloud Storage, Dataflow, Pub/Sub. Experience with Python, Java, or Scala for data processing. Familiarity with Apache Airflow or Cloud Composer. Strong experience with SQL and data warehousing. Google Cloud certification preferred. Good communication and teamwork skills.