bigQuery engineer

Role & responsibilities

Key Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines and ETL/ELT workflows using GCP services.
• Architect and implement data warehouse solutions using BigQuery, ensuring optimal performance and cost efficiency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
• Optimize and manage large-scale datasets, ensuring data quality, integrity, and security.
• Develop and enforce best practices for data governance, data modeling, and data lifecycle management.
• Implement streaming data solutions using tools like Dataflow, Pub/Sub, and Apache Beam.
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
• Lead the migration of on-premise data systems to GCP, ensuring seamless integration and minimal downtime.
• Mentor and guide junior engineers, fostering a culture of continuous learning and innovation.
• Stay updated with the latest advancements in GCP and data engineering technologies, and recommend improvements to existing systems.

Required Skills and Qualifications:

• 10+ years of experience in data engineering, with a focus on cloud-based solutions.
• Expertise in GCP services, including but not limited to:
• BigQuery
• Dataflow
• Pub/Sub
• Cloud Storage
• Cloud Composer (Airflow)
• Cloud Functions
• Strong proficiency in SQL and experience with BigQuery SQL for complex queries and performance optimization.
• Hands-on experience with Python or Java for building data pipelines and automation.
• Deep understanding of data modeling, data warehousing, and schema design.
• Experience with streaming data processing and tools like Apache Beam or Kafka.
• Familiarity with CI/CD pipelines and version control systems like Git.
• Strong knowledge of data security and compliance standards (e.g., GDPR, HIPAA).
• Proven experience in migrating on-premise data systems to GCP.
• Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.
• Strong communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.