GCP AI DATA Engineer

Position Description:

GCP AI Data Engineer

Skills Required:

TERRAFORM, Big Query, AIRFLOW, POSTGRES, Data Flow, Data Fusion, DataPro, Cloud Storage

Experience Required:

• 3+ years experience

Education Required:

Bachelor's Degree

Additional Information:

• Design and build GCP data driven solutions for enterprise data warehouse and data lakes • Design and implement scalable data architectures on GCP, including data lakes, warehouses, and real-time processing systems. • Utilize ML Services like Vertex AI. • Develop and optimize ETL/ELT pipelines using tools like Python, SQL, and streaming technologies (e.g., Kafka, Apache Airflow). • Architect Historical and Incremental Loads and Refine Architecture on an ongoing basis • Manage and optimize data storage, partitioning, and clustering strategies for high performance and reliability, utilizing services such as BigQuery, Spark, Pub/Sub, and Object Storage. • Collaborate with data scientists, Data engineers, and other stakeholders to understand data needs and deliver solutions aligned with business objectives, security, and data governance. • Automate infrastructure and deployments using Infrastructure as Code (IaC) using tools like Terraform and CI/CD practices (e.g., Tekton) to ensure reliability and scalability. • Operationalize machine learning models by building data infrastructure and managing structured and unstructured data, supporting AI/ML/LLM workflows, including data labeling, classification, and document parsing. • Monitor and troubleshoot data pipelines and systems to identify and resolve issues related to performance, reliability, and cost-effectiveness. • Document data processes, pipeline designs, and architecture, contributing to knowledge transfer and system maintenance.