SeniorData Engineer

Senior Data Engineer / GCP Big Data Intelligence Specialist

About the Project

"GCP for Big Data Intelligence,"

🎯 The Role: Senior Data Engineer / GCP Specialist

We are seeking a highly experienced Senior Data Engineer with deep expertise in the Google Cloud Platform, particularly with BigQuery and the Vertex AI ecosystem. You will be responsible for the end-to-end implementation of the data pipeline, from secure ingestion of 10M+ records to the deployment of predictive machine learning models for marketing intelligence.

🛠️ Key Responsibilities

Phase 1: Data Ingestion and Security

• Design and implement a secure, scalable data transfer strategy for over 10 million records from source systems (Google Drive/Local) into

  Cloud Storage (GCS)

  .
• Configure and manage

  IAM & Admin

  roles to ensure least-privilege access for data teams.
• Utilize the

  Storage Transfer Service

  for reliable, automated data ingestion into GCS.
• Define the schema and execute high-speed data loading into

  BigQuery

  for the raw dataset.

Phase 2: Data Cleaning and Standardization

• Develop complex

  BigQuery SQL

  and use

  Dataform

  to standardize and clean the raw dataset (e.g., field standardization, data quality checks).
• Implement advanced deduplication logic using window functions (ROW_NUMBER() OVER (PARTITION BY...)) to maintain the most accurate customer record.
• Create and maintain a production-ready, partitioned "clean table" (

  nu_skin_clean_contacts

  ) as the single source of truth for AI/ML tasks.

Phase 3: AI Intelligence and Segmentation

• Integrate and leverage

  Vertex AI

  services to generate marketing intelligence features.
• Execute

  Sentiment Analysis

  using the Natural Language API on unstructured text data.
• Develop and train a

  Vertex AI AutoML

  model to predict customer behaviors (e.g., Propensity to Attend a Seminar).
• Use model outputs in

  BigQuery

  to create hyper-segmented, targeted lists (e.g., Hyderabad_Top_50K_Attendees).
• Integrate AI-generated features (e.g., Propensity_Score, Predicted_Value) back into the master contact table.
• Explore and implement advanced AI models for

  Next Best Action

  and

  Lifetime Value (LTV)

  prediction.

Phase 4: Integration and Deployment

• Manage the export of final, segmented lists from

  BigQuery

  to

  GCS

  .
• Design and implement the integration pipeline using tools like

  Zoho Flow

  or native connectors to ensure seamless data flow into downstream CRM/Marketing Automation systems (Zoho Marketing Automation).

✅ Required Qualifications

• 5+ years

  of experience in Data Engineering, with a focus on building high-volume data pipelines.

• 3+ years

  of hands-on experience with

  Google Cloud Platform (GCP)

  services, including:

• BigQuery (Advanced SQL, Partitioning, Clustering)

• Cloud Storage (GCS)

• IAM & Admin

• Demonstrated experience in

  Vertex AI

  for model training, deployment, and using pre-trained APIs (e.g., Natural Language API, AutoML).
• Strong proficiency in scripting/programming (e.g., Python, SQL) for data manipulation and automation.

🌟 Preferred Qualifications

• GCP Professional Data Engineer or Cloud Architect certification.
• Experience with

  Dataform

  or similar tools for managing BigQuery code.
• Experience integrating GCP with marketing automation/CRM platforms (e.g., Zoho, Salesforce, HubSpot).
• Understanding of fundamental machine learning concepts and feature engineering.