Data Engineerâ Architect / Lead

Experience Level

Key Responsibilities

• Provide architectural guidance and define standards for data engineering implementations.
• Lead and mentor a team of data engineers, fostering best practices in design, development, and operations.
• Own and drive improvements in performance, scalability, and reliability of data pipelines and platforms.
• Standardize data architecture patterns and reusable frameworks across multiple projects.
• Collaborate with cross-functional stakeholders (Product, Analytics, Business) to align data solutions with organizational goals.
• Design data models, schemas, and dataflows for efficient storage, querying, and analytics.
• Establish and enforce strong data governance practices, ensuring security, compliance, and data quality.
• Work closely with governance teams to implement lineage, cataloging, and access control in compliance with standards.
• Design and optimize ETL pipelines using Databricks, PySpark, and SQL.
• Ensure robust CI/CD practices are implemented for data workflows, leveraging Terraform and modern DevOps practices.
• Leverage GCP services such as Cloud Functions, Cloud Run, BigQuery, Pub/Sub, and Dataflow for building scalable solutions.
• Evaluate and adopt emerging technologies, with exposure to Gen AI and advanced analytics capabilities.
  

Qualifications & Skills

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• Extensive hands-on experience with Databricks (Autoloader, DLT, Delta Lake, CDF) and PySpark.
• Expertise in SQL and advanced query optimization.
• Proficiency in Python for data engineering and automation tasks.
• Strong expertise with GCP services: Cloud Functions, Cloud Run, BigQuery, Pub/Sub, Dataflow, GCS.
• Deep understanding of CI/CD pipelines, infrastructure-as-code (Terraform), and DevOps practices.
• Proven ability to provide architectural guidance and lead technical teams.
• Experience designing data models, schemas, and governance frameworks.
• Knowledge of Gen AI concepts and ability to evaluate practical applications.
• Excellent communication, leadership, and stakeholder management skills.