Chennai

Posted:2 days ago

Job Description

Job ID
72214
Category
Enterprise Technology
Location
Chennai, India
Work Type
On-site

We are seeking a highly accomplished and visionary Advanced Data Engineer / Technical Anchor to join our team. In this role, you will serve as the technical lead and architectural authority, designing enterprise-grade Cloud Data Solution Architectures leveraging Python, Cloud Dataproc (Spark), BigQuery, Astro (Airflow), and Terraform. You will act as a strategic partner between engineering teams and business leadership, establishing organizational CI/CD standards (Tekton, GitHub) and security scanning controls (Checkmarx, FOSSA, SonarQube). You will govern Cloud Run audit signals and GCS log archival frameworks, and mentor engineering talent to maximize the long-term strategic value of our data platform.

  • Technical Anchor & Solution Architecture: Act as the lead engineer and domain expert, designing enterprise-grade Cloud Data Solution Architectures, implementing advanced design patterns, and guiding technology selection across the platform.
  • Strategic Roadmap Alignment: Partner with business and engineering leaders to align technical roadmaps with organizational goals. Provide strategic advice on "build vs. buy" decisions to maximize long-term business value.
  • IaC & CI/CD Strategy: Establish organizational standards for Infrastructure as Code using Terraform. Standardize automated release pipelines via GitHub and Tekton, embedding mandatory quality, security, and license compliance controls (Checkmarx, FOSSA, SonarQube).
  • High-Performance Distributed Processing: Architect and optimize large-scale real-time and batch data processing systems leveraging BigQuery, Dataflow, Dataproc, Cloud Run, Pub/Sub, and distributed computing frameworks on GCP.
  • Platform Health & Quality Metrics: Establish organizational coding patterns, reusable frameworks, and SLO compliance. Introduce and drive Software Engineering (SWE) metrics, including DORA metrics (lead time, deployment frequency, MTTR, change-failure rate), code coverage, cyclomatic complexity targets, and P99 latency standards.
  • Data Governance & Enterprise Security: Drive the strategy and implementation for enterprise data governance, metadata management, data lineage, and strict GCP security controls (IAM, VPC Service Controls, encryption).
  • Resilience, Observability & Incident Leadership: Establish deployment and observability standards; lead critical incident resolution and Root Cause Analysis (RCA) to implement permanent solutions and address technical debt.
  • Mentorship & Culture: Mentor engineers across teams, conduct high-impact code reviews (Code Guardians), drive cultural change, and represent organizational technical excellence internally and externally (tech talks, blogs, open-source).
  • Education & Experience: Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field, with extensive experience in enterprise data platform architecture, distributed systems, and technical leadership/anchor roles.
  • Data Architecture & Processing: Expert mastery of Python, distributed computing with Spark on Cloud Dataproc, and advanced BigQuery data modeling, query optimization, and cost management.
  • Programming & Systems Design: Expert mastery of Python, Java, or Scala for distributed computing, real-time streaming, and complex data pipeline creation.
  • CI/CD & Security Frameworks: Comprehensive experience integrating Tekton, GitHub, SonarQube, Checkmarx, and FOSSA into automated release gates. Deep knowledge of enterprise GCP security (IAM, Secret Manager) and audit architectures (Cloud Run, Cloud Storage).
  • Data Governance & Security Frameworks: Deep experience building data governance, metadata management, data lineage, and security infrastructure within GCP (VPC SC, IAM, KMS).
  • Engineering Metrics & Quality Control: Proven experience embedding SWE-centric operational metrics (DORA metrics, code coverage, cyclomatic complexity, latency optimization) into platform operations.
  • Executive Leadership & Influence: Exceptional communication skills with a proven track record of influencing cross-team strategies, mentoring engineering talent, driving platform health initiatives, and resolving complex technical blockers.

Skills Required

Python Spark Terraform Business Leadership Ci/cd GitHub Solution Architecture Design Patterns Data Processing Distributed Computing GCP Software Engineering

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

RecommendedJobs for You

Meghaz Software Solutions logo

Data Engineer

Meghaz Software Solutions

Bengaluru

Independent Recruitment Consultant (freelancer) logo

Data Engineer

Independent Recruitment Consultant (freelancer)

India

People Tech Technology logo

Data Engineer

People Tech Technology

Hyderabad

WPP logo

Chennai

Krato Solutions logo

Data Engineer

Krato Solutions

Chennai

Applied Systems logo

Data Engineer

Applied Systems

Bengaluru

HP logo

Bengaluru