Lead II - Data Engineering

7 - 9 Years
1 Opening
Kochi, Trivandrum

Role Overview

We are seeking a highly skilled Data Engineer with expertise in building scalable data pipelines, ETL/ELT workflows, and cloud-based data solutions. The ideal candidate will be proficient in Python, SQL, PySpark, and modern data warehousing platforms such as Snowflake, BigQuery, Delta Lake, and Lakehouse architectures. This role requires hands-on development, strong problem-solving ability, and end-to-end ownership of data engineering deliverables.

Key Responsibilities

Data Pipeline Development

• Design, build, and optimize ETL/ELT pipelines for ingesting, transforming, and joining large datasets.
• Develop high-quality code (Python, PySpark, SQL) following best practices and coding standards.
• Perform pipeline testing, debugging, and performance tuning to ensure efficiency, scalability, and cost-optimization.

Data Architecture & Modelling

• Create and maintain data schemas, data models (3NF, Star Schema, Wide/Tall projections) and metadata structures.
• Implement and optimize data warehouse solutions on Snowflake, BigQuery, Delta Lake, Data Lake systems.
• Work with data governance tools (e.g., Collibra) and maintain data cataloging standards.

Cloud & DevOps

• Build pipelines using cloud ETL tools (Informatica, AWS Glue, Azure ADF, GCP DataProc/Dataflow, Airflow).
• Understand infrastructure cost impacts, perform cloud resource optimizations, monitor usage, and ensure SLAs.
• Apply DevOps practices including Git-based version control and CI/CD processes.

Project Execution & Stakeholder Management

• Interpret business requirements and translate them into technical design and architecture.
• Support Project Manager in planning, sprint execution, and delivery of assigned modules.
• Conduct product demos, clarify requirements, and collaborate with customer architects.

Quality, Testing & Documentation

• Review and create design documents (HLD, LLD, SAD), test cases, infra costing, STMs, and user documentation.
• Perform RCA, defect prevention, and continuous pipeline reliability improvement.
• Ensure compliance with engineering standards, security policies, and configuration management guidelines.

Team Leadership & Contribution

• Mentor junior engineers and guide them on best practices and certification paths.
• Contribute reusable components, frameworks, templates, and knowledge-base assets.
• Proactively identify risks, support retention initiatives, and drive team engagement.

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience in Python (Pandas, PySpark), SQL optimization, and data manipulation.
• Expertise in ETL tools: Informatica, Airflow, Glue, DataProc, ADF, dbt (nice-to-have).
• Proficiency in Snowflake, including metadata management, RBAC, query profiling, and cost control.
• Knowledge of data modeling, data warehouse concepts, and performance tuning.
• Experience building robust pipelines with fault-tolerant architecture and metadata handling.
• Strong understanding of data security concepts, governance, RBAC, and compliance.

Cloud Skills

• Hands-on experience in AWS / Azure / GCP (data-specific services).
• Familiarity with Lakehouse, Delta Lake, S3/GCS/ADLS, NoSQL stores, and distributed data systems.

Soft Skills

• Strong communication skills for interacting with stakeholders and presenting solutions.
• Experience working in Agile/Scrum environments, participating in ceremonies, and maintaining scrum artefacts.
• Ability to estimate efforts, manage deliverables, and own end-to-end modules.

Success Metrics (KPIs)

• Adherence to engineering standards and project timelines
• Reduced defects and non-compliance issues
• Faster pipeline performance and resource optimization
• Quick turnaround for production issues
• Completion of required certifications and trainings
• High stakeholder satisfaction and sustained team engagement

Preferred Certifications

• Snowflake, AWS/GCP/Azure Data Engineer, Informatica
• Domain-specific certifications (as required)

Snowflake, Airflow, DBT, Data Engineering