Data Engineering Lead

As a Data Engineering Lead, your role will involve architecting, building, and optimizing scalable data pipelines and platforms to support crucial business decisions. Your expertise in PySpark, ETL frameworks, SQL Stored Procedures, and Data Modeling will be key in ensuring performance tuning across large datasets. Key Responsibilities: - Lead the design and development of scalable data pipelines using PySpark and SQL - Develop and maintain Stored Procedures for batch processes and data transformations - Design robust data models (Star, Snowflake, normalized models) for data warehouse and lake environments - Collaborate with BI, Analytics, and Product teams to understand data requirements and ensure data quality - Optimize data workflows for performance, cost, and scalability in distributed systems - Manage data ingestion from structured and semi-structured sources (e.g., APIs, files, databases, streams) - Support CI/CD and DevOps practices for data pipeline deployments - Guide and mentor junior data engineers, conduct code reviews, and assist in technical planning Qualifications Required: - 5+ years of experience in data engineering and data platform development - Proficiency in PySpark / Spark SQL for large-scale distributed data processing - Strong understanding of ETL/ELT architectures, data ingestion, and transformation strategies - Experience in data modeling techniques and tools - Demonstrated track record in performance tuning of queries, jobs, and pipelines - Familiarity with tools like Airflow or equivalent for orchestration and transformations Nice To Have: - Exposure to Data Virtualization platforms (Denodo, Dremio, etc.) - Familiarity with Purview or any Data Governance tools This job is based in Kochi initially, with the possibility of relocation to Bangalore. As a Data Engineering Lead, your role will involve architecting, building, and optimizing scalable data pipelines and platforms to support crucial business decisions. Your expertise in PySpark, ETL frameworks, SQL Stored Procedures, and Data Modeling will be key in ensuring performance tuning across large datasets. Key Responsibilities: - Lead the design and development of scalable data pipelines using PySpark and SQL - Develop and maintain Stored Procedures for batch processes and data transformations - Design robust data models (Star, Snowflake, normalized models) for data warehouse and lake environments - Collaborate with BI, Analytics, and Product teams to understand data requirements and ensure data quality - Optimize data workflows for performance, cost, and scalability in distributed systems - Manage data ingestion from structured and semi-structured sources (e.g., APIs, files, databases, streams) - Support CI/CD and DevOps practices for data pipeline deployments - Guide and mentor junior data engineers, conduct code reviews, and assist in technical planning Qualifications Required: - 5+ years of experience in data engineering and data platform development - Proficiency in PySpark / Spark SQL for large-scale distributed data processing - Strong understanding of ETL/ELT architectures, data ingestion, and transformation strategies - Experience in data modeling techniques and tools - Demonstrated track record in performance tuning of queries, jobs, and pipelines - Familiarity with tools like Airflow or equivalent for orchestration and transformations Nice To Have: - Exposure to Data Virtualization platforms (Denodo, Dremio, etc.) - Familiarity with Purview or any Data Governance tools This job is based in Kochi initially, with the possibility of relocation to Bangalore.