Data Engineering Specialist

We are seeking a seasoned Technical Lead Data Engineering to lead and operate a large-scale data processing and reconciliation platform for an eCommerce enterprise. The role involves managing real-time data ingestion and batch processing (SQL, Python, PySpark, Hive), and BI reporting pipelines (MySQL, Tableau, ClickView) to support the Finance team in month-end reconciliation and MIS reporting.

You will be responsible for triaging reconciliation issues, performing deep root cause analysis (RCA) using data from MySQL, HBase, and HDFS, and implementing fixes with a focus on process automation, data lineage (DataHub), and platform reliability.

This role also includes leading L2 and L3 engineers to support the production platform and drive continuous improvement initiatives including change management and platform monitoring.

Required skills :

• Strong hands-on experience with: SQL, Python( or PySpark) Based ETL for a Data Lake environment (or a large DWH platform).
• Proficiency in SQL and Python (PySpark) performance tuning, and building complex business logic.
• Prior experience with GCP or Azure, and familiarity with distributed systems would be preferable.
• Strong understanding of incident management, triage, and change management processes.
• Knowledge of data lineage, data governance frameworks, and BI tools (Tableau, ClickView).
• Fast learner and willing to learn new data analytics tools, basic java programming and microservices architecture

Key Responsibilities:

• Lead engineering team and drive best practice maintenance and development tasks for an end-to-end data ingestion and processing pipeline for a large scale data lake platform.
• Execute large scale data processing and report generation for in order to drive successful financial reconciliation on monthly basis.
• Work closely with the Finance Controllers.
• Perform RCA, issue triaging and resolve anomalies.
• Build and monitor data quality, lineage, and SLA adherence across systems.
• Enable observability through monitoring dashboards and diagnostics
• Stakeholder management