Job
Description
**Role Overview:** You will be a hands-on Data Engineering Lead with 6+ years of experience and a proven track record of managing at least 2 Data Engineers. Your primary mission will be to lead a small team, write code, design pipelines, and ensure timely delivery of clean, lineage-rich data to power analytics, AI, and customer-facing products. **Key Responsibilities:** - Lead a small data engineering team - Manage and mentor 2+ Data Engineers by setting clear goals, providing feedback, and conducting regular 1:1s - Recruit, onboard, and develop the team through code reviews, pairing sessions, and personalized learning plans - Take ownership of ingestion and integration processes - Perform source-system analysis and negotiate data contracts with internal and external partners - Acquire external data with agreed SLAs and quality gates - Develop batch and streaming ingestion using tools like Airflow DAGs, Kafka topics, and Debezium CDC - Shape models and governance strategies - Manage and evolve schemas and metadata across MongoDB, OpenSearch, and ClickHouse - Build transformations with dbt and Spark, including SCD Type 2 and incremental CDC models - Populate OpenSearch and ClickHouse from MongoDB to enable search and analytics - Optimize reliability and cost efficiency - Define freshness, completeness, and accuracy SLAs with automated monitoring and alerts - Reduce costs through partitioning, clustering, compression, and tiered storage strategies - Publish and protect data effectively - Deliver a clear semantic layer and BI artifacts in OpenSearch Dashboards - Ensure GDPR and DPDP compliance, lineage capture, and role-based access controls - Drive a practical Agile process - Collaborate with Product and Analytics teams to shape work, provide reasonable estimates, and maintain delivery predictability - Conduct effective standups, reviews, and retrospectives to enhance outcomes **Qualifications Required:** - 7+ years of experience in data engineering or related software roles with a proven track record of shipping production systems - Experience managing at least 2 Data Engineers while actively contributing hands-on - Proficiency in Python for pipeline development and tooling **Additional Company Details:** The company follows a Pragmatic Agile approach which includes lightweight ceremonies, strong ownership, prioritizing code quality, accountability without drama, and a focus on continuous improvement. They value readable code, tests, useful documentation, and aim to measure, learn, and improve with each sprint.,
