Senior Data Engineer

CivicDataLab is seeking a Senior Data Engineer to support our current and upcoming interventions through open data platforms and open-source tools, focusing on building robust data pipelines for automated data mining, cleaning, standardisation and transformation for AI readiness. We are looking for people who are strongly aligned with our values and have an innate sense of problem-solving, automating processes and adapting well to dynamic environments. They will work alongside data strategists, public policy researchers and other stakeholders to develop automated ETL (Extract, Transform and Load) data pipelines and know how to model, store and manage bulk datasets. This will help us co-create comprehensive data analytics tools and dashboards for our diverse stakeholders.

At CivicDataLab, we believe in using data, tech, design, and social science to strengthen civic engagement and drive evidence-based decision-making. Our projects centre on building data strategy, platforms, and applications to foster data-driven governance.

Key Responsibilities:

• Take ownership over developing scalable data infrastructure for our existing platforms and products from data orchestration pipelines using Prefect and Airflow, to Data Sharing Protocols for our open data collaboratives.

• Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources.

• Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India, and oversee dataset migration into new platforms.

• Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.

• Lead strategic planning and scoping for data engineering projects, aligning them with product roadmaps and stakeholder needs.

• Mentor and provide technical guidance to a team of data engineers, conducting code reviews and capacity-building sessions.

• Quality & compliance: Run structured checklists (licensing/ownership validation, PII checks, metadata completeness) and coordinate fixes with publishers.