Data Engineer (4–6 Years Experience) – GCS + Iceberg + Jira Integration

We’re looking for an experienced Data Engineer (4–6 years) who can work independently, explore new technologies, and build solutions through quick prototypes (POCs).

This role will focus on setting up the data backbone for our AI-driven SDLC automation platform — starting with Jira and XRay integration, building scalable pipelines on Google Cloud, and preparing data models that can later connect with code and test data.

What You’ll Work On

1. Ingestion & Lakehouse Setup

Build data pipelines to collect issues, epics, stories, comments, and changelogs from Jira and XRay using webhooks and APIs.

Store raw data in Google Cloud Storage (GCS) as the Bronze layer.

Set up and manage Apache Iceberg tables (Silver layer) to create clean, queryable datasets such as requirements.issue, requirements.issue_change, and requirements.link.

2. Data Modeling & Transformation

Design and implement ETL pipelines to convert Jira JSON into structured Iceberg tables using SQL or Spark/Trino.

Build an acceptance criteria parser that converts Jira text (Given–When–Then or bullet points) into structured JSON.

Create reusable dbt or SQL models for key metrics like requirement coverage, data freshness, and change churn.

3. APIs & Insights

Develop simple REST APIs to expose Jira data for our internal platform, including endpoints like:

/requirements/search?q=

/requirements/:id/history

/requirements/:id/impact (future integration with code repo).

Prepare initial Gold-layer analytics views (in BigQuery or Iceberg) to show insights such as coverage, velocity, and change frequency.

4. Prepare for Future Integration

Design schemas and join keys that will later link Jira → Code → Tests → Defects.

Make sure data contracts remain consistent with the upcoming code repository pipeline.

Tech Stack

Cloud / Infrastructure: Google Cloud Platform (GCS, BigQuery, Cloud Run, Terraform)

Data: Apache Iceberg (preferred) or Delta Lake, Trino/Spark, dbt (optional)

Backend: Python (FastAPI or Flask)

Integrations: Jira REST APIs, Webhooks, OAuth2

Version Control: GitHub (GitOps workflow)

We need resources who can start immediately and work independently.