Senior Backend Data & Integration Engineer

As a Data Engineer, your role will involve building data pipelines such as crawling/fetch pipelines, parsing job postings & CVs, and creating embeddings for similarity search. You will also be responsible for integrating with external systems like HR4YOU, SerpAPI, and BA job board, and ensuring batch/stream processing and telemetry for data quality. Key Responsibilities: - Develop crawling/fetch pipelines using an API-first approach and tools like playwright/requests - Normalize job postings & CVs, implement deduplication/delta logic, and perform embeddings/similarity search - Integrate with external systems such as HR4YOU, SerpAPI, BA job board, email/SMTP, etc. - Implement batch/stream processing, retry/backoff mechanisms, and dead-letter queues - Ensure telemetry for monitoring data quality metrics like freshness, duplicate rate, coverage, etc. - Collaborate with frontend team for data exports and admin configuration Qualifications Required: - 4+ years of backend/data engineering experience - Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium) and TypeScript - Experience with Azure services like Functions/Container Apps, AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics - Knowledge of messaging systems, databases like PostgreSQL, pgvector, and ETL/ELT patterns - Familiarity with testability (pytest), observability (OpenTelemetry), and clean code practices Additional Company Details: The company follows an API-first working method, emphasizes clean code, trunk-based development, and mandatory code reviews. Tools/stack used include GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo, Jira/Linear, and Notion/Confluence. The working culture also involves rotating on-call/support where "you build it, you run it." Please let me know if you need more information or clarification regarding the job description.,