Senior Backend Data & Integration Engineer

Role Overview: As a Senior Backend Data & Integration Engineer at our company, your primary responsibility will be to build data pipelines and connect external systems and interfaces. You will be working with a variety of technologies such as Python, TypeScript, Azure, and integrations to ensure smooth data flow and seamless integrations. Key Responsibilities: - Develop crawling/fetch pipelines using an API-first approach, utilizing playwright/requests where permitted. - Parse and normalize job postings & CVs, implement deduplication/delta logic utilizing seen hash and repost heuristics. - Work on embeddings/similarity search, controlling Azure OpenAI and ensuring vector persistence in pgvector. - Handle integrations with systems such as HR4YOU (API/webhooks/CSV import), SerpAPI, BA job board, and email/SMTP. - Implement batch/stream processing using Azure Functions/container jobs, along with retry/backoff and dead-letter queues. - Monitor data quality through telemetry, focusing on freshness, duplicate rate, coverage, and cost per 1,000 items. - Collaborate with the frontend team for exports using CSV/Excel, presigned URLs, and admin configuration. Qualifications Required: - 4+ years of backend/data engineering experience. - Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium) and solid TypeScript for smaller services/SDKs. - Experience with Azure technologies such as Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics. - Familiarity with messaging systems like Service Bus/Queues, understanding idempotence & exactly-once semantics, and adopting a pragmatic approach. - Strong knowledge of databases including PostgreSQL, pgvector, query design & performance tuning. - Ability to implement clean ETL/ELT patterns, ensure testability using pytest, and maintain observability with OpenTelemetry. Additional Details: - Nice-to-have skills include NLP/IE experience (spaCy/regex/rapidfuzz), document parsing using pdfminer/textract. - Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies while ensuring legal compliance. - Working method involves API-first approach, clean code practices, trunk-based development, and mandatory code reviews. - Proficiency with tools/stack such as GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, Notion/Confluence. - Participation in on-call/support duties will be on a rotating basis, following the "you build it, you run it" philosophy.,