Data Integration Specialist

As a Senior Backend Data & Integration Engineer at our company, your role will involve building data pipelines and connecting external systems and interfaces. Key responsibilities include: - Developing crawling/fetch pipelines using an API-first approach with tools like playwright/requests - Parsing and normalizing job postings & CVs, implementing deduplication/delta logic - Implementing embeddings/similarity search with control over Azure OpenAI and vector persistence in pgvector - Handling integrations with systems like HR4YOU, SerpAPI, BA job board, and email/SMTP - Working on batch/stream processing with Azure Functions/container jobs, implementing retry/backoff mechanisms and dead-letter queues - Implementing telemetry for monitoring data quality metrics such as freshness, duplicate rate, coverage, and cost per 1,000 items - Collaborating with the front-end team for data exports in CSV/Excel format, utilizing presigned URLs, and assisting in admin configuration Qualifications required for this role include: - At least 4 years of backend/data engineering experience - Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium) and solid TypeScript skills for smaller services/SDKs - Experience with Azure services such as Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, and Monitor/Log Analytics - Familiarity with messaging systems like Service Bus/Queues, and understanding of idempotence & exactly-once semantics - Strong knowledge of databases including PostgreSQL, pgvector, query design, and performance tuning - Ability to implement clean ETL/ELT patterns, ensure testability with pytest, and maintain observability using OpenTelemetry Nice-to-have qualifications include: - Experience with NLP/IE tools like spaCy, regex, rapidfuzz, and document parsing libraries such as pdfminer/textract - Familiarity with license/ToS-compliant data retrieval methods, and strategies for handling captchas/anti-bot measures - Proficiency in working with an API-first approach, writing clean code, and following trunk-based development practices - Experience with tools/stacks like GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, and Notion/Confluence,