Backend Data & Integration Engineer

As a Data Engineer at our company located in Kochi, you will be responsible for building data pipelines and connecting external systems and interfaces. Your role objective includes developing crawling/fetch pipelines, parsing/normalization of job postings & CVs, and embeddings/similarity search. Some key responsibilities include: - Development of crawling/fetch pipelines using an API-first approach; utilizing playwright/requests where permitted - Parsing/normalization of job postings & CVs, implementing deduplication/delta logic, and managing repost heuristics - Implementing embeddings/similarity search, controlling Azure OpenAI, and maintaining vector persistence in pgvector - Integrating with systems such as HR4YOU, SerpAPI, BA job board, and email/SMTP using APIs, webhooks, and CSV import - Handling batch/stream processing with Azure Functions/container jobs, managing retry/off, and dead-letter queues - Implementing telemetry for data quality, monitoring freshness, duplicate rate, coverage, and cost per 1,000 items - Collaborating with the Frontend team for exports, generating CSV/Excel files, presigned URLs, and admin configuration Qualifications required for this role include: - 4+ years of end/data engineering experience - Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium) and solid TypeScript for smaller services/SDKs - Experience with Azure services such as Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, and Monitor/Log Analytics - Familiarity with messaging systems like Service Bus/Queues, idempotence & exactly-once semantics, and clean ETL/ELT patterns - Strong database skills in PostgreSQL, pgvector, query design, and performance tuning - Knowledge of clean ETL/ELT patterns, testability with pytest, and observability with OpenTelemetry Nice-to-have qualifications include: - Experience in NLP/IE using spaCy/regex/rapidfuzz, and document parsing with pdfminer/textract - Knowledge of license/ToS-compliant data retrieval, captcha/anti-bot strategies, and working method emphasizing API-first and clean code practices - Familiarity with tools/stack such as GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, and Notion/Confluence - Willingness to participate in on-call/support duties on a rotating basis, following the "you build it, you run it" approach Join us in this exciting opportunity to contribute to our data engineering team and make a significant impact in the field of data processing and integration.,