Backend Data & Integration Engineer BEO Software Private Limited

4.0 - 8.0 years

0 Lacs

kochi, kerala

On-site

As a Data Engineer, your main objective will be to build data pipelines for crawling, parsing, and connecting external systems and interfaces. This includes developing crawling and fetching pipelines using an API-first approach and tools like playwright and requests. You will also be responsible for parsing and normalizing job postings and CVs, implementing deduplication and delta logic, and working on embeddings and similarity search. Additionally, you will be involved in integrating with various systems such as HR4YOU, SerpAPI, BA job board, and email/SMTP. Your role will also require you to work on batch and stream processing using Azure Functions or container jobs, implementing retry/backoff strategies, and setting up dead-letter queues for error handling. Monitoring data quality metrics such as freshness, duplicate rate, coverage, and cost per 1,000 items will be crucial. You will collaborate with the frontend team for data exports and admin configuration, ensuring seamless data flow across the system. The ideal candidate for this role should have at least 4 years of experience in backend/data engineering. Proficiency in Python, especially with FastAPI, pydantic, httpx/requests, and Playwright/Selenium, as well as solid experience in TypeScript for smaller services and SDKs is required. Familiarity with Azure services like Functions/Container Apps, Storage/Blob, Key Vault, and Monitor/Log Analytics is essential. Experience with messaging systems like Service Bus/Queues, databases such as PostgreSQL and pgvector, and clean ETL/ELT patterns is highly desirable. Knowledge of testability using pytest, observability with OpenTelemetry, and NLP/IE experience with tools like spaCy, regex, and rapidfuzz will be advantageous. Moreover, experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies, and a working method focused on API-first approach, clean code, and trunk-based development will be beneficial. Familiarity with tools like GitHub, Docker, GitHub Actions/Azure DevOps, pnpm/Turborepo, Jira/Linear, and Notion/Confluence is a plus. This role may involve rotating on-call support responsibilities and following the "you build it, you run it" approach to ensure operational efficiency and accountability.,

Posted 1 week ago

Apply

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.