Senior Data Engineer

Sr. Data Engineer - immediate joiners only

Year of Experience – 5-7 Years

Location – Hybrid, Pune

Shift Timings – 02:30 PM to 11:30 PM (Mon-Fri)

Department: Information Technology

Python-based data stacks, scalable APIs, modern databases

As a Senior Data Engineer – AI Systems at Bold Penguin, you’ll design and build the data and AI infrastructure that powers the next generation of commercial insurance experiences. You’ll work at the intersection of data engineering, machine learning, and AI agent systems, helping redefine how small business owners shop, quote, and bind insurance.

You’ll lead initiatives that combine traditional data engineering. with cutting-edge AI capabilities—LLMs, embedding, and agentic workflows—enabling intelligent automation, contextual insights, and faster decision-making across our platform.

Responsibilities

● Architect and implement large-scale data pipelines supporting analytics, machine learning, and AI agent workloads.

● Integrate AI models (LLMs, embedding, chat models) into data services to power intelligent search, summarization, and reasoning features.

● Design and maintain retrieval-augmented generation (RAG) pipelines, including vector storage, prompt templates, and context assembly mechanisms.

● Develop and orchestrate AI agent workflows using frameworks such as LangChain, CrewAI, LlamaIndex, or AutoGen.

● Collaborate with ML and product teams to transform business and insurance data into structured knowledge for use in chat or reasoning models.

● Build and scale microservices and APIs for data and AI integration, using Python, Java, or Node.js.

● Implement CI/CD, data quality, and monitoring pipelines using AWS native or CNCF-aligned tools.

● Optimize performance of large data processing and semantic search pipelines through parallelization and caching strategies.

● Provide technical leadership and mentorship to junior data engineers in architecture, coding best practices, and AI integration patterns.

● Troubleshoot, diagnose, and resolve production issues within defined SLAs.

Qualifications

● Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related technical field.

● 5+ years of experience in software or data engineering, with a proven record of delivering production-grade systems.

● Strong proficiency in Python and one other language (Java, Node.js, or Go).

● Deep experience with data modeling, ETL pipelines, and both document (MongoDB) and relational databases (PostgreSQL).

● Hands-on experience integrating LLMs, embedding, or vector databases (e.g., Pinecone, Weaviate, FAISS, or Chrome).

● Proficiency in prompt engineering and context management for AI and chat applications.

● Experience with AI agent frameworks (LangChain, CrewAI, LlamaIndex, or similar) and modern orchestration tools (Prefect, Airflow).

● Solid understanding of CI/CD, infrastructure-as-code, and observability within AWS or Kubernetes environments.

● Excellent communication and cross-functional collaboration skills.

● (Preferred) Familiarity with insurance, fin tech, or regulated data environments.