Data Engineer

About the Company

Company Overview - Rakuten Group

Founded in 1997, Rakuten Group, headquartered in Tokyo, is a global leader in internet services, boasting over 14,000 employees worldwide. What began as Japan's largest e-commerce company has evolved into one of the nation's premier internet conglomerates, offering over 70 diverse services globally. These services span across various sectors, including FinTech (Card, Securities, Bank, and Insurance), and a newly established mobile carrier business, positioning Rakuten as the 4th carrier in Japan. Rakuten's overarching vision is to become the world's largest membership company. The organization fosters a dynamic and innovative environment, blending the opportunities for growth inherent in a large corporation with the entrepreneurial spirit of a startup. Rakuten's global footprint extends across six continents and multiple industries. A key strategic pillar for Rakuten's future growth and innovation is its unwavering focus on "AI-nization" across all its services.

Company Overview - Rakuten India Development Centre (RIEPL)

Rakuten India Development Centre (RIEPL) stands as Rakuten's second-largest technology hub outside of Japan. RIEPL plays a crucial role in enabling and building platforms for global e-commerce, payments, digital services, AI, and data science services worldwide. As a vital research and development center, RIEPL is experiencing continuous growth, currently employing over 1500 professionals.

AI & Data Division (AIDD) and AIDD India

Rakuten's AI & Data Division (AIDD) is at the forefront of the company's transformative journey, spearheading cutting-edge research and the large-scale deployment of advanced AI technologies. This includes Large Language Models (LLMs), Generative AI, Conversational AI, and sophisticated Data Analytics. By leveraging AI-powered automation, intelligent search, and predictive analytics, AIDD is dedicated to building scalable, high-impact solutions. These solutions are designed to enhance customer interactions, optimize business processes, and unlock new opportunities across the entire Rakuten ecosystem.

AIDD India, a significant part of the global AIDD team, has made substantial contributions across multiple areas of Research, Engineering, and Services. The team has been instrumental in driving innovation. AIDD India's expertise spans from fundamental AI research to the engineering and deployment of complex AI systems, directly impacting Rakuten's diverse service offerings.

About the Role

You’ll build and operate the data pipelines that power our LLM-based agents—everything from document/knowledge ingestion and vector indexing to agent telemetry, evaluation datasets, and safety/PII workflows. You’ll partner closely with ML/Agent researchers, engineers, and product to ensure our agents are accurate, reliable, safe, and cost-effective at scale.

Responsibilities

• Build and maintain batch/streaming pipelines for:
• Knowledge ingestion and chunking/embedding
• Vector indexing/re-indexing and freshness SLAs
• Agent telemetry (traces, tool calls, prompts/responses, cost/latency)
• Offline eval datasets and golden sets
• Implement retrieval/RAG workflows: schema design, chunking strategies, reranking, metadata enrichment
• Own core agent analytics models (e.g., retrieval hit rate, hallucination proxies, tool success)
• Partner with ML/Agent teams on prompt/versioning, eval harnesses, caching
• Contribute to CI/CD, testing, and cost monitoring for data/vector infra

Qualifications

• 4+ years in data or analytics engineering with strong SQL and Python
• Strong experience in LLM app dev and observability with at least one of LangSmith, Langfuse, or similar for tracing, debugging, evaluations, prompt/version management, and basic production monitoring
• Hands-on with a retrieval stack using LangGraph or LlamaIndex (or equivalent custom framework)
• Experience with vector databases (e.g., Pinecone, Milvus, FAISS) and embedding pipelines
• Familiarity with core LLM/RAG concepts (chunking, embeddings, reranking, caching, prompt versioning, tool calling)

Required Skills

• Experience with a modern data warehouse (e.g., BigQuery or Redshift) and an orchestrator (e.g., Airflow or equivalent)
• Experience with data lakes (S3/GCS) and ETL/ELT tooling (dbt or equivalent)
• Practical containerization skills (Docker and Kubernetes)

```