Senior Data Engineer / Python Data Engineer

#Hiring #Python #Pyspark #LLM #PLSQL #Ab initio #Google ADK #Databricks

MLflow #Unity Catalog #SparkSQL #ETL

Job Title: Python Data Engineer – (PySpark, LangChain & LLM Integration)

Locations: Bangalore & Hyderabad

Experience for SE 3: 6–8 years

Experience for SE 4: 8-10 Years

Role Overview

Python Data Engineer

Primary skill:

Secondary Skill:

🛠️ Key Responsibilities

• Design and implement distributed data pipelines using

  PySpark

  and

  SparkSQL

• Build LLM-powered applications using

  LangChain

  , integrating with vector stores and document loaders
• Collaborate with ML teams to deploy models and manage embeddings (e.g., FAISS, Chroma)
• Optimize workflows using

  Databricks

  ,

  MLflow

  , and

  Unity Catalog

• Enable natural language querying over structured data using LangChain’s

  SparkSQLToolkit

• Ensure data quality, lineage, and governance across all systems

Required Skills

• Strong programming skills in

  Python

• Hands-on experience with

  PySpark

  and distributed data processing
• Familiarity with

  LangChain

  and LLM frameworks (OpenAI, HuggingFace, etc.)
• Experience with

  Databricks

  ,

  AWS EMR

  , or similar cloud platforms
• Solid understanding of

  SQL

  ,

  ETL

  , and data modeling

Good-to-Have Skills

• Experience with

  Google ADK Prompt Engineering

  for LLM optimization and fine-tuning
• Knowledge of

  LangChain Agents

  ,

  Toolkits

  , and

  Document Loaders

• Exposure to

  Azure Data Factory

  ,

  SageMaker

  , or

  EMR Serverless

• Background in NLP, generative AI, or conversational agents