Python Data Engineer

Key Responsibilities

• Design and implement distributed data pipelines using

  PySpark

  and

  SparkSQL

• Build LLM-powered applications using

  LangChain

  , integrating with vector stores and document loaders
• Collaborate with ML teams to deploy models and manage embeddings (e.g., FAISS, Chroma)
• Optimize workflows using

  Databricks

  ,

  MLflow

  , and

  Unity Catalog.

• Enable natural language querying over structured data using LangChains

  SparkSQLToolkit

• Ensure data quality, lineage, and governance across all systems

Required Skills

• Strong programming skills in

  Python

• Hands-on experience with

  PySpark

  and distributed data processing
• Familiarity with

  LangChain

  and LLM frameworks (OpenAI, Hugging Face, etc.)
• Experience with

  Databricks

  ,

  AWS EMR

  , or similar cloud platforms
• Solid understanding of

  SQL

  ,

  ETL

  , and data modeling

Good-to-Have Skills

• Experience with

  Google ADK Prompt Engineering

  for LLM optimization and fine-tuning
• Knowledge of

  LangChain Agents

  ,

  Toolkits

  , and

  Document Loaders

• Exposure to

  Azure Data Factory

  ,

  SageMaker

  , or

  EMR Serverless

• Background in NLP, generative AI, or conversational agents