Data Engineer, Senior Data Engineer

Role: Senior Data Engineer Role Overview

• Build and optimize

  end-to-end data pipelines

  for AI/LLM/GenAI workloads.
• Develop ingestion, transformation, validation, and metadata frameworks for structured, semi-structured, and unstructured data.
• Enable data preprocessing for AI: feature extraction, embeddings, chunking, enrichment, semantic layers, etc.
• Design on-prem data architecture componentsstaging, processing, storage, orchestration.
• Implement ETL/ELT workflows ensuring performance, scalability, and operational reliability.
• Collaborate with client SMEs, product owners, and tech teams to translate requirements into data models and pipeline designs.
• Work within Agile/Scrum ceremonies—estimations, sprint planning, technical story writing.
• Ensure compliance with

  secure network constraints

  , including access controls, logging, governance, encryption, and auditability.
• Contribute to CI/CD, code reviews, documentation, and best practices for on-prem data engineering.

Required Skills & Experience

• 6–10+ years in Data Engineering or Big Data pipeline development.
• Strong hands-on expertise in

  Python, PySpark, Spark, SQL

  .
• Experience with ETL/ELT frameworks, data modeling, data warehousing, and orchestration tools (Airflow, Dagster, Prefect, etc.).
• Experience of developing ETL pipelines and working with structure and unstructured data sources.
• Proficiency building pipelines for AI/ML/LLM use cases (data enrichment, embeddings) – preferred.
• Experience in on-prem or secure-network deployments with resource and access constraints.
• Knowledge of streaming/messaging systems (Kafka or similar).
• Strong analytical, problem-solving, and communication skills; comfortable as an individual contributor in client-facing environments.

Good-to-Have Skills

• Exposure to GenAI pipelines (RAG, vector databases, semantic search).
• Familiarity with AWS data services (S3, Glue, Lambda, Redshift, RDS, EMR) or equivalent cloud tools.
• Understanding of knowledge graphs, metadata-driven pipelines, or ontology-based systems.
• Experience in regulated environments (Healthcare, Insurance, Networks, Government).

Education