Data Engineer + AI

Data Engineer + AIJob Summary:We are looking for a skilled and versatile Data Engineer with expertise in PySpark, Apache Spark, and Databricks, along with experience in analytics, data modeling, and Generative AI/Agentic AI solutions
This role is ideal for someone who thrives at the intersection of data engineering, AI systems, and business insights contributing to high-impact programs with clientsRequired Skills & Experience: Advanced proficiency in PySpark, Apache Spark, and Databricks for batch and streaming data pipelines Strong experience with SQL for data analysis, transformation, and modeling Expertise in data visualization and dashboarding tools (Power BI, Tableau, Looker) Solid understanding of data warehouse design, relational databases (PostgreSQL, Snowflake, SQL Server), and data lakehouse architectures Exposure to Generative AI, RAG, embedding models, and vector databases (eg, FAISS, Pinecone, ChromaDB) Experience with Agentic AI frameworks: LangChain, Haystack, CrewAI, or similar Familiarity with cloud services for data and AI (Azure, AWS, or GCP) Excellent problem-solving and collaboration skills with an ability to bridge engineering and business needsPreferred Skills: Experience with MLflow, Delta Live Tables, or other Databricks-native AI tools Understanding of prompt engineering, LLM deployment, and multi-agent orchestration Knowledge of CI/CD, Git, Docker, and DevOps pipelines Awareness of Responsible AI, data privacy regulations, and enterprise data compliance Background in consulting, enterprise analytics, or AI/ML product developmentKey Responsibilities: Design, build, and optimize distributed data pipelines using PySpark, Apache Spark, and Databricks to support both analytics and AI workloads Support RAG pipelines, embedding generation, and data pre-processing for LLM applications Create and maintain interactive dashboards and BI reports using Power BI, Tableau, or Looker for business stakeholders and consultants Conduct adhoc data analysis to drive data-driven decision making and enable rapid insight generation Develop and maintain robust data warehouse schemas, star/snowflake models, and support data lake architecture Integrate with and support LLM agent frameworks such as LangChain, LlamaIndex, Haystack, or CrewAIfor intelligent workflow automation Ensure data pipeline monitoring, cost optimization, and scalability in cloud environments (Azure/AWS/GCP) Collaborate with cross-functional teams including AI scientists, analysts, and business teams to drive use-case delivery Maintain strong data governance, lineage, and metadata management practices using tools like Azure Purview or DataHub