2 Annoy Jobs

Your day at NTT DATA We are seeking an experienced Data Engineer to join our team in delivering cutting-edge Generative AI (GenAI) solutions to clients. The successful candidate will be responsible for designing, developing, and deploying data pipelines and architectures that support the training, fine-tuning, and deployment of LLMs for various industries. This role requires strong technical expertise in data engineering, problem-solving skills, and the ability to work effectively with clients and internal teams. What youll be doing Key Responsibilities: Design, develop, and manage data pipelines and architectures to support GenAI model training, fine-tuning, and deployment Data Ingestion and Integration: Develop data ingestion frameworks to collect data from various sources, transform, and integrate it into a unified data platform for GenAI model training and deployment. GenAI Model Integration: Collaborate with data scientists to integrate GenAI models into production-ready applications, ensuring seamless model deployment, monitoring, and maintenance. Cloud Infrastructure Management: Design, implement, and manage cloud-based data infrastructure (e.g., AWS, GCP, Azure) to support large-scale GenAI workloads, ensuring cost-effectiveness, security, and compliance. Write scalable, readable, and maintainable code using object-oriented programming concepts in languages like Python, and utilize libraries like Hugging Face Transformers, PyTorch, or TensorFlow Performance Optimization: Optimize data pipelines, GenAI model performance, and infrastructure for scalability, efficiency, and cost-effectiveness. Data Security and Compliance: Ensure data security, privacy, and compliance with regulatory requirements (e.g., GDPR, HIPAA) across data pipelines and GenAI applications. Client Collaboration: Collaborate with clients to understand their GenAI needs, design solutions, and deliver high-quality data engineering services. Innovation and R&D: Stay up to date with the latest GenAI trends, technologies, and innovations, applying research and development skills to improve data engineering services. Knowledge Sharing: Share knowledge, best practices, and expertise with team members, contributing to the growth and development of the team. Bachelors degree in computer science, Engineering, or related fields (Masters recommended) Experience with vector databases (e.g., Pinecone, Weaviate, Faiss, Annoy) for efficient similarity search and storage of dense vectors in GenAI applications 5+ years of experience in data engineering, with a strong emphasis on cloud environments (AWS, GCP, Azure, or Cloud Native platforms) Proficiency in programming languages like SQL, Python, and PySpark Strong data architecture, data modeling, and data governance skills Experience with Big Data Platforms (Hadoop, Databricks, Hive, Kafka, Apache Iceberg), Data Warehouses (Teradata, Snowflake, BigQuery), and lakehouses (Delta Lake, Apache Hudi) Knowledge of DevOps practices, including Git workflows and CI/CD pipelines (Azure DevOps, Jenkins, GitHub Actions) Experience with GenAI frameworks and tools (e.g., TensorFlow, PyTorch, Keras) Nice to have: Experience with containerization and orchestration tools like Docker and Kubernetes Integrate vector databases and implement similarity search techniques, with a focus on GraphRAG is a plus Familiarity with API gateway and service mesh architectures Experience with low latency/streaming, batch, and micro-batch processing Familiarity with Linux-based operating systems and REST APIs

Key Responsibilities: Design and Development: Design, architect, and deploy AI/GenAI models and solutions using various technologies and frameworks (e.g., TensorFlow, PyTorch, LangChain, Vellum etc) on non-cloud infrastructure. Agentic AI: Lead the development and integration of agentic AI systems, enabling autonomous decision-making and action-taking capabilities in AI solutions. Vector Database: Design and implement vector databases (e.g., Faiss, Annoy, Hnswlib) for efficient similarity search and retrieval in AI applications. Technical Leadership: Provide technical guidance and mentorship to junior team members, ensuring high-quality deliverables and adherence to best practices. Security of LLMs: Design and implement robust security measures to prevent data poisoning, model inversion attacks, and membership inference attacks, including data encryption, access controls, model watermarking, and regular security audits. Client Engagement: Collaborate with clients to understand their AI requirements, develop tailored solutions, and deliver high-quality results. Act as a trusted technical advisor. Model Development: Develop and fine-tune AI/GenAI models for specific use cases, such as natural language processing, computer vision, or predictive analytics. Testing and Validation: Design and oversee thorough testing and validation of AI/GenAI models, including performance evaluation, bias detection, and explainability. Deployment and Maintenance: Lead the deployment of AI/GenAI models in production environments, ensuring seamless integration with existing systems and infrastructure. Knowledge Sharing: Share knowledge and expertise with the team, contributing to the development of best practices and staying up-to-date with industry trends. Lead training sessions for team members. Collaboration: Work closely with cross-functional teams, including data science, engineering, and product management, to ensure successful project delivery. Requirements: Education: Bachelor/Master's in Computer Science, AI, ML, or related fields. Experience: 8+ years of experience in engineering solutions, with a track record of delivering AI solutions. Technical Skills: Advanced Proficiency in AI/GenAI technologies, including deep learning frameworks, NLP, and computer vision. Experience with vector databases and similarity search algorithms. Experience with security measures for LLMs, including data encryption, access controls, and model watermarking. Programming Skills: Strong programming skills in languages like Python or R Communication: Excellent communication and interpersonal skills, with the ability to work effectively with clients and internal teams. Problem-Solving: Strong problem-solving skills, with the ability to analyse complex problems and develop creative solutions. Nice to have: Experience with containerization (Docker) and orchestration (Kubernetes) Nice to have: Experience with ReactJS for rapid prototyping