Sr Data and MLOps Engineer - Hybrid(Locations Bangalore/Chennai/Trich

Job Description

Title: Sr Data and MLOps Engineer Location: Hybrid (Bangalore/Chennai/Trichy) Description: • Experience within the Azure ecosystem, including Azure AI Search, Azure Storage Blob, Azure Postgres, with expertise in leveraging these tools for data processing, storage, and analytics tasks. • Proficiency in data preprocessing and cleaning large datasets efficiently using Azure Tools, Python, and other data manipulation tools. • Strong background in Data Science/MLOps, with hands-on experience in DevOps, CI/CD, Azure Cloud computing, and model monitoring. • Expertise in healthcare data standards, such as HIPAA and FHIR, with a deep understanding of sensitive data handling and data masking techniques to protect PII and PHI. • In-depth knowledge of search algorithms, indexing techniques, and retrieval models for effective information retrieval tasks. Experience with chunking techniques and working with vectors and vector databases like Pinecone. • Ability to design, develop, and maintain scalable data pipelines for processing and transforming large volumes of structured and unstructured data, ensuring performance and scalability. • Implement best practices for data storage, retrieval, and access control to maintain data integrity, security, and compliance with regulatory requirements. • Implement efficient data processing workflows to support the training and evaluation of solutions using large language models (LLMs), ensuring that models are reliable, scalable, and performant. • Proactively identify and resolve data quality issues, pipeline failures, or resource contention to minimize disruption to systems. • Experience with large language model frameworks, such as Langchain, and the ability to integrate them into data pipelines for natural language processing tasks. • Familiarity with Snowflake for data management and analytics, with the ability to work within the Snowflake ecosystem to support data processes. • Knowledge of cloud computing principles and hands-on experience with deploying, scaling, and monitoring AI solutions on platforms like Azure, AWS, and Snowflake. • Ability to communicate complex technical concepts effectively to both technical and non-technical stakeholders, and collaborate with cross-functional teams. • Analytical mindset with attention to detail, coupled with the ability to solve complex problems efficiently and effectively. • Knowledge of cloud cost management principles and best practices to optimize cloud resource usage and minimize costs. • Experience with ML model deployment, including testing, validation, and integration of machine learning models into production systems. • Knowledge of model versioning and management tools, such as MLflow, DVC, or Azure Machine Learning, for tracking experiments, versions, and deployments. • Model monitoring and performance optimization, including tracking model drift and addressing performance issues to ensure models remain accurate and reliable. • Automation of ML workflows through CI/CD pipelines, enabling smooth model training, testing, validation, and deployment. • Monitoring and logging of AI/ML systems post-deployment to ensure consistent reliability, scalability, and performance. • Collaboration with data scientists and engineering teams to facilitate model retraining, fine-tuning, and updating. • Familiarity with containerization technologies, like Docker and Kubernetes, for deploying and scaling machine learning models in production environments. • Ability to implement model governance practices to ensure compliance and auditability of AI/ML systems. • Understanding of model explainability and the use of tools and techniques to provide transparent insights into model behavior. Must Have: • Minimum of 10 years experience as a data engineer • Hands-on experience with Azure Cloud eco-system. • Hands-on experience using Python for data manipulation. • Deep understanding of vectors and vector databases. • Hands-on experience scaling POC to production. • Hands-on experience using tools such as Document Intelligence, Snowflake, function app. Azure AI Search • Experience working with PII/PHI • Hands-on experience working with unstructured data. Role & responsibilities