Data Engineer with MLOps

Data_and_MLOps_Engineer - Contract - onsite

• Experience within the Azure ecosystem, including Azure AI Search, Azure Storage Blob, Azure Postgres, with expertise in leveraging these tools for data processing, storage, and analytics tasks.

• Proficiency in data preprocessing and cleaning large datasets efficiently using Azure Tools, Python, and other data manipulation tools.

• Strong background in Data Science/MLOps, with hands-on experience in DevOps, CI/CD, Azure Cloud computing, and model monitoring.

• Expertise in healthcare data standards, such as HIPAA and FHIR, with a deep understanding of sensitive data handling and data masking techniques to protect PII and PHI.

• In-depth knowledge of search algorithms, indexing techniques, and retrieval models for effective information retrieval tasks.

• Experience with chunking techniques and working with vectors and vector databases like Pinecone.

• Ability to design, develop, and maintain scalable data pipelines for processing and transforming large volumes of structured and unstructured data, ensuring performance and scalability.

• Implement best practices for data storage, retrieval, and access control to maintain data integrity, security, and compliance with regulatory requirements.

• Implement efficient data processing workflows to support the training and evaluation of solutions using large language models (LLMs), ensuring that models are reliable, scalable, and performant.

• Proactively identify and resolve data quality issues, pipeline failures, or resource contention to minimize disruption to systems.

• Experience with large language model frameworks, such as Langchain, and the ability to integrate them into data pipelines for natural language processing tasks.

• Familiarity with Snowflake for data management and analytics, with the ability to work within the Snowflake ecosystem to support data processes.

• Knowledge of cloud computing principles and hands-on experience with deploying, scaling, and monitoring AI solutions on platforms like Azure, AWS, and Snowflake.

• Ability to communicate complex technical concepts effectively to both technical and non-technical stakeholders, and collaborate with cross-functional teams.

• Analytical mindset with attention to detail, coupled with the ability to solve complex problems efficiently and effectively.

• Knowledge of cloud cost management principles and best practices to optimize cloud resource usage and minimize costs.

• Experience with ML model deployment, including testing, validation, and integration of machine learning models into production systems.

• Knowledge of model versioning and management tools, such as MLflow, DVC, or Azure Machine Learning, for tracking experiments, versions, and deployments.

• Model monitoring and performance optimization, including tracking model drift and addressing performance issues to ensure models remain accurate and reliable.

• Automation of ML workflows through CI/CD pipelines, ensuring efficient delivery, testing, and deployment.

• Monitoring and logging of AI/ML systems post-deployment to ensure reliability and performance.

• Collaboration with data scientists and engineering teams to deliver integrated retraining, fine-tuning, and updating.

• Familiarity with containerization technologies, such as Docker and Kubernetes, for scaling machine learning models in production environments.

• Ability to implement model governance practices to ensure compliance and provide transparency into models in production environments.

• Understanding of model explainability and interpretability techniques to provide transparent insights into model behavior.

Must Have:

• Minimum of 10 years experience as a data engineer.

• Hands-on experience using Azure Cloud ecosystem.

• Deep knowledge in AI/ML and MLOps.

• Hands-on experience with DevOps, CI/CD, model monitoring.

• Hands-on experience in healthcare domain.

• Experience with containerization technologies.

• Hands-on experience working with unstructured data.

Good to Have:

• Experience with Snowflake, MLflow, DVC, experiment intelligence, Snowflake, function app, Azure AI Search.