Data engineer Lead- Python Azure

Description - External

Key Responsibilities:

• Develop and deploy data pipelines and machine learning models on cloud platforms (Azure, Databricks, AzureML).
• Utilize Python and its libraries (scikit-learn, Pandas, NumPy, TensorFlow, PyTorch) for data processing and model development.
• Implement and manage RAG modeling using vector databases (Milvus, Chroma, FAISS).
• Monitor and maintain the health and performance of deployed ML models using model observability tools.
• Design, build, and optimize AI models and generative AI applications.
• Create and fine-tune prompts for generative AI systems, including zero-shot, few-shot, and chain-of-thought approaches.
• Ensure the scalability, reliability, and efficiency of AI/ML solutions in production environments.
• Conduct exploratory data analysis (EDA) to extract meaningful insights and prepare data for modeling.
• Apply MLOps practices to streamline model deployment and management.
• Build and maintain CI/CD pipelines (Terraform).
• Ensure proper versioning of models and data using tools like MLFlow, Delta Lake, DVC, and LakeFS.

Qualifications - External

Required Skills:

• Proficiency in Python and its libraries (scikit-learn, Pandas, NumPy, TensorFlow, PyTorch).

• Experience with cloud platforms (Azure,

  Databricks

  , AzureML).
• Knowledge of

  Lang Chain and RAG

  modeling with

  vector databases

  (like Milvus, Chroma, FAISS).
• Understanding of

  deep learning, particularly in NLP

  .
• Familiarity with model observability tools for tracking, alerting, and compliance monitoring.
• Strong MLOps skills.
• Experience with model versioning tools (MLFlow) and data versioning tools (Delta Lake, DVC, LakeFS).
• Ability to build and maintain CI/CD pipelines.

Preferred Qualifications:

• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.
• Proven track record of working on AI/ML projects.