Lead Engineer (AI/ML - Notebook)

Prodapt Solutions

5 - 10 years

11 - 15 Lacs

chennai

Posted:6 days ago| Platform:

Apply

Skills Required

python backend development data engineering debugging troubleshooting hive continuous integration kubernetes rbac ci/cd pyspark jupyter notebook machine learning sso artificial intelligence docker dataproc tableau spark gcp devops hadoop bigquery

Work Mode

Work from Office

Job Type

Full Time

Job Description

Overview

Join the Prodapt team in supporting a unified, scalable, and secure Jupyter-based environment for data science and machine learning. You will help build, maintain, and optimize the platform that empowers analysts, engineers, and scientists to explore data, develop models, and collaborate at scale.

Responsibilities

Develop, maintain, and enhance the Notebook platform built on JupyterLab, supporting both cloud and on-premises deployments.
Integrate and optimize connections to diverse data sources (BigQuery, Hive, Teradata, Hadoop, Spark, etc.).Enable and support distributed data processing and analytics using Spark/PySpark and Dataproc.Implement and maintain platform features for code execution, visualization, collaboration, and scheduling (e.g., Airflow integration).Ensure platform security, including SSO, role-based access control, and compliance with governance and data privacy standards.Support and automate ML workflows, including model training, experiment tracking (e.g., MLflow), and deployment. Monitor platform health, troubleshoot issues, and optimize for performance, scalability, and reliability (e.g., GKE, Kubernetes, Openshift). Collaborate with data scientists, engineers, and product teams to deliver robust, user-friendly solutions. Contribute to documentation, onboarding guides, and best practices for users and developers.

RequirementsRequired Technical Skills

Proficiency in Python, especially for backend development and data engineering.
Experience with JupyterLab/Jupyter Notebooks, including kernel and extension development.
Familiarity with distributed data processing frameworks (Spark, PySpark, Dataproc).
Experience with cloud platforms (GCP preferred), Kubernetes, and containerization (Docker).
Strong understanding of data source integration (BigQuery, Hive, Teradata, Hadoop).
Experience with CI/CD, DevOps, and monitoring tools for cloud-based platforms.
Knowledge of security best practices (SSO, RBAC, credential management).
Familiarity with ML/AI workflows, experiment tracking (MLflow), and model deployment.
Excellent troubleshooting, debugging, and communication skills.

Preferred Qualifications

Experience with platform governance, code classification, and compliance automation.
Familiarity with VS Code integration, workflow automation tools (Zonkey, Airflow), and dashboarding (Tableau).
Experience supporting multi-tenant, highly available, and scalable notebook environments.
Exposure to large-scale financial/ML platforms.

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

Prodapt Solutions

Software Development

Chennai Tamilnadu

Login to

Please Verify Your Phone or Email

Confirm Action

Lead Engineer (AI/ML - Notebook)