Job
Description
Technical ExpertiseAzure Cloud Stack & DevOpsAzure Databricks (including ML workspace for Feature Store and Model Store) Azure Data Factory (ADF) for orchestration and compute Azure Data Lake Storage (ADLS) implementing medallion architecture (raw, bronze, silver, gold) Azure Event Hub: Experience in defining topics, managing consumer groups, and integrating ETL events Azure Streaming Analytics: Real-time data processing for telemetry and operational data Azure Key Vault Azure App Service Azure Container Registry (ACR) Azure IoT Hub for connecting edge devices Azure DevOps & GitHub Actions (for CI/CD pipelines) GitHub self-hosted runners for ML workflow automation Edge and On-Prem IntegrationStrong experience in OT-IT integration and data extraction from industrial systems Edge VM deployment using: Docker and Portainer for container orchestration RabbitMQ for messaging (read/write services from edge) OPC UA for interfacing with PLCs (e.g., FX Filter, NH3 Compressor) IDMZ deployment practices and edge-to-cloud data service integration Machine Learning Platform (MLP) and MLOpsEnd-to-end ML lifecycle implementation: Feature Engineering, Model Training & Validation, Model Export, Versioning, and Deployment Hands-on with ADB ML workspace, Feature Store, Model Store Monitoring deployed models at 1-minute intervals Understanding of training vs inference, cloud vs edge deployment Cadence for ML models (Weekly Refresh, Monthly Retrain, Quarterly Revamp) Use of GitHub monorepo structure for managing model code Data Architecture & IntegrationImplementation of medallion architecture in the data platform Integration with Unity Catalog (UC) for governance, data sharing, and cataloging Experience with CDC tools (e.g., Aecorsoft) for real-time SAP data ingestion Consumption layer design for BI, ML, and operational workloads Familiarity with streaming and API-based ingestion from external environments Template-driven ingestion and mapping using configurations Governance and Data ModelingDefine and enforce data governance standards using Unity Catalog and enterprise frameworks. Design scalable data models to support operational analytics and ML features. Implement policies for access control, quality, and metadata tagging across DLZ/zones. Key Responsibilities Architect Integrated Solutions: Lead architectural design across edge, cloud, and ML across zones Build and Govern Data Platform: Oversee ingestion, transformation, and cataloging across Raw Gold layers, aligned to UC. Enable Scalable ML Platform: Support ML teams with infrastructure for feature storage, model ops, and deployment pipelines. Edge Integration and Automation: Design robust and secure OT-IT interfaces with RabbitMQ, OPC UA, and container orchestration tools. Monitor and Optimize Pipelines: Set up real-time monitoring for ML and ETL pipelines; optimize for performance and cost. Governance and Security Compliance: Ensure enterprise compliance, tagging, and secure access across all zones and services. Lead CI/CD Automation: Use GitHub Actions and Azure DevOps to streamline deployment of ML workflows and platform components.
