Job
Description
We are looking for a highly skilled Solution Architect to lead the design and implementation of a Data and Machine Learning Platform spanning edge, cloud, and on-prem components. The ideal candidate will have deep experience in Azure cloud services, data engineering, edge computing, and ML lifecycle management. Technical Expertise Azure Cloud Stack DevOps Azure Databricks (including ML workspace for Feature Store and Model Store) Azure Data Factory (ADF) for orchestration and compute Azure Data Lake Storage (ADLS) implementing medallion architecture (raw, bronze, silver, gold) Azure Event Hub: Experience in defining topics, managing consumer groups, and integrating ETL events Azure Streaming Analytics: Real-time data processing for telemetry and operational data Azure Key Vault Azure App Service Azure Container Registry (ACR) Azure IoT Hub for connecting edge devices Azure DevOps GitHub Actions (for CI/CD pipelines) GitHub self-hosted runners for ML workflow automation Edge and On-Prem Integration Strong experience in OT-IT integration and data extraction from industrial systems Edge VM deployment using: - Docker and Portainer for container orchestration - RabbitMQ for messaging (read/write services from edge) - OPC UA for interfacing with PLCs (eg, FX Filter, NH3 Compressor) - IDMZ deployment practices and edge-to-cloud data service integration Machine Learning Platform (MLP) and MLOps End-to-end ML lifecycle implementation: Feature Engineering, Model Training Validation, Model Export, Versioning, and Deployment Hands-on with ADB ML workspace, Feature Store, Model Store Monitoring deployed models at 1-minute intervals Understanding of training vs inference, cloud vs edge deployment Cadence for ML models (Weekly Refresh, Monthly Retrain, Quarterly Revamp) Use of GitHub monorepo structure for managing model code Data Architecture Integration Implementation of medallion architecture in the data platform Integration with Unity Catalog (UC) for governance, data sharing, and cataloging Experience with CDC tools (eg, Aecorsoft) for real-time SAP data ingestion Consumption layer design for BI, ML, and operational workloads Familiarity with streaming and API-based ingestion from external environments Template-driven ingestion and mapping using configurations Governance and Data Modeling Define and enforce data governance standards using Unity Catalog and enterprise frameworks. Design scalable data models to support operational analytics and ML features. Implement policies for access control, quality, and metadata tagging across DLZ/zones. Key Responsibilities 1. Architect Integrated Solutions: Lead architectural design across edge, cloud, and ML across zones 2. Build and Govern Data Platform: Oversee ingestion, transformation, and cataloging across Raw Gold layers, aligned to UC. 3. Enable Scalable ML Platform: Support ML teams with infrastructure for feature storage, model ops, and deployment pipelines. 4. Edge Integration and Automation: Design robust and secure OT-IT interfaces with RabbitMQ, OPC UA, and container orchestration tools. 5. Monitor and Optimize Pipelines: Set up real-time monitoring for ML and ETL pipelines; optimize for performance and cost. 6. Governance and Security Compliance: Ensure enterprise compliance, tagging, and secure access across all zones and services. 7. Lead CI/CD Automation: Use GitHub Actions and Azure DevOps to streamline deployment of ML workflows and platform components. Skills Devops, Azure, Rabbitmq, Data Modeling, Scala, Compliance, Machine Learning, Vault, Workflow, Docker, Cadence, Cloud Service
