ion Architect - Data & Machine Learning Platform (MLP)

Key Responsibilities

Architect Integrated Solutions:

• Lead the architectural design and implementation across edge devices, cloud infrastructure, and machine learning workflows covering raw-to-gold data layers.

Build and Govern the Data Platform:

• Manage data ingestion, transformation, and cataloging across medallion architecture zones (Raw, Bronze, Silver, Gold), aligned with Unity Catalog for governance.

Enable Scalable ML Platform:

• Support ML teams through the development and maintenance of infrastructure for feature storage, model operations, deployment, and monitoring.

Edge Integration and Automation:

• Design secure and scalable OT-IT integration using Docker, Portainer, RabbitMQ, OPC UA, and edge-to-cloud communication practices (including IDMZ).

Monitor and Optimize Pipelines:

• Implement real-time monitoring for ETL and ML pipelines using tools such as Prometheus, and optimize workloads for performance and cost efficiency.

Governance and Security Compliance:

• Enforce data governance standards, including access control, metadata tagging, and enterprise compliance across zones using Unity Catalog and Azure-native tools.

Lead CI/CD Automation:

• Automate the deployment of platform components and ML workflows using Azure DevOps, GitHub Actions, and self-hosted runners in a monorepo structure.

Technical Expertise

Azure Cloud & DevOps:

• Azure Data Factory (ADF) for orchestration
• Azure Databricks (ADB) with ML workspace: Feature Store, Model Store
• Azure Data Lake Storage (ADLS) using medallion architecture
• Azure Event Hub: Topic design, consumer groups, ETL integration
• Azure Streaming Analytics for real-time telemetry data
• Azure Key Vault, App Service, Container Registry (ACR)
• Azure IoT Hub for edge device integration
• Azure DevOps & GitHub Actions for CI/CD automation
• GitHub self-hosted runners for workflow management

Edge and On-Prem Integration:

• Edge VM deployment using Docker and Portainer
• Messaging via RabbitMQ (read/write from edge)
• OPC UA for PLC integration (e.g., FX Filter, NH3 Compressor)
• IDMZ architecture for secure edge-to-cloud integration

Machine Learning Platform & MLOps:

• End-to-end ML lifecycle: Feature engineering, model training, validation, deployment
• Monitoring of deployed models at high frequency (e.g., 1-minute intervals)
• Cloud vs edge deployment strategy, cadence management (weekly, monthly, quarterly)
• MLflow, ADB ML workspace, monorepo structures for model code

Data Architecture & Integration:

• Implementation of medallion architecture
• Integration with Unity Catalog for data governance and sharing
• Real-time SAP ingestion using CDC tools (e.g., Aecorsoft)
• Streaming and API-based data ingestion
• Template-driven ingestion and mapping using configurations
• Consumption layer design for ML, BI, and operational reporting

Governance & Data Modeling:

• Define and implement data governance policies
• Scalable data model design for operational analytics and ML feature generation
• Metadata tagging, access control, and quality enforcement across data layers