QA Engineer of ML/Data systems & python automation

About the Role:

The QA Engineer will own quality assurance across the ML lifecyclefrom raw data validation through feature engineering checks, model training/evaluation verification, batch prediction/optimization validation, and end-to-end (E2E) workflow testing. The role is hands-on with Python automation, data profiling, and pipeline test harnesses in Azure ML and Azure DevOps. Success means provably correct data, models, and outputs at production scale and cadence.

Key Responsibilities:

Test Strategy & Governance

Define an ML-specific Test Strategy covering data quality KPIs, feature consistency
checks, model acceptance gates (metrics + guardrails), and E2E run acceptance (timeliness, completeness, integrity).

Establish versioned test datasets & golden baselines for repeatable regression of
features, models, and optimizers.

Data Quality & Transformation

Validate raw data extracts and landed datalake data: schema/contract checks, null/outlier thresholds, time-window completeness, duplicate detection, site/material coverage.

Validate transformed/feature datasets: deterministic feature generation, leakage detection, drift vs. historical distributions, feature parity across runs (hash or statistical similarity tests).

Implement automated data quality checks (e.g., Great Expectations/pytest + Pandas/SQL) executed in CI and AML pipelines.

Model Training & Evaluation

Verify training inputs (splits, windowing, target leakage prevention) and
hyperparameter configs per site/cluster.

Automate metric verification (e.g., MAPE/MAE/RMSE, uplift vs. last model, stability
tests) with acceptance thresholds and champion/challenger logic.

Validate feature importance stability and sensitivity/elasticity sanity checks (price-
volume monotonicity where applicable).

Gate model registration/promotion in AML based on signed test artifacts and
reproducible metrics.

Predictions, Optimization & Guardrails

Validate batch predictions: result shapes, coverage, latency, and failure handling.

Test model optimization outputs and enforced guardrails: detect violations and prove idempotent writes to DB.

Verify API push to third party system (idempotency keys, retry/backoff, delivery receipts).

Pipelines & E2E

Build pipeline test harnesses for AML pipelines (data-gen nightly, training weekly,
prediction/optimization) including orchestrated synthetic runs and fault injection (missing slice, late competitor data, SB backlog).

Run E2E tests from raw data store -> ADLS -> AML -> RDBMS -> APIM/Frontend; assert
freshness SLOs and audit event completeness (Event Hubs -> ADLS immutable).

Automation & Tooling

Develop Python-based automated tests (pytest) for data checks, model metrics, and API contracts; integrate with Azure DevOps (pipelines, badges, gates).

Implement data-driven test runners (parameterized by site/material/model-version) and store signed test artifacts alongside models in AML Registry.

Create synthetic test data generators and golden fixtures to cover edge cases (price gaps, competitor shocks, cold starts).

Reporting & Quality Ops

Publish weekly test reports and go/no-go recommendations for promotions; maintain a
defect taxonomy (data vs. model vs. serving vs. optimization).

Required Qualifications

5 to 7+ years in QA with 3+ years focused on ML/Data systems (data pipelines + model validation).

Python automation (pytest, pandas, NumPy), SQL (PostgreSQL/Snowflake), and CI/CD (Azure
DevOps) for fully automated ML QA.

Strong grasp of ML validation: leakage checks, proper splits, metric selection
(MAE/MAPE/RMSE), drift detection, sensitivity/elasticity sanity checks.

Experience testing AML pipelines (pipelines/jobs/components), and message-driven integrations
(Service Bus/Event Hubs).

API test skills (FastAPI/OpenAPI, contract tests, Postman/pytest-httpx) + idempotency and retry
patterns.

Familiar with feature stores/feature engineering concepts and reproducibility.

Solid understanding of observability (App Insights/Log Analytics) and auditability requirements.

Education:

Bachelors or Masters degree in Computer Science, Information Technology, or related field.

Certification in Azure Data or ML Engineer Associate is a plus.