CI/CD for ML

CI/CD for ML extends traditional continuous-integration and continuous-delivery practice to the machine-learning lifecycle: on every change to training code, training data, or feature definitions, a pipeline automatically runs tests, retrains affected models, evaluates them against a held-out baseline, and (optionally) promotes them through staging into production. Unlike pure software CI/CD — where a test suite runs in minutes and the build is deterministic — ML pipelines are slow (training takes hours to weeks), expensive (GPUs cost real money), and stochastic (two runs with the same inputs produce different models). The discipline adapts by running cheap checks on every change (smoke tests, data validation, small-scale sanity training), expensive pipelines on a cadence or on meaningful changes, and promotion gates on explicit eval thresholds.

The 2026 tooling layers on top of general CI/CD. **GitHub Actions, GitLab CI, Jenkins, CircleCI** handle the code-change triggers and pipeline orchestration. **Vertex AI Pipelines, SageMaker Pipelines, Azure ML Pipelines, Databricks Workflows** provide ML-aware pipeline primitives (step reuse, artifact caching, GPU scheduling). **Kubeflow Pipelines** and **Argo Workflows** offer the Kubernetes-native alternative. **DVC** and **Pachyderm** handle data versioning so data changes can trigger pipelines. **MLflow Projects** and **Metaflow** (Netflix) wrap ML code into reproducible runnable units. Model-change deployments typically integrate with the model registry for gated promotion.

For APAC mid-market teams, the right progression is **CI before CD**. Start with cheap CI — on every pull request: run unit tests, validate data schema, do a small-scale training smoke test (1-2% data, 5 minutes of compute), run evaluation on the existing test set. This catches 80% of breakage without requiring the full ML infrastructure. Add CD (automated promotion through staging to production) only when models are stable enough that you trust the eval thresholds. Many teams never need full CD — periodic manual retraining with reviewed promotion works adequately up to ~20 production models.

The non-obvious failure mode is **treating ML CI like code CI**. A team wires 'run full training on every PR' into their pipeline, hits GPU quota, blows through their cloud budget in a week, and disables the automation rather than rethinking it. ML CI has to be tiered — cheap per-PR checks, expensive per-branch or per-merge training, full retrains on schedule — and compute budgets need to be explicit. Blind automation of an expensive, slow process creates more friction than it removes. Start cheap, add expensive pipelines deliberately, and always gate expensive steps on cheap ones passing first.