MLOps for GenAI: CI/CD and Drift Detection

MLOps for GenAI is the set of practices — versioning, automated testing, staged deployment, and statistical monitoring — that catch a degraded model or a silently-changed input distribution before users do, instead of after. It's the same discipline as regular DevOps applied to a system whose "correctness" is a statistical judgment call rather than a pass/fail test, which is exactly why generic CI/CD advice (run the tests, ship on green) doesn't fully transfer, and why "add monitoring" is usually said without anyone specifying which statistical test actually fires an alert.

Key facts

Google Cloud's own MLOps architecture guide defines three maturity levels — manual process, ML pipeline automation, and full CI/CD pipeline automation — and most teams running GenAI in production are still at level 0 or 1, not level 2.
The same guide breaks a level-2 pipeline into six concrete stages: development and experimentation, pipeline continuous integration, pipeline continuous delivery, automated triggering, model continuous delivery, and monitoring.
Evidently AI's open-source drift-detection library doesn't use one generic "drift score." It switches statistical test by data type and sample size: a Kolmogorov-Smirnov test (triggering below a 0.05 p-value) for numeric columns under 1,000 rows, a Wasserstein distance test (triggering above 0.1) for numeric columns over 1,000 rows, and a binary classifier's ROC AUC score (triggering above 0.55) for text data at any size.
A 2022 MLOps survey paper is explicit that these practices are a framework for reducing manual-process risk, not an empirically proven failure-rate reduction — a distinction worth keeping, since "MLOps prevents outages" overstates what the research actually shows.

The three maturity levels, and why "we have monitoring" doesn't mean level 2

Google Cloud's own continuous-delivery architecture for machine learning names three maturity levels, and the gap between them is bigger than the names suggest.

Level 0 is a manual process: a person runs data analysis, training and validation by hand, usually in a notebook, and deployment happens whenever someone decides to deploy. There's no CI/CD and no continuous training — a new model version ships because a person pushed it, not because a pipeline did.

Level 1 automates the ML pipeline itself — training re-runs automatically against fresh data on a trigger — but the pipeline's own code still deploys manually. This is the level most teams that say "we have an automated retraining job" are actually at.

Level 2 automates the pipeline's own build-test-deploy cycle, the same way application CI/CD does for regular code. A change to training code or configuration goes through the same build-and-test gate a product-code change would, before it's allowed anywhere near production.

flowchart TD A["Level 0 — manual: a person runs training and ships by hand"] --> B["Level 1 — pipeline automation: training re-runs on fresh data automatically"] B --> C["Level 2 — CI/CD automation: changes to the pipeline itself are built, tested, and deployed automatically"] style A fill:#e34948,stroke:#e34948,color:#fff style B fill:#eb6834,stroke:#eb6834,color:#fff style C fill:#1baf7a,stroke:#1baf7a,color:#fff

"We retrain automatically" is Level 1. Level 2 means the retraining pipeline's own code changes go through CI/CD too, not just the model.

The six pipeline stages, named

Google's guide doesn't stop at the maturity level — it names the stages a Level 2 pipeline actually runs: development and experimentation, pipeline continuous integration, pipeline continuous delivery, automated triggering, model continuous delivery, and monitoring. Each stage has a distinct job, and skipping straight from "experimentation" to "monitoring" — which is what a lot of GenAI teams do — leaves out the parts that catch a bad change before it reaches a user.

flowchart LR A["Dev & experimentation"] --> B["Pipeline CI"] B --> C["Pipeline CD"] C --> D["Automated triggering"] D --> E["Model CD"] E --> F["Monitoring"] F -.->|"drift or performance drop triggers"| D style A fill:#2a78d6,stroke:#2a78d6,color:#fff style B fill:#4a3aa7,stroke:#4a3aa7,color:#fff style C fill:#4a3aa7,stroke:#4a3aa7,color:#fff style D fill:#eb6834,stroke:#eb6834,color:#fff style E fill:#eb6834,stroke:#eb6834,color:#fff style F fill:#1baf7a,stroke:#1baf7a,color:#fff

Monitoring isn't the end of the line — it feeds back into automated triggering, which is the loop that makes continuous training continuous instead of a one-off.

Pipeline CI is testing the pipeline's own code — the data-validation step, the feature-engineering code — the same way you'd unit-test an application. Pipeline CD packages that tested pipeline so it can run on demand. Automated triggering is what actually kicks off a new training run: a schedule, a data-availability signal, or — the stage most directly relevant to GenAI — a monitoring alert. Model CD is the staged rollout of the newly-trained model itself, distinct from deploying the pipeline that produced it.

What "monitoring" actually checks: the statistical tests behind drift detection

"We monitor for drift" is a sentence that hides a specific decision: which statistical test, on which data, with what threshold. Evidently AI's open-source drift-detection library — a widely-used reference implementation, not Langoedge's own tooling — makes that decision explicit rather than hiding it behind a single score, and the actual rule set is a genuine worked example of what "drift detection" means in practice.

For a numeric input (say, average call length, or token count per request) with 1,000 or fewer observations, it runs a two-sample Kolmogorov-Smirnov test and flags drift when the p-value drops to 0.05 or below. Past 1,000 observations, the same numeric column switches to a Wasserstein distance test, flagging drift once the distance reaches 0.1. Categorical columns use a chi-squared test under the same size threshold, or Jensen-Shannon divergence above it, also at a 0.1 trigger. Text data — the input type most relevant to a GenAI system, where "has the distribution of prompts changed" is the real question — doesn't use any of those directly. Instead it trains a binary classifier to tell current inputs apart from a reference set, and treats a classifier that succeeds too well (ROC AUC above 0.55) as itself the evidence of drift: if a simple model can reliably distinguish today's inputs from last month's, something about the input distribution has moved.

flowchart TD A["What kind of data is being checked?"] --> B{"Numeric, under 1,000 rows"} A --> C{"Numeric, over 1,000 rows"} A --> D{"Text (e.g. prompts)"} B --> E["Kolmogorov-Smirnov test — drift if p ≤ 0.05"] C --> F["Wasserstein distance — drift if ≥ 0.1"] D --> G["Classifier trained to separate old vs. new inputs — drift if ROC AUC > 0.55"] E --> H["🚨 Trigger retraining pipeline"] F --> H G --> H style A fill:#2a78d6,stroke:#2a78d6,color:#fff style B fill:#4a3aa7,stroke:#4a3aa7,color:#fff style C fill:#4a3aa7,stroke:#4a3aa7,color:#fff style D fill:#4a3aa7,stroke:#4a3aa7,color:#fff style E fill:#eb6834,stroke:#eb6834,color:#fff style F fill:#eb6834,stroke:#eb6834,color:#fff style G fill:#eb6834,stroke:#eb6834,color:#fff style H fill:#e34948,stroke:#e34948,color:#fff

"Drift detection" isn't one test. It's a dispatch table — which test runs depends on the data type and sample size, and the threshold is a number someone has to actually pick.

A worked example makes the ROC AUC rule concrete: if you train a simple classifier to guess whether a given support ticket came from this week's traffic or three months ago, and it genuinely can't do better than a coin flip (ROC AUC near 0.5), nothing's changed. If it can tell them apart with 56% or better accuracy expressed as AUC, Evidently's default treats that as real, structural drift — not noise — because a classifier finding a reliable pattern means the two distributions are measurably different, not just randomly varied.

Where this fits next to Langoedge's own evaluation layer

CI/CD and drift detection are infrastructure-level checks — they watch the pipeline and the input/output distributions over time, independent of any single call. That's a different layer from the per-call evaluation Langoedge's own architecture already does: a self-reflecting agent's quality-gate checkpoint catches a bad output during one conversation, before the agent acts on it, and LLM-as-judge scoring grades a finished call after it ends. Neither of those two is a substitute for the other, and neither is a substitute for distribution-level drift monitoring: a self-reflecting agent can pass its own quality gate on every single call while the underlying distribution of what customers are calling about quietly shifts over a month, which is exactly the kind of slow, cross-call change only a drift test over time is positioned to catch.

Worth saying plainly: none of this is unique to Langoedge, and that's the point — it's the same CI/CD and statistical-monitoring discipline any team running models in production should have, with or without a voice AI platform involved. A 2022 MLOps survey paper is explicit that these practices are a risk-reduction framework, not a proven guarantee against a specific failure rate — a caveat worth keeping rather than quietly dropping in the name of a tidier pitch.

FAQ

What's the difference between MLOps Level 1 and Level 2?

Level 1 automates the model training pipeline itself — it retrains automatically when triggered by fresh data. Level 2 goes further and automates the build-test-deploy cycle for the pipeline's own code, the way application CI/CD does for a regular codebase, so a change to the training or data-validation logic is tested and deployed the same way a product-code change would be.

What statistical test should I use to detect drift in GenAI inputs?

It depends on the data type and volume, not one universal test. Evidently AI's reference implementation uses Kolmogorov-Smirnov for small numeric samples, Wasserstein distance for large numeric samples, chi-squared or Jensen-Shannon divergence for categorical data, and a trained classifier's ROC AUC score for text — treating a classifier that can reliably separate old from new inputs as evidence the distribution has shifted.

Does CI/CD for GenAI replace per-call evaluation like LLM-as-judge?

No — they check different things. CI/CD and drift detection watch the pipeline and the input/output distribution over time; LLM-as-judge and similar per-call evaluators grade one specific call or output. A system can pass every per-call check while the underlying traffic distribution drifts slowly enough that only a time-series drift test would notice.

Is MLOps overkill for a small GenAI deployment?

Full Level 2 automation is a real engineering investment, and a small team rarely needs all six pipeline stages on day one. But basic drift monitoring — even just tracking whether your input distribution looks statistically different month over month — is cheap relative to finding out a model degraded from a support ticket instead of a dashboard.

Does MLOps actually reduce production failures, or is that assumed?

It's reasoned about more than it's proven. The MLOps survey research cited below frames CI/CD, versioning and monitoring as a response to a real, named problem — manual ML workflows causing operational issues — by analogy to DevOps's track record with regular software, rather than as a practice with its own measured failure-rate reduction for ML systems specifically. Treat it as sound engineering discipline, not a statistically proven guarantee.

Sources

About the author. Arnab Chakraborty is the founder of Langoedge, a platform for building production voice and text AI agents with LangGraph. This post was substantially rewritten on 2026-10-04 as part of a weekly content audit — the original December 2025 version used a fictional engineering-team crisis narrative with no real citations, a near-identical template to another post in this archive.