Retraining triggers, evaluation gates and rollback

A monitoring event can open an investigation or start candidate training. Promotion changes user-facing behavior and needs its own evaluation decision. A system that replaces the model whenever a distribution threshold fires can repeatedly deploy worse models trained on a broken feed.

Choose the trigger from the reason to update

Continuous training automates this workflow. Online learning updates parameters incrementally as observations arrive; it has additional concerns such as poisoned feedback, ordering and fast rollback. A recurring batch retrain does not become online learning because its scheduler runs frequently.

A guarded policy with concrete numbers

Illustrative forecast policy: outcomes become mature after 24 hours. The reference mean absolute error (MAE) is 20 units for matched weekday/hour cohorts. Consider a retraining candidate when MAE exceeds 24 units, a 20% relative increase, in three consecutive non-overlapping daily windows. Each window must contain at least 2,000 mature predictions with adequate coverage of the affected segments.

These values are example policy parameters, not recommended universal thresholds. Calibrate the window, sample size and effect against ordinary variability, business loss and training cost. Overlapping windows are correlated, so three alerts from nearly identical data provide less evidence than three independent periods.

Make the training workflow resumable

Persist the trigger ID, reference/current window IDs, metric definition, diagnosis, input snapshot, training configuration and candidate status. Deduplicate the trigger; use a lease or equivalent coordination to prevent competing trainers. An illustrative seven-day cooldown limits repeated training, while a severe verified incident can follow a separately authorized override procedure.

A candidate moves through eligibility checks, snapshot construction, training, evaluation and either rejection or release. Failed tasks resume from durable artifacts where valid. Exhausted retries produce a visible terminal failure and an owner; the next monitoring tick should not silently create a new unbounded loop.

Snapshot both features and labels with their availability times. A rolling training window adapts faster but can forget rare regimes; an expanding window retains history but may overweight obsolete behavior. Compare the choice through time-appropriate validation rather than assuming that more recent data is always better.

Compare the candidate with the incumbent

Evaluate both on the same untouched, time-appropriate holdout and important slices. Gate predictive loss, calibration, mistake cost, fairness where relevant, input compatibility, inference latency, memory and startup behavior. Preserve a regression set for rare but costly cases. Repeatedly tuning to the same holdout eventually contaminates the release decision; refresh evaluation governance as the development process evolves.

A registry records the candidate artifact and evidence. Resolve its immutable version into a deployment release, run shadow or canary evaluation, and retain rollback state. Moving an MLflow alias changes a registry pointer; running servers need an explicit reload or deployment mechanism before their loaded model changes.

Rollback restores the compatible model, preprocessing, threshold and feature contract. Reverting only weights can leave a new encoder or schema paired with an old model. Delayed labels also mean that an operationally healthy canary cannot immediately prove long-term predictive benefit.

Select the smallest effective remedy

For AI, the versioned release includes prompts, retrieval configuration, tools, adapters and provider settings. Fine-tuning a generator is one possible response after isolating a learned-behavior gap. AI release evaluation owns those stage-specific quality gates; drift diagnosis owns the statistical distinctions.