Memory hook: Healthy endpoint is not healthy prediction.
Must remember
- Monitor service latency, throughput, errors and resource saturation separately from prediction quality. An endpoint can return fast, valid JSON with poor business outcomes.
- Data drift changes input distribution; concept drift changes the relationship between features and target; training-serving skew comes from inconsistent feature preparation or availability.
- Collect appropriate inference data with privacy, sampling and retention controls. Delayed labels mean quality measurements can lag operational symptoms.
- Use thresholds and comparisons that account for seasonality and sample size. Investigate missing features, upstream schema changes and model version differences before retraining automatically.
- Configure alerts and retraining triggers with named owners and evaluation gates. A retrained model can fail acceptance and should not automatically replace the current champion.
- Measure subgroup quality and business error cost. Document intended use, limitations and the rollback or human-review path when confidence degrades.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Endpoint latency is normal but error outcomes rise | Investigate model quality, data changes and labels rather than only compute. |
| Input drift alert fires during a known seasonal event | Compare expected patterns and outcome evidence before promoting a new model. |
Traps
- Drift is not automatic proof that retraining will improve performance.
- More frequent retraining can amplify bad or mislabeled data.
Active recall
1. What is concept drift?
A change in the relationship between input features and the desired prediction.
2. Why monitor input schema?
Missing or changed features can break quality without causing an HTTP error.
3. What should a retraining trigger produce?
A candidate evaluated through the release process, not an unconditional deployment.
4. Why inspect subgroup metrics?
Aggregate quality can hide harmful performance gaps.
5. How respond to severe quality regression?
Use the defined rollback, restriction or human-review path while investigating evidence.