Putting machine learning into production requires transforming an experimental model into a service that is observable, reproducible, secure, and operable under failure conditions. The contractual SLA should cover not only availability and latency, but also data quality, predictive behavior, updates, rollback, and responsibilities shared between the provider and the client.
Why a Good Prototype Can Fail in Production
A notebook proves that a particular pattern can be learned from a data sample. By itself, it does not demonstrate that the solution will continue to work with real traffic, behavioral changes, unstable integrations, and cost constraints.
The main causes of failure are:
- Difference between training and production: fields, scales, categories, or data-entry rules change.
- Data drift: the input distribution moves away from the distribution used during training.
- Concept drift: the relationship between inputs and outcomes changes, even if the data appears similar.
- Training-serving skew: the transformation applied during training differs from the one performed during inference.
- External dependencies: APIs, databases, queues, or legacy systems become slow or unavailable.
- Lack of rollback: a new model replaces the previous one without a fast path for reversal.
- Inappropriate metric: the team improves average accuracy but ignores false negatives, calibration, or performance by segment.
- Lack of ownership: no one knows who should act when quality degrades outside business hours.
The transition to production is therefore a systems engineering problem. The model is only one of the components.
SLI, SLO, and SLA for Machine Learning Services
The three concepts should be separated in both the project and the contract:
- SLI, or Service Level Indicator: an observed measurement, such as monthly availability or p95 latency.
- SLO, or Service Level Objective: the internal target for the indicator.
- SLA, or Service Level Agreement: a contractual commitment, including scope, exclusions, support, and consequences of noncompliance.
An internal SLO should be stricter than the SLA. If the contract requires 99.5% availability, for example, operating with an identical internal target eliminates the margin required for maintenance and incidents.
Availability can be calculated as:
availability = valid requests served / valid requests received
For APIs, this definition is usually more useful than considering only the amount of time a process remained online. A service may be running and still return errors or unusable predictions.
Indicators That May Be Included in the Contract
| Dimension | Possible SLI | Criterion that must be defined |
|---|---|---|
| Availability | Percentage of valid requests served | Monthly window, regions, and exclusions |
| Latency | p95 or p99 response time | Endpoint, load, and input size |
| Errors | Rate of 5xx responses and timeouts | What counts as a provider error |
| Data | Percentage of valid records | Schema, required fields, and tolerances |
| Prediction | Precision, recall, F1, MAE, or calibration | Population, threshold, and evaluation window |
| Update | Time required to deploy or revert a model | Approval, testing, and data availability |
| Incident | Acknowledgment and recovery time | Severity and coverage hours |
Values such as 99.9% availability or p95 latency below 300 ms may serve as design examples, but they should not be copied without load testing and process analysis. The higher the requirement, the greater the expected cost of redundancy, on-call coverage, spare capacity, and observability.
From Notebook to Operational Service
The evolution should take place in stages with objective exit criteria.
1. Make the Experiment Reproducible
The first step is to remove critical rules from the notebook and convert them into version-controlled code. The package should record:
- code, data, and hyperparameter versions;
- environment and dependencies;
- seed and execution conditions;
- global metrics and metrics by relevant segment;
- resulting artifact and its signature;
- source and time period of the training dataset.
Rerunning the pipeline with the same inputs should produce an equivalent result within a documented tolerance.
2. Create Tests Beyond Accuracy
A continuous integration pipeline should validate the code, data, and model behavior. The minimum set includes:
- unit tests for transformations;
- validation of schemas and allowed ranges;
- compatibility testing between producer and consumer;
- comparison with a baseline model;
- performance testing by subgroup;
- verification of latency, memory, and artifact size;
- scanning of dependencies and secrets;
- load testing and external service failure testing.
The candidate model should advance only if it meets previously defined criteria. Approval based solely on “it looks better” is not auditable.
3. Separate Deployment from Release
Deploying an artifact does not mean immediately directing all traffic to it. Safe strategies include:
- shadow: the new model receives a copy of the traffic, but its response does not affect the user;
- canary: a small portion of requests uses the new version;
- blue-green: two complete environments enable fast switching and reversal;
- champion-challenger: the current model is continuously compared with candidates.
Rollback should be automated and triggerable by errors, latency, or metric degradation. If restoring the previous version requires manual reconstruction, the organization does not yet have a reliable release process.
Observability Across Four Layers
Monitoring CPU and memory does not reveal whether a prediction remains useful. An MLOps operation must observe four related layers.
Infrastructure and Application
Track availability, saturation, memory usage, queues, errors, timeouts, and p50, p95, and p99 latency. Logs should contain correlation identifiers, the model version, and the input validation result without exposing unnecessary personal data.
Data Quality
Monitor missing fields, previously unseen categories, cardinality, duplicates, delays, invalid ranges, and schema changes. Data contracts help prevent a silent change in a source system from reaching the model.
Statistical Behavior
Compare production and training distributions using metrics appropriate for each variable type. PSI, Jensen-Shannon distance, and Kolmogorov-Smirnov can indicate drift, but their thresholds must be calibrated for the context; a statistical alert does not prove a loss of quality.
Business Outcomes and Predictive Quality
When labels arrive after hours, days, or months, quality metrics will necessarily be delayed. The architecture should connect each prediction to the observed outcome to recalculate recall, precision, error, calibration, and process indicators.
In healthcare, fraud, and risk, the global average can conceal serious failures. Cohorts, time periods, operating units, and higher-impact classes must be evaluated while respecting privacy and governance.
Retraining Should Not Be Automatic by Default
Retraining weekly does not guarantee improvement. Recent data may be incomplete, contaminated by incidents, or reflect a temporary event.
A responsible trigger combines:
- a minimum volume of new labeled examples;
- a persistent signal of drift or degradation;
- validation against the baseline and current model;
- tests by segment and regression analysis;
- automated or human approval depending on the risk;
- gradual deployment with rollback.
Low-risk, high-volume use cases allow for broader automation. Clinical and financial decisions, or decisions with legal implications, generally require a stricter audit trail and human approval.
What Must Be Included in the Contractual SLA
An ML SLA should avoid vague expressions such as “high accuracy” or “fast response.” The document must specify:
- covered endpoints, environments, regions, and hours;
- the formula, source, and window for each indicator;
- expected volume, peaks, and request limits;
- schema versions and responsibilities regarding the data;
- predictive metric, evaluated population, and label arrival time;
- incident severity levels and communication channels;
- time to acknowledgment, mitigation, and recovery;
- maintenance, update, and deprecation policies;
- log retention, auditing, and personal data processing;
- RTO and RPO when stateful or batch processing is involved;
- exclusions, such as an external source failure or input outside the contract;
- credits, penalties, or other agreed-upon consequences.
Predictive quality should not be promised without control over the quality and representativeness of the data sent by the client. A practical solution is to separate the technical SLA, which is measurable in real time, from quality objectives evaluated in windows compatible with label availability.
Production Readiness Checklist
Before release, confirm:
- [ ] code, data, configuration, and artifact are versioned;
- [ ] a model registry with metadata and approval exists;
- [ ] training and inference share consistent transformations;
- [ ] schemas and data contracts are validated;
- [ ] functional, load, security, and regression tests have passed;
- [ ] dashboards cover the application, data, model, and business;
- [ ] alerts have a threshold, owner, and response procedure;
- [ ] canary, shadow, or an equivalent strategy has been defined;
- [ ] rollback has been performed in a test, not merely documented;
- [ ] RTO, RPO, SLI, SLO, and SLA have been approved;
- [ ] costs per thousand inferences or per batch are tracked;
- [ ] runbooks and incident escalation procedures are accessible.
How Predictor Solutions Solves This
Predictor Solutions structures applied AI projects with data engineering, APIs, cloud/DevOps, security, and observability, taking the model from experimentation to a measurable operation. In healthcare systems, it also works with HL7 v2 and FHIR integrations, a context in which data contracts, traceability, and fault tolerance are particularly relevant.
Execution begins with the definition of technical and business metrics, followed by reproducible pipelines, testing, artifact registration, gradual deployment, monitoring, and runbooks. The SLA is designed based on load, criticality, dependencies, and label availability instead of applying generic percentages.
The company, headquartered in Lavras, Minas Gerais, has served 9 medium-sized and large organizations. Across the projects reported by Predictor Solutions, the results include average savings of R$ 1.32 million per client per year, an average 70% increase in productivity, and 43% profit growth in six months; these figures cover its portfolio and do not replace the definition of specific indicators for each MLOps project.
Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246.