Putting machine learning into production requires transforming an experimental artifact into a reproducible, monitored, secure service with measurable operational behavior. The contractual SLA should cover controllable indicators—such as availability, latency, recovery time, and update time—while predictive quality must be addressed through metrics, tolerance ranges, and explicit rules regarding data and context changes.
Why a Good Model Is Still Not a Product
A notebook can demonstrate that a predictive signal exists, but it does not answer the main production questions: where the data comes from, which version is active, how to detect degradation, how to undo a deployment, and who takes action when something fails.
Moving from a prototype to an operational service involves at least six components:
- Data pipeline: collection, validation, transformation, and availability of variables.
- Training pipeline: reproducible code to train, evaluate, and register models.
- Serving: API, batch processing, streaming, or embedded execution.
- Observability: technical, data, model, and business metrics.
- Governance: versioning, approval, traceability, and access control.
- Operations: alerts, on-call coverage, runbooks, rollback, and incident response.
MLOps is the discipline that integrates these parts. It is not limited to automating deployment: it applies software engineering, data, and operations practices to the model lifecycle.
Define the Operational Contract Before the Architecture
The first step should not be choosing Kubernetes, a feature store, or an experimentation platform. Before that, it is necessary to document how the prediction will be consumed and what impact a failure will have.
Questions That Change the Solution
- Will inference be synchronous, batch-based, or event-driven?
- What is the average volume, and what is the expected peak?
- Can the consumer wait seconds, minutes, or hours?
- Is there a deterministic fallback rule?
- How long can the system remain unavailable?
- When will the actual value used as the label become available?
- Does an incorrect prediction result in a recommendation, automatic blocking, or a clinical decision?
- What personal or sensitive data will be processed?
A product recommendation tolerates different conditions than a hospital system. In critical contexts, the model should not be the sole decision-making mechanism without risk validation, appropriate oversight, and contingency procedures.
Minimum Architecture for Machine Learning in Production
There is no universal stack, but there is a minimum set of responsibilities that must be covered.
Data and Features
Inputs must have a contract defining schema, type, unit, acceptable domain, missing-value policy, and the party responsible for the source. Validations must occur before training and inference.
Examples of failures that should block or flag the pipeline include:
- missing required column;
- negative age or age outside the defined domain;
- unit changed from milligrams to grams;
- unsupported new category;
- abnormal increase or decrease in volume;
- delay beyond the update window.
Training and inference must also share the same transformation logic. Duplicating code between the notebook and the API creates training-serving skew, a situation in which the model receives production data that differs from the data used during validation.
Training and Registry
Each run must record, at a minimum:
- code and data versions;
- hyperparameters;
- environment and dependencies;
- overall metrics and metrics for relevant segments;
- produced artifact;
- date, author, or responsible process;
- approval criteria.
The model registry must distinguish candidates, approved versions, production versions, and archived versions. Promoting a model means approving an immutable artifact, not rerunning a notebook and expecting the same result.
Serving and Contingency
The model can be exposed through an API, queue, stream, or scheduled batch. The choice should follow the process requirements, not the most sophisticated tool.
Every deployment should provide a safe fallback: return the last valid prediction, use a deterministic rule, forward the case for human review, or suspend automation. The contingency behavior must be included in the design and, when applicable, in the contract.
CI, CD, and CT in the MLOps Lifecycle
In machine learning, automation has three dimensions:
- CI—continuous integration: tests code, transformations, data contracts, security, and compatibility.
- CD—continuous delivery or deployment: packages, approves, and publishes the service through a controlled rollout.
- CT—continuous training: starts new training according to a schedule, new data, or degradation triggers.
CT does not mean automatically promoting the latest model. A candidate must pass minimum tests for:
- data quality;
- training reproducibility;
- comparison with the active model;
- metrics for critical segments;
- latency and resource consumption;
- dependency vulnerabilities;
- input and output compatibility;
- human validation when required by the level of risk.
For deployment, strategies such as shadow, canary, and blue-green reduce risk. In shadow deployment, the new model receives traffic without affecting responses. In canary deployment, it serves a limited share. In blue-green deployment, two environments enable fast switching and rollback.
What Can Actually Be Included in an SLA
An SLA is the contractual commitment; an SLO is the operational objective; an SLI is the measured indicator. A useful SLA specifies the definition, calculation window, measurement source, exclusions, responsibilities, and consequences of noncompliance.
| Dimension | SLI example | Contractual consideration |
|---|---|---|
| Availability | valid requests served / total requests | Define error codes and excluded maintenance periods |
| Latency | 95th or 99th percentile | Avoid using only the average |
| Recovery | time until service is restored | Associate it with incident severity levels |
| Update | time between data availability and processing | Also depends on the data source |
| Freshness | maximum age of features | Define the reference clock and time zone |
| Technical quality | valid response rate | Separate service failure from predictive error |
Values such as 99.9% availability or p95 latency below 300 ms can serve as design examples, but they should only become commitments after load testing, dependency analysis, and cost calculation. Stricter targets require compatible redundancy, spare capacity, observability, and operations.
Should Accuracy Be Included in the SLA?
Predictive quality depends on the data distribution, label delay, and factors that may be outside the provider's control. Therefore, promising fixed accuracy without defining the population, period, minimum sample size, and measurement method creates an ambiguous contract.
A more verifiable approach is to define:
- a metric appropriate to the problem, such as precision, recall, F1, MAE, or AUC;
- evaluation cohort and window;
- minimum sample size;
- maximum delay for obtaining the label;
- tolerance range relative to the approved baseline;
- procedure in the event of drift;
- deadline for investigation or retraining.
For imbalanced classes, accuracy can hide serious failures. For rare-event detection, recall, precision, the precision-recall curve, and the cost of false positives or false negatives are usually more informative.
Observability: Infrastructure, Data, Model, and Business
Monitoring only CPU and memory confirms that the process is active, not that the prediction remains useful. Observability must operate across four layers.
Technical Layer
Monitor availability, errors, timeouts, latency by percentile, throughput, saturation, queues, and cost per inference. Logs must include a correlation identifier, model version, and input contract version while respecting personal data minimization.
Data and Model Layer
Monitor missing values, schema changes, new categories, feature distributions, prediction distributions, and divergence between training and production. Drift is a signal for investigation, not automatic proof of quality loss.
When labels arrive, calculate actual performance by period and relevant segments. The alert should account for a minimum volume to avoid decisions based on small samples.
Business Layer
The model metric must be connected to the process: conversion, time saved, human review rate, avoided waste, or another verifiable outcome. Correlation does not demonstrate causality; controlled tests or quasi-experimental designs may be necessary to attribute impact.
Checklist for Reaching a Contractual SLA
Before going into production, confirm that:
- [ ] the problem, user, and supported decision are documented;
- [ ] the technical and business baseline has been defined;
- [ ] the data has a contract, validation, and responsible party;
- [ ] training is reproducible and versioned;
- [ ] the model and dependencies are immutable artifacts;
- [ ] tests cover code, data, security, and performance;
- [ ] deployment uses canary, shadow, or an equivalent mechanism;
- [ ] rollback has been tested, not merely documented;
- [ ] metrics have a source, formula, and calculation window;
- [ ] alerts point to runbooks and responsible parties;
- [ ] contingency operation without the model has been validated;
- [ ] data retention, access, and processing are defined;
- [ ] the SLA separates availability from predictive quality;
- [ ] responsibilities for external sources are explicit;
- [ ] operating costs have been tested under a representative load.
How Predictor Solutions Addresses This
Predictor Solutions, a software house based in Lavras, Minas Gerais, structures applied artificial intelligence projects by combining data engineering, software development, cloud/DevOps, and security. The work starts with the business process and measurable operational criteria to define pipelines, APIs, observability, versioning, controlled deployment, and indicators that can support SLOs and SLAs.
In healthcare, the company develops Predictor Health, focused on dashboards and wearables, and Predictor AI Hospitals, aimed at predicting sepsis, heart attacks, and pneumonia in ICUs. These scenarios require special attention to integration—including HL7 v2 and FHIR—traceability, data quality, access control, and operational contingency.
Across its software and automation projects, Predictor Solutions reports serving 9 medium-sized and large companies, average savings of R$ 1.32 million per client per year, an average productivity increase of 70%, and profit growth of 43% in six months. These results should be interpreted within the context of each project; a new system must establish its own baseline and measurement method.
Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246