Putting machine learning into production requires more than publishing an API: operational and statistical indicators must be defined, data, tests, and deployments must be automated, degradation must be monitored, and response responsibilities must be established. A sustainable contractual SLA must be derived from measurable SLIs and realistic SLOs, including availability, latency, data quality, model updates, and incident recovery.
Why an ML prototype is not yet a product
A notebook demonstrates that a hypothesis can work on a dataset. A production system must continue working with new data, infrastructure failures, behavioral changes, traffic spikes, and audit requirements.
The difference appears across five dimensions:
- Reproducibility: code, data, parameters, and environment must be versioned.
- Reliability: inference must meet availability and latency targets.
- Quality: the model must maintain acceptable performance after deployment.
- Operability: the team must detect, diagnose, and resolve incidents.
- Governance: decisions, versions, access, and changes must be traceable.
A prototype with good accuracy may fail in production because a column changed type, an unseen category appeared, user behavior changed, or a dependency was updated. MLOps treats these conditions as engineering problems rather than exceptions resolved manually.
What MLOps needs to cover
MLOps combines machine learning, data engineering, DevOps, and governance practices. Its scope must cover the entire lifecycle:
- data collection, validation, and versioning;
- feature creation and reuse;
- reproducible training;
- technical and business evaluation;
- model registration and approval;
- controlled deployment;
- infrastructure, data, and prediction monitoring;
- retraining or rollback;
- auditing and version decommissioning.
Not every project needs to start with Kubernetes, a feature store, and complex pipelines. For low volumes, a container, a simple orchestrator, an artifact registry, and appropriate observability may be sufficient. The architecture must address risk and scale, not tool popularity.
Step 1: Transform the experiment into a reproducible artifact
The first step is to move everything that must operate predictably out of the notebook. Preparation, training, and inference must be separated into testable components.
Minimum reproducibility requirements
- Git repository with code review;
- environment declared through a dependency file and container image;
- recorded random seeds when applicable;
- parameters and metrics associated with each run;
- version or snapshot of the training data;
- serialization of the complete pipeline, including preprocessing;
- model input and output signature;
- record of the code version that generated the artifact.
A model version is not just a binary file. It must connect code + data + configuration + environment + metrics. Without this association, investigating a regression or reproducing a decision becomes expensive and, in some cases, impossible.
Unit tests must cover critical transformations. Integration tests must verify source reading, schema compatibility, and result persistence. Regression tests compare a candidate version with the current model using the same reference dataset.
Step 2: Define the data contract
A large share of ML incidents originates before the model. A data contract explicitly defines what the pipeline accepts and must include:
- field names, types, and required status;
- valid ranges and units of measurement;
- allowed categories and handling of unknown values;
- policy for null or delayed fields;
- expected update frequency and window;
- source, owner, and sensitivity classification;
- behavior when the contract is violated.
Validation must occur at both training input and inference. Depending on the risk, an invalid record may be rejected, quarantined, or processed through a fallback rule. Silently correcting all data often hides systemic problems.
In regulated sectors or environments involving personal data, data minimization, access control, encryption, defined retention, and audit trails must also be implemented. The pipeline must not carry attributes that are unnecessary for its purpose.
Step 3: Choose the inference pattern
How the model is served determines a significant portion of the cost and SLA.
Batch inference
It is appropriate when predictions can be calculated at intervals, such as daily segmentation or demand forecasting. It simplifies scalability and reduces latency requirements, but produces less up-to-date results.
Online inference
A synchronous API supports decisions within seconds or milliseconds. It requires concurrency control, timeouts, caching, autoscaling, and dependency isolation. The SLA must consider the entire request path, not only the model’s internal processing time.
Asynchronous or event-driven inference
Queues decouple producers and consumers, absorb spikes, and enable retries. In exchange, they add delay, duplication risk, and the need for idempotency. This pattern is appropriate when the result does not need to return over the same connection.
The criterion must combine tolerable latency, volume, cost, criticality, and freshness. There is no technical advantage in providing a response within 100 ms when the business process can accept ten minutes.
Step 4: Build CI/CD/CT for models
Traditional CI/CD validates and delivers software. In ML, the pipeline must also evaluate statistical artifacts. The term CT, continuous training, describes automated retraining or retraining triggered by defined criteria.
A secure workflow may follow these steps:
- validate code, schema, and dependencies;
- run unit, integration, and security tests;
- train in a controlled environment;
- compare metrics with the current model and a simple baseline;
- verify criteria by relevant segment, not only the global average;
- register the approved model;
- deploy in shadow, canary, or staging mode;
- promote gradually after observing metrics;
- keep the previous version ready for rollback.
Automated retraining does not mean automated promotion. In high-impact applications, requiring human approval and documented evidence is reasonable. Triggers may be time-based, based on accumulated volume, distribution changes, or performance degradation after ground-truth labels become available.
From SLI to contractual SLA
An SLI is an observed measurement, such as the proportion of successful requests. An SLO is the internal target for that measurement. An SLA is the contractual commitment, including scope, exclusions, calculation method, and consequences of noncompliance.
The internal SLO must be stricter than the SLA, preserving a margin for incidents. Promising exactly the best historical result eliminates that margin.
Indicators that may be included in the SLA
- Availability: percentage of valid requests served within a monthly window.
- Latency: p95 or p99 percentiles, not only the average.
- Error rate: 5xx responses, timeouts, or unprocessed messages.
- Freshness: maximum time since the last successful data update.
- RTO: timeframe for restoring the service after an outage.
- RPO: maximum time-based volume of data that may be lost.
- Response time: timeframe for acknowledging and beginning incident handling by severity.
Metrics such as precision, recall, and mean error require contractual care. Ground-truth labels may arrive weeks later, depend on the client, or undergo concept changes. In these cases, the contract must define the evaluated population, window, label source, minimum sample size, and handling of incomplete data.
It is not technically correct to guarantee “95% accuracy” without these conditions. In imbalanced classification, 95% may even represent a useless model. Depending on the case, recall, precision, F1, AUROC, AUPRC, calibration, or cost per error are more appropriate measures.
Example of a measurable specification
Instead of stating that “the API will be fast and accurate,” a specification may establish:
- monthly availability calculated over valid requests;
- p95 latency measured at the gateway for an agreed load limit;
- documented exclusion of maintenance windows;
- acknowledgment timeframe by incident class;
- monthly recall evaluation after labels become available;
- fallback procedure if performance falls below the threshold;
- supplier and client responsibilities for data and integrations.
Numerical values must come from load testing, assisted operation, and risk analysis. Copying targets from another system creates a commitment without evidence.
Observability: Infrastructure, data, and model
Monitoring only CPU and memory does not indicate whether predictions remain useful. Observability must have three layers.
Service
This includes availability, percentile-based latency, errors, saturation, queues, resource consumption, and cost per inference. Logs must contain correlation identifiers while avoiding unnecessary exposure of personal data.
Data
Schema, null values, ranges, categories, volume, freshness, and distribution changes must be observed. Drift is an alert for investigation, not automatic proof of quality degradation.
Model
When labels arrive, global performance, performance across relevant segments, calibration, and business impact must be calculated. The distribution of predictions, abstention rate, and fallback usage should also be monitored.
Every alert must have an owner, severity, channel, runbook, and closing condition. Alerts without a defined action only generate operational fatigue.
Strategies to reduce deployment risk
Four patterns are especially useful:
- Shadow: the new version receives a copy of the traffic but does not control the response.
- Canary: a small share of requests uses the candidate model.
- Blue-green: two complete environments allow rapid switching.
- Champion-challenger: the current model is continuously compared with candidates.
Rollback must include the model, image, configuration, and transformations. Reverting only the model file may preserve an incompatibility introduced during preprocessing.
A fallback is also necessary: the previous model, a deterministic rule, manual processing, or a controlled unavailability response. The choice depends on the potential harm of an incorrect response versus no response.
Checklist before signing the SLA
- Are SLIs calculated automatically and auditable?
- Do data sources and data quality have clearly defined owners?
- Have load tests been performed at the contracted volume and traffic profile?
- Do statistical metrics have a specified population, window, and label?
- Are external dependencies and exclusions explicit?
- Are there runbooks, escalation procedures, and on-call coverage compatible with the promised timeframe?
- Has rollback been tested rather than merely documented?
- Have RTO, RPO, backup, and restoration been validated?
- Has the cost of meeting the SLA been included in the architecture and contract?
- Is there a procedure for drift, data failures, and performance below the threshold?
If any critical answer is “no,” the system is probably still operating experimentally, even if it is already receiving real traffic.
How Predictor Solutions solves this
Predictor Solutions structures machine learning projects by integrating data engineering, applied AI, cloud/DevOps, and security. The work begins with the data contract and business criteria, continues through reproducible pipelines, testing, gradual deployment, and observability, and concludes with SLIs and SLOs capable of supporting a contractual SLA.
This practice also applies to sensitive contexts, such as Predictor Health and Predictor AI Hospitals, which focuses on predicting sepsis, heart attacks, and pneumonia in ICUs. In these scenarios, traceability, data integration, monitoring, and fallback are engineering requirements, not optional additions. The company, based in Lavras, Minas Gerais, has served 9 medium-sized and large organizations, with reported average results of R$ 1.32 million in savings per client per year, a 70% increase in productivity, and 43% profit growth within six months.
Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246.