Putting machine learning into production requires transforming an experimental model into a reproducible, observable, secure, and operable service governed by measurable targets. The path to a contractual SLA involves defining technical and business SLOs, automating data and models, controlling versions, monitoring degradation, and establishing responsibilities for incidents.
Why a Good Prototype Is Still Not a Product
In a prototype, the goal is usually to prove that the data contains a predictive signal. In production, the goal changes: continuously delivering reliable predictions within a real-world process while keeping impact under control.
A notebook with good accuracy does not answer essential operational questions:
- Which version of the data generated the model?
- Can the training process be reproduced?
- What latency can the system support?
- What happens when a variable is missing?
- How can data drift and concept drift be identified?
- Is it possible to revert to the previous version?
- Who responds when the service is unavailable?
- How should results be measured when the ground-truth label arrives weeks later?
MLOps is the set of practices that addresses these questions through software engineering, data engineering, automation, observability, and governance. It is not merely a deployment tool or a training pipeline.
The Minimum Architecture for ML in Production
An architecture does not need to be complex from the start, but it must separate responsibilities. The minimum components are:
- Data ingestion and validation: receives data and verifies types, ranges, null values, duplicates, and integrity.
- Feature processing: applies consistent transformations during training and inference.
- Reproducible training: records code, configuration, data, dependencies, metrics, and artifacts.
- Model registry: identifies versions, approval stages, and promotion history.
- Inference service: runs predictions online, in batches, or through streaming.
- Monitoring: tracks infrastructure, data quality, model behavior, and business impact.
- Feedback loop: connects predictions to actual outcomes when they become available.
The choice between online and batch inference changes the entire design. A recommendation that must appear during a session may require an online API. A daily demand forecast can be processed in batches, with lower costs and fewer failure points.
Online, Batch, or Streaming
Use online when a response must be calculated on demand. Use batch when predictions can be precomputed at defined intervals. Use streaming when continuous events require near-real-time processing.
Do not adopt low latency as an automatic requirement. Reducing latency increases costs, operational complexity, and risk. The requirement should come from the business process, not from the capabilities of the technology.
The Path from Prototype to SLA
1. Define the Data Contract
Before automating the model, document the contract between data producers and consumers. It should specify:
- field names and types;
- units and formats;
- nullability rules;
- keys and cardinality;
- update frequency and delay;
- handling of sensitive information;
- expected behavior in the event of missing or corrupted data.
A silent change in unit, category, or semantics can keep the API available while invalidating its predictions. Therefore, technical availability is not equivalent to ML service quality.
2. Make Training Reproducible
At a minimum, each run should record:
- code version;
- dataset reference;
- parameters and hyperparameters;
- environment and dependencies;
- metrics by relevant segment;
- final artifact;
- person responsible and reason for the run.
Seeds help, but they do not guarantee determinism across all libraries and infrastructures. The practical goal is to be able to explain and repeat the process within known tolerances.
3. Create Promotion Gates
Models should not reach production simply because they exceeded an average metric. Promotion must go through automated criteria and a review proportional to the risk.
An approval checklist may require:
- performance above the current baseline;
- no critical regression by segment;
- validation of latency and resource consumption;
- data leakage analysis;
- API contract tests;
- security and dependency verification;
- rollback plan;
- human approval for high-impact decisions.
Minimum thresholds should be defined based on the cost of errors. In fraud detection, false negatives and false positives have different impacts. In healthcare, a prediction should support clinical workflows and governance, not automatically replace professional judgment.
4. Deploy Progressively
Deploying directly to the entire user base increases the impact of failures. Safer strategies include:
- shadow: the new model receives traffic without influencing decisions;
- canary: only a controlled portion uses the new version;
- blue-green: old and new versions coexist for fast switching and rollback;
- A/B: versions are compared across defined groups when the experimental design is valid.
Rollback must include the model, preprocessing code, configuration, and feature schema. Reverting only the model file may create incompatibility with the rest of the pipeline.
SLI, SLO, and SLA Applied to Machine Learning
These terms are not synonymous:
- SLI: an observed indicator, such as availability, latency, or percentage of valid inputs.
- SLO: an internal target for an SLI over a defined window.
- SLA: a contractual commitment, usually accompanied by scope, exclusions, responsibilities, and consequences.
An ML SLA should not be limited to API uptime. An API may respond normally while returning unusable predictions. The contract must distinguish at least four dimensions.
Availability and Operational Performance
Possible indicators include:
- percentage of requests served;
- latency by percentile, not only the average;
- error rate by endpoint;
- batch processing time;
- maximum data delay;
- capacity under expected load.
Contractual figures should reflect the architecture, budget, and impact of unavailability. The stricter the SLA, the greater the likely need for redundancy, on-call support, recovery testing, and reserved capacity.
Data Quality
Monitor completeness, validity, distribution, and freshness. Examples of SLIs include the percentage of accepted records, frequency of missing fields, and time since the last valid data load.
It is important to distinguish failures by the model provider from failures in sources controlled by the client or third parties. This boundary must be stated explicitly in the contract.
Predictive Quality
Metrics such as precision, recall, F1, AUC, absolute error, and calibration depend on the problem. However, predictive metrics cannot always be included in a real-time SLA because the ground-truth label may arrive long after the prediction.
In this case, the contract may combine:
- immediate operational metrics;
- drift indicators as alerts, not as isolated proof of degradation;
- periodic evaluation using labeled data;
- a revalidation and retraining procedure;
- rules for suspending or replacing the model.
Do not promise fixed accuracy when the data distribution is beyond the parties’ control. Instead, define the measurement method, evaluated population, time window, minimum sample size, and handling of exceptions.
Business Outcomes
Conversion, savings, loss reduction, and productivity depend on both the model and the process in which it operates. To include them in a contract, it is necessary to establish a baseline, attribution method, observation window, and external variables. Without these elements, the indicator mixes the model’s effect with pricing, operations, seasonality, and human decisions.
Monitoring: Infrastructure, Data, Model, and Business
Observability should connect four layers:
- Infrastructure: CPU, memory, queues, errors, availability, and latency.
- Data: schema, null values, new categories, delays, and distribution shifts.
- Model: prediction distribution, confidence, calibration, and performance when labels are available.
- Business: adoption, changed decisions, cost of errors, and operational outcomes.
Drift does not automatically mean that the model has deteriorated. Data drift indicates a change in inputs; concept drift represents a change in the relationship between inputs and outcomes. Both require investigation, but automatic retraining without validation can incorporate bad data and degrade the system.
Alerts also need runbooks. Each useful alert should report severity, likely hypothesis, responsible party, evidence, and recommended action. Alerting on everything causes fatigue; alerting only on unavailability leaves silent degradation without a response.
What Must Be Included in the Contractual SLA
Before signing, validate this checklist:
- covered services, endpoints, and environments;
- measurement window and official source of indicators;
- availability, latency, and processing SLOs;
- data quality and delay criteria;
- method and frequency of predictive evaluation;
- external dependencies and justified exclusions;
- severity levels and response times;
- communication and escalation process;
- RTO and RPO when disaster recovery applies;
- update, rollback, and deprecation policy;
- responsibilities for data, access, and security;
- retention of logs and evidence;
- consequences of noncompliance;
- SLA review procedure.
Use error budgets to balance reliability and evolution. If the failure budget for a window is close to being exhausted, the priority shifts from new features to stabilization. This mechanism turns reliability into an operational decision, not merely a report.
Build, Buy, or Hire a Specialized Team
Building internally provides greater control, but it requires expertise in data, software, cloud, security, and operations. Managed platforms reduce infrastructure work, but they may increase technology dependency and costs at scale. A specialized team is useful when a project needs to accelerate the transition from experiment to an auditable operation without building every discipline from scratch.
The decision should consider criticality, volume, frequency of change, data sensitivity, internal capabilities, and total operating cost. For a few batch models, a simple architecture may be sufficient. For critical online models subject to audits, automation, redundancy, and governance become part of the product.
How Predictor Solutions Handles This
Predictor Solutions structures ML projects from data contracts through CI/CD, model registries, progressive deployment, observability, security, and the definition of SLIs, SLOs, and SLAs. Its work combines custom software, applied artificial intelligence, data engineering, and cloud/DevOps; in healthcare, it includes systems with HL7 v2 and FHIR integration, in addition to the Predictor Health and Predictor AI Hospitals products.
As a practitioner, the company works with cases such as Ártemis AI, Avea, AMF, CEIS, Corrigiu, MiniMe Labs, and NexusML. Across its portfolio of nine medium-sized and large companies served, it reports average savings of R$ 1.32 million per client per year, an average productivity increase of 70%, and profit growth of 43% in six months. These results do not replace the definition of baselines and specific metrics for each new project, but they demonstrate the importance of connecting engineering to verifiable indicators.
Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246