Machine Learning in Production: From Prototype to a Contractual SLA with MLOps

A machine learning prototype is only ready for production when it becomes an observable, reproducible, secure service that can operate under failure conditions. MLOps connects data, models, infrastructure, and governance to convert experimental metrics into measurable SLOs and, when necessary, into a contractual SLA with explicit responsibilities and consequences.

Why a Good Metric in a Notebook Is Not Enough

An experiment typically assumes available data, a controlled environment, and evaluation against a static dataset. In production, the model becomes dependent on APIs, databases, pipelines, permissions, infrastructure, user behavior, and changes in the process that generates the data.

A classifier with an F1-score of 0.92 can fail operationally if:

• the API exceeds the maximum response time;
• attributes arrive as null or use different units;
• the transformation used during training differs from the one used during inference;
• the population profile changes;
• the model version cannot be traced;
• an update prevents rollback;
• predictions are stored without the controls required by the LGPD.

Therefore, there are at least three separate dimensions of quality:

1. Predictive quality: precision, recall, F1, AUC, MAE, RMSE, or another metric compatible with the problem.
2. Operational quality: availability, latency, error rate, throughput, and recovery time.
3. Governance quality: traceability, security, explainability, approval, and evidence retention.

Moving to production requires minimum criteria across all three dimensions, not only the first one.

From Business Requirements to Measurable SLOs

Before choosing tools, the team must identify which decision the model supports, how much an error costs, and how quickly the response must be available. This determines the architecture, metrics, and level of operations.

SLA, SLO, and SLI Are Not Synonyms

• SLI, or Service Level Indicator: an observed measurement, such as the proportion of requests answered within 300 ms.
• SLO, or Service Level Objective: an internal target, such as keeping 99.9% of inferences within that limit.
• SLA, or Service Level Agreement: a contractual commitment, including scope, measurement method, exceptions, and potential compensation.

The internal SLO should be stricter than the SLA. If the contract promises 99.5% availability, operating exactly at that threshold eliminates the margin for incidents, maintenance, and measurement errors.

Monthly availability can be calculated as:

valid requests served / total valid requests

The contract must clarify what constitutes a valid request, which windows are excluded, and whether measurement occurs at the gateway, inference service, or client application.

Not Every Model Metric Should Become an SLA

Latency and availability can be measured immediately. Recall, precision, and false-negative rate, however, depend on ground-truth labels, which often arrive days or months later. They can also vary when the population or operational process changes.

In these cases, a safer structure separates:

• the online service SLA;
• the input data quality SLO;
• periodic evaluation of predictive performance;
• triggers for investigation, recalibration, or retraining;
• responsibility for providing reliable labels.

In healthcare, fraud, or predictive maintenance, this distinction prevents guarantees involving metrics that are impossible to verify in real time.

Minimum MLOps Architecture for Production

There is no universal stack, but the components must fulfill clear functions:

1. Versioning: code, data or immutable references, configuration, dependencies, and model artifacts.
2. Experiment tracking: parameters, metrics, evaluation dataset, and the origin of each run.
3. Data pipeline: validation of schemas, types, ranges, cardinality, and handling of missing values.
4. Model registry: stage, owner, approval, metrics, and linkage to the training code.
5. CI/CD/CT: testing, packaging, deployment, and, when applicable, controlled continuous training.
6. Serving: batch, synchronous, or asynchronous inference, depending on latency and volume.
7. Observability: logs, metrics, traces, data quality, drift, and subsequent performance.
8. Incident management: actionable alerts, runbooks, owners, and a rollback mechanism.

A feature store can reduce inconsistencies between training and inference, but it adds operational cost. It makes sense when multiple teams reuse features, computation is complex, or online features must be served with low latency. For a single batch model, versioned transformations within the pipeline itself may be sufficient.

Pipeline from Prototype to Deployment

1. Reproduce Training

Training must run outside the notebook through a deterministic command or pipeline. The run must record the code version, configuration, data period, random seed when applicable, dependencies, and artifact hash.

Containers help with portability, but they do not guarantee data reproducibility. Snapshots, immutable partitions, or queries with documented time-based criteria must be preserved.

2. Create a Test Suite

An ML pipeline must combine different tests:

• unit tests for transformations and business rules;
• contract tests for input and output schemas;
• integration tests for databases, queues, and APIs;
• data quality tests for nulls, ranges, and categories;
• predictive regression tests against a reference model;
• load tests for latency, throughput, and resource consumption;
• security tests for authentication, authorization, and vulnerable dependencies.

The candidate model should not be promoted simply because it outperforms a metric. It must also comply with size, cost, latency, and degradation limits across relevant segments.

3. Define Promotion Gates

A production gate may require, for example:

• no data contract violations;
• the primary metric to outperform the defined baseline;
• no degradation above the threshold in critical segments;
• p95 latency within budget;
• a signed and registered artifact;
• human approval for high-impact cases;
• a tested rollback.

The values must come from risk and the business process. There is no technical justification for adopting an F1-score of 0.80 or a latency of 200 ms as universal values.

4. Release Progressively

Blue-green, canary, and shadow deployments reduce the risk of direct replacement. In shadow mode, the new model receives a copy of the traffic without influencing the decision. In a canary deployment, it serves a controlled fraction and can be compared with the current model.

Rollback must account for the model, preprocessing code, and schema. Reverting only the model file may preserve the incompatibility that caused the incident.

Monitoring That Detects Useful Failures

Monitoring CPU and memory does not reveal whether the model has lost validity. Observability must cover four layers:

• Service: availability, p50, p95, and p99 latencies, errors, queues, and saturation.
• Data: volume, freshness, nulls, ranges, unknown categories, and schema violations.
• Predictions: output distribution, confidence, abstention rate, and concentration by segment.
• Actual performance: metrics calculated when ground-truth labels become available.

Drift does not automatically mean a loss of performance. A change in the distribution may be legitimate while the relationship between features and outcomes remains stable. The alert should initiate a diagnosis, not necessarily automatic retraining.

Alerts must also be actionable. Each one should indicate severity, owner, runbook, diagnostic data, and resolution condition. Without this, the team accumulates ignored alarms and reduces its ability to respond to real incidents.

Error Budget, Incidents, and Operational Capacity

If the monthly SLO is 99.9%, the error budget is 0.1% of the period or requests, depending on the adopted definition. This budget guides decisions: while consumption remains controlled, the team can accelerate changes; when it approaches the limit, reliability must be prioritized.

The operational plan must answer:

• Who receives the alert outside business hours?
• What are the expected acknowledgment and restoration times?
• Is there a fallback based on a rule, previous model, or human review?
• How will affected decisions be identified and corrected?
• What evidence will be preserved for auditing?

In high-impact systems, unavailability and an incorrect prediction are different incidents. Each category requires its own severity, communication, and procedure.

Checklist for a Contractual ML SLA

Before signing, verify whether the document defines:

• the service, endpoint, regions, and covered hours;
• availability and latency by percentiles, not only by average;
• the official source of measurements;
• expected volume and load pattern;
• limits, exclusions, and scheduled maintenance;
• the quality and format of data supplied by the client;
• predictive evaluation frequency and label delay;
• the change, approval, and versioning process;
• RTO for restoration and RPO when there is persistent state;
• security, privacy, retention, and incident response;
• support, escalation, and compensation;
• criteria for termination or renegotiation.

The main trade-off is economic: increasing availability, reducing latency, and maintaining continuous support require redundancy, spare capacity, and specialized operations. The SLA must reflect the actual impact of failure, avoiding both underprovisioning and unnecessary infrastructure.

How Predictor Solutions Solves This

Predictor Solutions structures ML projects from data engineering through serving, observability, cloud/DevOps, and offensive security. In healthcare applications, it also works with HL7 v2 and FHIR integrations; its Predictor Health and Predictor AI Hospitals products apply this foundation to dashboards, wearables, and models focused on predicting sepsis, heart attacks, and pneumonia in ICUs.

Execution begins by defining the decision, risk, baseline, and SLOs. Reproducible pipelines, tests, model registration, progressive deployment, monitoring, and runbooks compatible with the SLA are then implemented. The company, headquartered in Lavras, Minas Gerais, has already served 9 medium and large organizations; its reported consolidated results include average savings of R$ 1.32 million per client per year, an average productivity increase of 70%, and profit growth of 43% in six months.

Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246