Data engineering for SMBs should begin with the decision that needs to be made, not with the dashboard that looks interesting. A useful pipeline collects only the necessary data, validates its quality, calculates actionable metrics, and delivers alerts or tasks in the system where the team already works.
The problem is not a lack of reports
Many SMBs have data scattered across ERP systems, CRMs, spreadsheets, advertising platforms, payment gateways, and WhatsApp customer service. The common response is to connect everything to a BI tool and create dozens of charts.
This improves visualization, but it does not guarantee decisions. A dashboard only creates value when someone knows:
- which indicator to monitor;
- how often to check it;
- which threshold requires intervention;
- who should act;
- which action should be taken;
- how to measure the outcome of the action.
Without these definitions, the report becomes an endpoint. Decision-oriented data engineering treats visualization as one possible interface, not as the main product.
For example, showing the trend in delinquency is informative. Generating a daily list of customers with overdue installments, the amount at risk, contact history, and the recommended next action is operational.
Start with the decision and work backward
Before choosing a database, integration tool, or BI platform, describe the decision in one sentence:
When a specific condition occurs, a person or system must perform a specific action within a defined timeframe.
For a distributor, this could mean: “When projected inventory falls below expected demand before the next replenishment, the buyer should receive a purchase order recommendation.” For a service company: “When an opportunity has had no interaction beyond the commercial deadline, the owner should receive a task in the CRM.”
The six minimum questions
Each use case should answer:
- What decision will be made? Replenish inventory, collect from a customer, prioritize a lead, or review a contract.
- Who decides? Sales, finance, operations, management, or an automated process.
- Which data is essential? Include only fields that affect the decision.
- What latency is acceptable? Monthly, daily, hourly, or real time.
- What is the action criterion? A threshold, a composite rule, or a score.
- How will impact be measured? Time saved, losses avoided, conversion, or margin.
If these questions do not have clear answers, there is not yet a data engineering requirement; there is only an intention to “use data better.”
The simple architecture that usually works for SMBs
A pipeline does not need to begin with a data lake, streaming, and dozens of microservices. For many use cases, a batch architecture with five components is sufficient:
- Sources: ERP, CRM, spreadsheets, APIs, transactional database, and customer service.
- Ingestion: connectors, scripts, or API calls executed at defined intervals.
- Central storage: analytical database or managed data warehouse.
- Transformation: version-controlled SQL queries that standardize and combine the data.
- Activation: dashboard, alert, CRM task, message, or update in another system.
A practical storage organization uses three layers:
- Raw: a copy close to the original data, useful for auditing and reprocessing.
- Staging: typed, deduplicated, and normalized data.
- Marts: tables prepared for specific decisions, such as sales, inventory, or finance.
This separation prevents critical rules from being hidden in spreadsheets or directly within BI charts.
Batch or real time?
Real time makes sense when a few minutes change the action: fraud detection, operational downtime, or clinical alerts, for example. If the team reviews orders only once every morning, updating the pipeline every second adds cost and complexity without changing the decision.
Use the lowest latency that produces an operational effect:
- Monthly: accounting close and strategic analyses.
- Daily: collections, inventory, and sales monitoring.
- Hourly: operations with high order turnover.
- Minutes or seconds: only when a defined immediate response exists.
The intervals are design criteria, not universal rules. Frequency should align with the actual timeframe for taking action.
Example: from the CRM to the next sales action
Consider an SMB that records opportunities in its CRM but loses sales due to a lack of follow-up. The pipeline can be structured as follows:
- Extract opportunities, stages, owners, and interactions.
- Normalize dates, statuses, and customer identifiers.
- Remove duplicates using explicit rules.
- Calculate the time since the last interaction and the time spent in the stage.
- Apply priority criteria defined by the company.
- Create CRM tasks for the owners.
- Record whether the task was completed and what the outcome was.
The dashboard may still exist, but it is no longer the only consumption point. The information arrives as prioritized work within the sales workflow.
The primary metric also changes. Instead of counting dashboard visits, the company measures recovered opportunities, response time, completed tasks, and conversion after the intervention.
Data quality must block poor decisions
A pipeline that runs without technical errors can still deliver incorrect data. Quality must therefore be tested during every run.
Essential tests
- Completeness: are the fields required for the decision populated?
- Uniqueness: do orders, customers, and transactions appear only once?
- Validity: do dates, documents, values, and statuses follow accepted formats?
- Referential integrity: does every order point to an existing customer?
- Freshness: was the source updated within the expected window?
- Reconciliation: do financial totals match the source system?
Thresholds should reflect risk. For an exploratory campaign, some incomplete records may be tolerable. For billing or healthcare, relevant inconsistencies should stop publication and trigger an alert.
It is also important to record lineage: which source the field came from, which transformations were applied, and which version of the rule produced the result. This reduces disputes over “which number is correct.”
LGPD, security, and least-privilege access
Centralizing data increases responsibility for it. From the outset, an SMB should apply:
- collection of only the data required for the purpose;
- credentials stored in a secrets vault, never in code;
- encryption in transit and at rest;
- role-based access profiles;
- separation between development and production environments;
- access and change logs;
- retention and disposal policies;
- masking or anonymization whenever possible.
Personal data should not be replicated across every tool merely for convenience. If a decision requires only an identifier, segment, and aggregate value, there is no technical reason to expose a name, phone number, or identification document.
Metrics for evaluating the pipeline
Evaluation should combine technical reliability and business impact.
Technical indicators
- percentage of completed runs;
- delay between source and destination;
- number of passed quality tests;
- recovery time after a failure;
- infrastructure cost per run or per processed volume;
- number of manual interventions.
Decision indicators
- time between the event and the action;
- percentage of alerts that resulted in action;
- hours of manual work eliminated;
- losses avoided or revenue recovered;
- reduction in rework;
- improvement in conversion, margin, or service level.
A pipeline can be technically healthy and still be useless. If alerts are ignored or no action is assigned, the problem lies in the operational design, not necessarily in the technology.
When to build and when to use third-party tools
Managed tools accelerate connectors, scheduling, storage, and observability. However, they may charge by volume, limit customization, or create vendor dependence.
Custom scripts provide control and can support small integrations, but they require versioning, testing, monitoring, and maintenance as APIs change. “A script that works” is not automatically a production pipeline.
Consider a managed solution when standardized connectors are available and there is little technical differentiation. Consider custom development when the rules are specific, the systems do not have reliable connectors, or activation requires deep integration with operations.
A hybrid approach is often efficient: managed components for repetitive infrastructure and custom code for rules that represent the business.
Checklist for a first pipeline
Before starting, confirm:
- [ ] There is a specific decision and an owner.
- [ ] The expected action is documented.
- [ ] The essential sources have been identified.
- [ ] There is authorization and an appropriate legal basis for using the data.
- [ ] The frequency matches the actual decision timeframe.
- [ ] The transformation rules are version-controlled.
- [ ] Completeness, uniqueness, and freshness tests are in place.
- [ ] Failures generate alerts for a responsible person.
- [ ] The result reaches the channel where the team works.
- [ ] There is an impact metric beyond dashboard usage.
For the first project, choose a frequent, measurable decision with data that is already available. Avoid starting by integrating the entire company. Deliver a complete flow from source to action, then expand after proving adoption and reliability.
How Predictor Solutions solves this
Predictor Solutions, a software company based in Lavras, Minas Gerais, structures data engineering projects around operational decisions. Its work involves system integration, pipelines, modeling, quality testing, cloud/DevOps, security, and data activation in CRMs, platforms, alerts, or WhatsApp automations.
In healthcare, the company also works with HL7 v2 and FHIR integrations, in which traceability, interoperability, and access control are core requirements. When necessary, the pipelines feed artificial intelligence applications, such as the Predictor Health and Predictor AI Hospitals products.
According to the consolidated results reported by the company, nine medium-sized and large organizations have been served, with average savings of R$ 1.32 million per client per year, an average productivity increase of 70%, and profit growth of 43% in six months. These indicators depend on the context of each operation and should be validated using metrics defined before each implementation.
Contact: contato@predictorsolutions.com / WhatsApp +55 31 98835-3246.