Real-Time vs Batch Processing: How to Match Your Pipeline to Business Requirements

Batch processing collects data and runs transformations on a schedule. Real-time processing handles events continuously as they happen. Simple enough distinction on paper, but the downstream implications touch everything from infrastructure costs to whether your on-call engineer sleeps through the night. This guide breaks down when each approach makes sense, what hybrid architectures look like, and how to match your processing strategy to actual business requirements.

Key takeaways

The decision between batch and real-time processing shapes your infrastructure costs, team workload, and how quickly insights reach the people who need them.

• Batch collects, then processes: Data accumulates over a period and runs through transformations on a schedule (hourly, nightly, or weekly).
• Real-time processes continuously: Data flows through the system as events happen, with latency measured in milliseconds to seconds.
• Value decay drives the choice: If acting on stale data costs you revenue or safety, real-time makes sense. If a few hours of delay changes nothing, batch is simpler.
• Most teams run both: Batch handles the heavy historical work while real-time covers time-sensitive signals, and both feed the same serving layer.
• Domo supports either path: You can start with batch pipelines and layer in real-time processing later without rebuilding your data foundation.

What is real-time data processing?

A payment gateway approving a transaction before you lift your thumb from the screen. That's real-time. The system ingests, transforms, and acts on the data continuously as events occur, with end-to-end latency in the milliseconds-to-seconds range.

Near real-time is different. It takes seconds to a few minutes and often uses micro-batch techniques rather than true streaming. Vendors sometimes blur these lines, so ask for specific latency numbers when evaluating tools. A system marketed as "real-time" that actually delivers 30-second latency may not meet your fraud detection requirements. That gap between marketing copy and technical reality catches more teams than expected.

Real-time systems share a few characteristics that set them apart from batch:

• Unbounded data: Streams have no defined end. Events keep arriving.
• Event-time vs processing-time: The system tracks when something happened versus when it saw the record. This matters for late-arriving data.
• Stateful processing: The system maintains context across events, like session windows or running totals.
• Backpressure: When producers send data faster than consumers can handle, the system must queue, drop, or slow down.

Here is something that catches teams off guard: they build streaming pipelines that achieve sub-second processing, then dump the output into a dashboard that refreshes every 15 minutes. If your serving layer lags, the streaming pipeline delivers no actual speed advantage.

What is batch data processing?

The nightly job that reconciles millions of transactions, produces an auditable ledger, and can be re-run from scratch if someone finds a bug. That determinism is batch's core strength. Batch processing collects data over a period and processes it as a complete, bounded set at scheduled intervals.

Micro-batch sits in the middle. It uses small, frequent batches (one to five minutes) to approximate streaming latency while keeping batch semantics. This works when near real-time is acceptable and full streaming complexity is overkill.

Batch offers advantages that often get overlooked:

• Reproducibility: Run the same job on the same input, get the same output. Essential for audits and debugging.
• Lineage and governance: Tracing transformations is easier when jobs operate as discrete units.
• Cost efficiency: Compute spins up, runs, and shuts down. No always-on infrastructure required.

When does batch fit? Accuracy matters more than speed. Data volumes are large and bounded. Auditability is a compliance requirement. Or operational simplicity is a priority.

{{custom-cta-1}}

Real-time vs batch processing key differences

Latency and speed

Latency falls into three bands. Real-time typically targets sub-second to a few seconds at the 95th percentile (p95) end-to-end service-level agreement (SLA). Near real-time covers seconds to a few minutes. Batch spans minutes to hours to days.

Measure latency across the full lifecycle (ingest to process to serve). A fraud score loses value after two seconds. A weekly sales report? It doesn't degrade if it runs at 6:00am instead of 6:05am.

Scalability and resource utilization

Batch scales by spinning up compute for a window, processing, then releasing resources. Bursty but efficient.

Real-time requires always-on infrastructure that handles sustained throughput and absorbs traffic spikes without data loss. When producers outpace consumers, streaming systems experience backpressure, meaning they must queue, drop, or slow down. Batch avoids this entirely by processing after collection finishes.

Cost considerations

Real-time typically costs more. Always-on compute, 24/7 monitoring, on-call readiness. It adds up. The actual delta depends on event volume and your latency SLA.

Cost breaks into three areas. Compute involves continuous versus burst usage. Storage requires hot storage for streaming state versus cold storage for batch archives. Operations for streaming demand dedicated expertise and incident response; batch failures wait until the next scheduled run.

Teams often provision streaming infrastructure "just in case" without validating whether the latency gain justifies the spend. On-call fatigue doesn't appear on any invoice. If the business value of sub-minute latency doesn't clearly exceed the added cost, batch is the safer default.

Real-time and batch processing use cases

How quickly does the value of a signal decay? If acting on stale data costs you revenue, safety, or customer experience, real-time is justified. If delay is tolerable, batch is simpler and cheaper.

Real-time processing use cases

These scenarios need sub-minute latency:

• Fraud detection: Score transactions before authorization completes, usually sub-second. McKinsey's payments fraud research pegs total consumer losses at over $1 trillion in 2024, a number that underscores why milliseconds matter when blocking fraudulent charges.
• Internet of Things (IoT) safety alerts: Shut down equipment when sensor readings exceed thresholds, within seconds.
• Personalization: Serve recommendations while the customer is still browsing, in seconds.
• Observability and alerting: Detect anomalies in system metrics before outages cascade, seconds to low minutes.
• Streaming analytics dashboards: Surface live key performance indicators (KPIs) for trading floors or operations centers, low minutes.

Real-time is overkill if the downstream action happens on a daily cadence. Streaming data for a daily email campaign adds cost without delivering additional value.

Batch processing use cases

These scenarios make scheduled processing the right default:

• Financial close and reconciliation: Accuracy and auditability outweigh speed. Runs nightly or monthly.
• ML model retraining: Requires large historical datasets processed in bulk. Daily or weekly.
• Feature engineering for ML: Compute features once, serve them many times. Hourly to daily.
• Billing and invoicing: Deterministic, auditable runs that can be re-executed if errors surface. Monthly.
• Compliance reporting: Lineage and reproducibility are regulatory requirements. On-demand or scheduled.

Batch isn't legacy. It's the appropriate choice when correctness, cost, or auditability are the binding constraints.

{{custom-cta-2}}

Data pipeline architecture for real-time and batch

Some teams adopt a streaming tool before defining their latency SLA. Then they discover they built complexity they didn't need.

Pipeline layers follow a consistent pattern regardless of tooling. Ingestion handles connectors, change data capture, and message queues. Processing involves batch engines, stream processors, or micro-batch frameworks. Storage relies on data lakes, warehouses, feature stores, or materialized views. Serving delivers data to dashboards, application programming interfaces (APIs), AI agents, and embedded analytics.

Most production environments combine both. Batch handles heavy historical processing. Streaming manages time-sensitive signals. Both feed a shared serving layer.

Domo provides connectors, transformation, and serving across both patterns, with governance that carries from ingestion through AI agents to final delivery.

How to choose real-time vs batch for your data strategy

Defaulting to real-time because it sounds modern often means paying for complexity that delivers no incremental value.

Work through these questions:

1. Define the latency SLA: How quickly must the insight reach the decision-maker or system?
2. Quantify value decay: What's the cost of acting on stale data?
3. Assess volume and velocity: High-frequency events may justify streaming. Large, bounded datasets favor batch.
4. Evaluate operational readiness: Does the team have 24/7 monitoring and on-call capacity?
5. Consider governance requirements: Does the workload require lineage, reproducibility, or audit trails?

Choose batch if your latency SLA is measured in hours or days, if you need to re-run jobs for audits, or if your team lacks dedicated streaming expertise.

Choose real-time if data loses value within seconds or minutes, if you're triggering automated operational responses, or if the business case clearly funds always-on infrastructure.

When a hybrid approach makes sense

Most production environments are hybrid. Batch handles historical processing and backfills. Streaming handles time-sensitive signals. Both feed a shared serving layer.

Lambda architecture combines batch and speed layers. Kappa architecture uses streaming only and replays data for corrections. Lambda is more common because batch remains simpler for many workloads. Teams sometimes implement Lambda without a clear plan for reconciling outputs from both layers, which creates data consistency headaches downstream. You'll want to sort that out before you ship.

A phased migration works well: start with batch to establish governed data and prove value, identify high-value signals where delay causes measurable loss, add a minimal streaming path for those signals only, then validate ROI before expanding scope.

Hybrid adds integration complexity. You have to reconcile batch and streaming outputs.

Why Domo supports both real-time and batch processing

Many teams end up stitching together scheduled warehouse loads for batch workloads and separate streaming stacks for time-sensitive events. Domo supports both on a governed foundation, unified by design and modular by adoption.

The platform moves organizations from raw data to business outcomes through three layers: Foundation makes data AI-ready with connectors and governed transformation. Activation turns AI into action through agents and apps on governed data, where people set the objectives and constraints and agents execute within guardrails (human-in-the-loop). Distribution delivers outcomes into the workflows people already use (mobile, embedded analytics, AI assistants).

What this looks like in practice:

• 1,000+ connectors: Ingest from cloud apps, databases, and streaming sources without custom code.
• Governed transformation: Clean and model data once, reuse across batch and real-time paths.
• Real-time alerts and automated actions: Trigger the next step when thresholds are crossed, and surface the signal in dashboards when a visual view helps.
• AI agents on governed data: Activate AI that reads governed context and delivers outcomes with human-in-the-loop oversight.
• Distribution into existing workflows: Outcomes reach the people who change workflows through mobile apps, embedded analytics, and AI assistants.

Governance runs across every layer. Role-based access carries from data ingestion through AI agents to final delivery. Domo runs on top of your existing cloud data platform and preferred AI inference.

If you're ready to map your latency SLAs to an architecture you can actually operate (and afford), book a consultation to see what a practical batch + real-time setup looks like in Domo.