A dashboard needs updates within seconds, while finance needs corrected daily totals. Late events, source revisions and new business rules can change old results. The architecture must explain how live estimates and historical truth converge.
Lambda: live results plus historical recomputation
Lambda commonly maintains a batch path over retained history and a low-latency speed path for recent data. Serving combines their results under a defined cutoff.
Source of replayable events
Recompute settled history
Update recent events quickly
History before the published cutoff
Events from the published cutoff
Reconcile event identity and coverage
- HistoryEvent time < 12:00
- RecentEvent time ≥ 12:00
- Retained event log to Batch processing.
- Retained event log to Stream processing.
- Batch processing to Batch view.
- Stream processing to Recent view.
- Batch view to Serving query.
- Recent view to Serving query.
Both paths consume the retained source. The serving query combines disjoint coverage, such as settled history before 12:00 and recent events from 12:00. A changed cutoff needs coordinated publication to avoid duplicates or missing events.
Reference: Microsoft.
The difficult part is reconciliation. When a batch view advances, its coverage replaces the corresponding speed result. Independent implementations can disagree about rounding, deduplication or late events. Shared transformation definitions and explicit version/cutoff metadata reduce that risk.
Choose separate paths when low-latency and large historical computation genuinely need different execution strategies and the team can maintain their consistency. A batch-only pipeline is often sufficient when a few minutes of delay meets the product requirement.
Kappa: recompute by replaying through a stream-processing path
Kappa emphasizes a retained log and a stream-processing computation that can process new data or replay old data. A replacement processor can rebuild a separate output, catch up, validate and switch consumers.
Replay enough history to rebuild the result
Continues processing live events
Runs the new processing logic
Readers continue using this result
Isolated rebuild; readers have not switched
Check progress, counts and business aggregates
Publish the candidate and retain a rollback path
- Retained event log to Current processor.
- Retained event log to Replay processor.
- Current processor to Active view.
- Replay processor to Candidate view.
- Active view to Catch up and reconcile.
- Candidate view to Catch up and reconcile.
- Catch up and reconcile to Switch the reader pointer.
The current processor continues serving while a new version replays retained events. The candidate catches up and is reconciled with the active result. A controlled reader switch publishes the accepted candidate.
Reference: Microsoft.
Replay requires enough retained history and compatible schemas, reference data and processing logic. Resetting a consumer offset without resetting or isolating state can double-count old events. If history has expired, recovery may need an object-store archive or a new source snapshot.
Changing keys, windows or aggregation semantics can require a state migration or complete rebuild. A unified batch/stream API can reduce duplicated logic while still using different execution plans.
Medallion describes data quality and responsibility
Bronze, silver and gold commonly describe source-oriented, validated/conformed and consumer-oriented datasets. They need contracts such as source provenance, key uniqueness and accepted lateness. Those responsibilities can be implemented with different schedules, storage systems and numbers of physical tables.
A silver contract can remain stable while versioned adapters handle upstream schema changes. Unknown fields can remain in raw records until a mapping is approved. A streaming topology can implement the same quality responsibilities with stateful operators and durable outputs.
A lakehouse using medallion layers can contain a Data Vault integration model and Kimball stars in its presentation layer. Lambda/Kappa concerns how results are computed and rebuilt; these choices describe different parts of that system.
Source records and retained bytes
Types, identity and history
Consumer facts and shared metrics
Rejected records, reason and repair status
- Bronze to Silver.
- Silver to Gold.
- Silver to Quarantine.
- Quarantine to Silver.
Bronze preserves replayable source records. Silver establishes typed entities, keys and history. Gold publishes consumer facts and metrics. Rejected records retain enough context to be repaired and checked again.
Reference: Databricks data warehousing architecture.
The schema-contract reference explains stable Silver mappings and consumer migrations. The lake design reference places catalogs, table commits and serving engines around these datasets.
Event sourcing and CQRS solve another problem
Event sourcing makes the event history the application’s authoritative record. Command Query Responsibility Segregation (CQRS) separates write operations from read models. An ordinary change-data-capture feed derived from mutable database rows is not automatically an event-sourced domain model.
An event-sourced service may publish a warehouse projection, but replay must suppress or reconcile external side effects such as charging a card or sending an email. Analytical replay rebuilds data with business side effects disabled or reconciled.
A hybrid with bounded responsibilities
Use streaming for a fraud feature that needs second-level freshness, periodic warehouse jobs for settled financial reports, and an archive for reconstruction. Label provisional versus finalized measures, share source identifiers and define when late changes restate history.
| Requirement | Starting direction | Evidence that the choice needs revision |
|---|---|---|
| Hourly or daily reports | Batch / micro-batch | Freshness misses a business decision |
| Fast view plus independent historical repair | Lambda-style paths | Reconciliation and duplicate logic dominate cost |
| Retained events and replayable state | Kappa-style computation | Expired history or incompatible state blocks rebuild |
| Progressive validation and consumer contracts | Medallion responsibilities | Copies add latency without improving quality |
| Several legitimate latency needs | Bounded hybrid | Overlapping ownership creates conflicting truth |
A hybrid needs a purpose, publication contract and repair procedure for every path. Recovery and backfills works through that procedure.