Define source and output promises, choose bounded or ongoing computation, and estimate volume and recovery capacity. Durable input, checkpoints and publication records explain what a rerun can safely repeat.
In this chapter
- ETL, ELT, and the source-to-warehouse pathFollow an event through the pipeline and decide where transformation belongs.
- Data capacity and back-of-the-envelope calculationsVolume, state, headroom, file counts, replay time and cost envelopes.
- Pipeline fundamentals: contracts, checkpoints and publicationDefine durable input, stage progress and verified consumer releases.