A pipeline can complete successfully and still publish a broken dataset. Investigate data quality, schema changes, downstream contracts and recovery together: a new source field may ingest correctly but break a transformation or change the meaning of a metric.
In this chapter
- Data pipeline CI/CD and release validationTest semantics, compatibility and candidate outputs before promotion.
- Schema evolution, data contracts, and safe migrationsSource schema changes, curated mappings, consumer contracts and migration plans.
- Streaming acknowledgments, retries and replay safetyPub/Sub deadlines, idempotent warehouse writes and backpressure.
- Diagnosing data failures from symptoms and evidenceTrace wrong values, freshness lag and resource problems to discriminating evidence.
- Resumable pipelines, replay, backfills and safe recoveryRecover uncertain effects, isolate historical repairs and validate RPO/RTO.
- Pipeline metadata, lineage and tracking toolsChoose control records, artifacts, OpenLineage, catalogs and telemetry.