Text splitting works when documents are mostly narrative prose. It fails when meaning depends on relationships encoded in layout, tables, images or code structure. A value such as 14.6 means little without its column heading, currency, period and entity. A code fragment may refer to a symbol defined in another file.
Preserve document layout before embedding
Document ingestion first identifies source type and its internal structure. Optical character recognition (OCR) converts pixels to text, while layout extraction identifies headings, tables, captions, lists and reading order. The parser should preserve source-page and bounding-region references so a retrieval hit can be traced back to the evidence a reviewer sees.
For a table, capture the table title, headers, row identity, units and dates. When a long table must split, repeat headers and stable identifiers in every fragment. If a question requires exact sums, ranges or joins across thousands of table rows, place the structured records in SQL and let retrieval locate the table definition or relevant identifiers.
For images, store captions, surrounding text and image references. Multimodal encoders can represent image content for retrieval, but require their own modality-aligned training and evaluation. OCR alone can miss a chart’s trends, legend or visual relationships.
Use syntax-aware units for code
An abstract syntax tree (AST) identifies functions, classes and declarations. Index a method together with its signature, file path, language, imports and surrounding documentation. Stable symbols support links to callers and tests. A function-only snippet may answer “where is this implemented?” while a parent file excerpt answers “what state does it rely on?”
Swipe or scroll sideways to read the full diagram.
Source type determines extraction: prose becomes sections; tables keep headings and typed rows; code becomes AST-aware symbols; images keep captions or aligned multimodal embeddings. All indexed units preserve source references.
Parent-child and multi-vector retrieval
A short child chunk can be an excellent search key while omitting the parent section needed for an accurate answer. Store links from child chunks to larger parents. Retrieve children for precision, then fetch selected parents for complete context; account for the extra tokens.
For tables, it can be useful to index both the table summary and row-level representations. For code, index names, docstrings and behavior descriptions alongside symbols. This improves recall but creates multiple entries per source object, so deduplication and version tracking become essential.
Failure-specific tests
Build evaluation questions that require reading a table across a page break, recognizing units, resolving a variable defined in a different module, interpreting a figure or citing an exact line. Validate the parser and extraction output separately from retrieval; an encoder cannot retrieve a cell lost during extraction.
Record document and section revisions with source-span provenance, then check that updates to a table or function invalidate the matching chunks and answers. The incremental indexing guide owns that update process.