RAG Chunking Strategies for Enterprise Data

Chunking is a measurable retrieval hypothesis. A fixed chunk size is not an architecture goal. The right strategy is the one that makes relevant evidence reliably retrievable for real questions without needlessly destroying context boundaries and document structure.

A retriever does not search an abstract document. It searches the units created during indexing. If a condition is separated from its instruction, or a table is reduced to unassignable text fragments, stronger embeddings, rerankers and models can compensate only so far.

For the wider architecture, see Build a RAG system the right way. For ingestion quality before chunking, see Connect RAG data sources.

Why fixed token limits often ignore the document

A token limit cannot tell whether it is splitting a warning, a table row or the prerequisite for a configuration step. Small units can be very precise to retrieve yet lose the reasoning context. Large units preserve context but can easily mix topics and pull in irrelevant material.

There is therefore no best chunk size independent of the query set and the evidence a question requires.

The scenario: a service manual, not a demo snippet

Use a 40–60-page technical service manual for a product family as the reference document. It contains chapters and subchapters, configuration steps, troubleshooting, cross-references, warnings, tables, product codes and spare-part numbers.

The questions users later ask are specific:

• “What torque applies to component X?”
• “Which prerequisite must be met before step 4?”
• “What does error code E217 mean?”
• “Which warning applies only to model B?”
• “Which spare-part number belongs to assembly Y?”

It does not matter whether every chunk is formally the same size. Torque and component, prerequisite and step, error code and meaning, or warning and model scope need to be available together as dependable evidence.

Three strategies against the same questions

Fixed-size: the technical baseline

Fixed-size splits extracted text at a fixed character or token limit. It is simple, reproducible and useful as a baseline. In the manual, however, the boundary can cut through:

• a warning for model B and its related work step,
• a table row containing an assembly and spare-part number,
• step 4 and the prerequisite directly above it.

That often creates small, easily indexed units with incomplete evidence. Overlap can soften the boundary, but cannot replace a meaningful one.

Structure-aware: respect the document boundaries

Structure-aware chunking follows the boundaries already carried by the manual: H1/H2/H3, sections, lists, tables and semantically related blocks. A chunk can, for example, contain a heading, the configuration steps and the related warning.

This preserves the section path for statements such as “applies only to model B” and prevents step 4 from being indexed without its prerequisite. The units will vary in size. Very long sections need a controlled split, preferably at further logical boundaries rather than in the middle of the content.

Parent/Child: retrieve precisely, answer completely

Parent/Child deliberately separates retrieval context from answer context. Small child units are indexed for search; after a hit, the larger parent section is loaded as answer context.

Retrieval Unit and Context Unit do not need to be identical.

In the manual, a child can match “E217” or “assembly Y” exactly. The parent then provides the error description, cause, affected models, warning and next diagnostic step, or the complete spare-parts table. This reduces irrelevant material at retrieval time, but can still load too much material for the answer. The parent must therefore remain a meaningful business boundary, such as a troubleshooting subsection rather than the whole chapter.

Comparison: what evidence reaches retrieval for the five questions?

This is not a general ranking. It is a hypothesis for this manual. The same assessment could change for a corpus of short, homogeneous notes.

Special cases: structure is evidence

Do not blindly split a table in the middle of a row. In a spare-parts table, the header, assembly, product code and spare-part number need to remain available so the relationship is unambiguous. If extraction has already corrupted that structure, solve parsing and normalisation first; it is not a chunk-size problem.

Keep lists and configuration steps as logical units where possible: the prerequisite before step 4 does not belong in a different retrieval context from step 4. Do not separate warnings from the related section—especially not from the model, configuration or work step they qualify. Headings, codes and product identifiers belong in the indexable representation because they often carry the essential domain reference.

Overlap: compensation, not a default

Overlap copies a boundary region into the next chunk. With fixed-size splitting, it can make a boundary between a warning and a step, or a table header and row, less damaging.

It does not replace a meaningful boundary. Duplicates enlarge the index, create near-identical hits and can add redundant answer context. Use overlap when evaluation shows that relevant evidence is lost at unavoidable boundaries—not as the default response to poorly preserved document structure.

Treat chunking as an evaluation experiment

For this manual, test each of the three variants with the same questions from a golden dataset. For every question, define the expected evidence: for example, the specific table row, warning with model scope or steps 1–4.

• Recall@k: Does the expected evidence appear in the first k retrieval results?
• Evidence Coverage: Does the retrieved context include every part needed for the answer?
• Context Precision: How much irrelevant context is retrieved as well?
• Answer Correctness: Can the system answer the question correctly from that evidence?

These metrics show whether a boundary lost evidence, whether a parent is too large or whether search fails to find the right child. The existing RAG Evaluation: Metrics, Golden Datasets & Regression Tests article covers ground truth, regression and the broader method.

Test chunking and hybrid search together

For exact product codes, part numbers and error codes such as E217, Hybrid Search can combine lexical search with semantic retrieval. That can improve the chance of finding the right candidate. It does not repair a separated table row or a detached warning. Test chunk boundaries and ranking separately.

Production decision rule

For structured enterprise documents, I would not begin with blind 500-token chunks. I would preserve document structure first, then test real questions to see whether smaller retrieval units or Parent/Child retrieval find the evidence more reliably.

Fixed-size remains a useful comparison baseline. Structure-aware is the obvious first production candidate for this manual. Parent/Child is justified when tests show that precise matches regularly need more section context to support correct answers. The best chunk size depends on the query set and the evidence expected.