RAG Chunking Strategies for Enterprise Data

Chunking defines which information units a RAG system can retrieve at all. If a relevant statement is split across poor boundaries or buried inside too much unrelated text, stronger embeddings, rerankers or LLMs can only compensate to a limited extent.

Chunking is therefore not cosmetic preprocessing. It is a retrieval and information-architecture decision.

For the broader pipeline, see Build a RAG system the right way. For measuring alternatives, see RAG Evaluation: Metrics, Golden Datasets & Regression Tests.

The short version

| Strategy | Good for | Main risk | | --- | --- | --- | | Fixed-size | simple baseline, homogeneous text | breaks logical units | | Recursive | general documents with paragraphs | structure is only indirectly represented | | Structural | manuals, wikis, policies, Markdown | parser must preserve structure | | Semantic | long text with topic shifts | more compute and tuning | | Hierarchical | long documents with chapters and details | more complex retrieval | | Contextualised | chunks missing standalone context | higher ingestion cost |

Recommendation: for enterprise RAG, start from document structure and then verify whether added complexity improves results on a golden dataset.

Why chunking matters so much

A retriever does not search an abstract “document”. It searches the units created during indexing.

If a policy states:

Travel expenses above 1,000 euros require additional approval.

and “above 1,000 euros” lands in a different chunk from “additional approval”, later retrieval may surface incomplete evidence.

The opposite is also problematic: a huge chunk can contain several topics. Its embedding represents a mixture and the important passage becomes less distinctive.

Four goals of good chunking

A useful chunking strategy balances several goals:

1. Retrievability – relevant evidence must be findable.
2. Semantic coherence – a chunk should remain understandable.
3. Evidence quality – it should be usable as a source.
4. Operational efficiency – index size, latency and cost must stay acceptable.

These goals can conflict, which is why there is no universal “correct” chunk size.

1. Fixed-size chunking

Fixed-size chunking splits text by character or token limits.

Advantages

• simple,
• reproducible,
• fast,
• useful as a technical baseline.

Weaknesses

• breaks paragraphs, lists or tables unnaturally,
• ignores hierarchy,
• can separate headings from content.

It may be sufficient for homogeneous prose. For complex enterprise documents, it is often better treated as the baseline to beat.

2. Recursive chunking

Recursive splitters first try larger natural boundaries such as paragraphs, then fall back to smaller separators when necessary.

This is pragmatic and more robust than blind fixed-size splitting.

But the parser still needs to correctly identify headings, paragraphs, lists and tables. Poor extraction cannot be fully repaired by a smarter splitter later.

3. Structural chunking

Enterprise documents often already contain useful structure:

• chapters,
• headings,
• subheadings,
• lists,
• table rows,
• FAQ entries,
• tickets,
• wiki sections.

Instead of “every 800 tokens”, a retrieval unit can be heading + section + parent context.

This is especially useful for:

• SharePoint,
• Confluence,
• document-management systems,
• technical manuals,
• policies,
• product documentation.

IBM describes chunking as splitting documents into smaller meaningful segments and notes that chunk size and overlap need to be selected for the specific RAG use case. Source: IBM – Chunking strategies for RAG.

4. Semantic chunking

Semantic chunking tries to detect topic changes and place boundaries there.

It can help with long prose, but it is not a free quality upgrade.

Potential costs include:

• additional embedding or model calls,
• less deterministic boundaries,
• more parameters,
• longer ingestion,
• no guaranteed benefit for already structured documents.

Compare it against a simpler baseline before adopting it.

5. Hierarchical chunking

Long documents often have several useful levels:

Document → chapter → section → detail

A hierarchical approach can first identify a broader relevant area and then retrieve smaller evidence units.

This helps when users ask both broad and specific questions.

Example:

• “What are our business-travel rules?” → chapter level,
• “Does a rail ticket above 500 euros require approval?” → detail level.

6. Contextualised chunking

A chunk can be locally correct yet difficult to understand without its parent context.

Example:

“The threshold is 25 percent.”

Without a document title, section or subject, this statement is weak retrieval evidence.

Anthropic's “Contextual Retrieval” describes an approach where chunk-specific document context is prepended before embedding and lexical indexing. Source: Anthropic – Contextual Retrieval.

For enterprise RAG, the idea can be useful, but it needs to earn its additional ingestion cost on the actual corpus.

Decision matrix by document type

| Document type | Useful baseline | | --- | --- | | Wiki / Confluence / SharePoint | heading + section, structurally | | technical manuals | chapter/section + size cap | | policies | paragraph/clause + parent heading | | contracts | clause/section boundaries, avoid blind splitting | | tables | structured rows/objects rather than raw text blocks | | tickets | ticket as unit or threaded segments with metadata | | product catalogue | structured fields, not primarily prose chunking | | long unstructured PDFs | parsing first, then test recursive/semantic |

The key principle is simple: chunking should match the source type.

See also Connect RAG data sources.

Tables are a special case

PDF extraction often destroys table relationships.

A clean table can turn into text where values and columns are no longer associated correctly.

At that point the issue is not “chunk size”. It is document understanding and structured extraction.

For business-critical tables, evaluate whether they should be indexed as structured data or queried separately.

Overlap: useful, but not free

Overlap copies part of one chunk into the next.

It can preserve context around boundaries.

Too much overlap creates:

• larger indexes,
• more near-duplicate hits,
• redundant context,
• higher embedding cost.

Treat overlap as an experiment parameter, not a quality dial to maximise.

Chunk metadata is part of the strategy

A chunk should contain more than text.

Useful metadata can include:

• document ID,
• title,
• section path,
• source,
• modification date,
• language,
• tenant,
• ACL / permission information,
• document type,
• version.

These fields support filtering, citations, freshness and security trimming.

Think about chunking and hybrid search together

Chunking defines candidates. Retrieval determines which candidates are found.

For exact codes or rare domain terms, hybrid search can combine lexical and semantic signals.

But hybrid search cannot fully repair bad chunk boundaries. If the relevant information is not represented well, ranking remains constrained.

How to evaluate chunking

Compare several strategies against identical questions.

For example:

| Variant | Chunking | | --- | --- | | A | fixed-size | | B | recursive | | C | structural | | D | structural + parent context |

Measure:

• Recall@k,
• rank of relevant chunks,
• context completeness,
• answer quality,
• citation accuracy,
• index size,
• ingestion time,
• latency.

See RAG Evaluation for the methodology.

Anti-patterns

“We'll just use 500 tokens”

Without document and query context, that is only a starting value.

“More overlap improves quality”

More duplication can also harm retrieval.

“Semantic chunking is always more advanced”

Complexity only matters when it produces measurable gains.

“The PDF was extracted, therefore the content is clean”

Tables, headers, footers, columns and lists can already be damaged before chunking begins.

Next step

If you are building RAG for SharePoint, DMS, ERP, CRM, databases or other enterprise sources, plan chunking together with parsing, permissions and retrieval.

See RAG implementation & AI integration.

If a system already exists and you need to determine which chunking strategy actually performs better, establish a baseline and evaluation set first: AI Production Readiness.