Avi Santoso
Open main menu
RAG

What's The Best Chunking Strategy For RAG Retrieval in 2026?

Start RAG retrieval with recursive or sentence-based splitting, then use evaluation results to decide whether semantic chunking is worth the cost.

By Avi Santoso5 min read

A practical 2026 baseline for choosing a RAG chunking strategy

TLDR

  • Start with recursive text splitting or sentence-based splitting, 400–512 tokens, 10–20% overlap.
  • If your documents have strong structure (headers, tables, sections), use structure-aware chunking instead.
  • Semantic chunking improves retrieval recall by roughly 2–10%, but costs 2–10x more to ingest.
  • Only move to semantic chunking once you've got an evaluation set and can prove it's worth it on your data.

When you build an LLM pipeline with cited documents, you often won't feed your documents to the LLM whole.

Instead, you split them into smaller pieces called chunks.

This keeps your overall context window low and lets you cite your documents more accurately.

Each chunk gets converted into a vector embedding and stored in your vector database.

When a user asks a question, your system searches for the chunks most similar to that question and injects them into the LLM's context window.

Chunking sits right at the start of this process, which means it directly affects everything downstream.

If your chunks are bad, the right information never makes it into the LLM's context, and no amount of prompt engineering will fix that.

The tradeoff about chunking is between precision and context.

Small chunks are focused. They're more likely to cover a single topic, which makes the embedding precise and retrieval accurate.

The downside is you lose surrounding context. If a user asks about an invoice, you might pull the chunk with the totals but miss the individual line items.

Those are in separate chunks, focused around the line item itself rather than the overall invoice, and may be hard to find.

Large chunks give you more context, but they also bring in noise.

A chunk covering multiple topics forces the embedding to represent all of them at once, which dilutes it.

You might retrieve the right chunk, but it's full of information the LLM doesn't need.

Irrelevant results usually come from chunks that are too large.

Incomplete results usually come from chunks that are too small.

For most documents, start with recursive text splitting or sentence-based splitting, with 10–20% overlap.

Use the lower end of that range if you're doing sentence-based splitting, since sentence boundaries give you a natural buffer already.

The default chunk size is 400–512 tokens, a reasonable middle ground for most use cases.

If your documents are strongly structured, use structure-aware chunking instead.

Think Markdown files with clear headers and subheaders, or reports with tables, facts, and figures that belong together.

It lets you split along the document's own logical boundaries (sections and tables) rather than imposing artificial ones.

If you're working with LangChain, the recommended starting point is RecursiveCharacterTextSplitter 1.

In LlamaIndex, it's SentenceSplitter, which is already the default 2.

Both frameworks are pointing you toward the same idea: respect the natural structure of the text, enforce a size limit, and keep it simple.

Semantic chunking is a real alternative to sentence- or character-based splitting, and it's worth knowing how it works.

The idea is similar to fitting a trend line to a graph: you go through the document sentence by sentence, and as long as each new sentence fits within your similarity threshold, it stays in the current chunk.

When a new sentence breaks that threshold, meaning it's semantically different enough from those in your current window, you close the current chunk and start a new one.

The result is chunks that follow the document's actual topical flow rather than arbitrary character or token counts.

The catch is that semantic chunking can cost 2–5x what recursive splitting costs at ingestion time, since it embeds every sentence rather than every chunk 3. If you're using an LLM-based approach, it can cost nearly ten times as much.

A poorly configured semantic chunker can also perform worse than recursive splitting.

The Vecta/FloTorch 2026 benchmark found a poorly configured semantic chunker scoring 54% end-to-end accuracy versus 69% for recursive splitting. That's a big drop 3.

So it's not something you can switch on and expect immediate gains.

The parameters you use need to be tuned to your situation, and this requires an evaluation setup.

When semantic chunking is well configured, it can improve retrieval recall by roughly 2–10% over recursive splitting 4.

Whether that improvement is worth the cost depends on your situation.

If your RAG system is upstream of a larger automated workflow, a 10% retrieval improvement could matter a lot, since errors compound.

If it's a high-stakes workflow and chunking is a small fraction of your total spend, the economics probably favour it.

The research numbers put semantic chunking at roughly 2–5x the monthly ingestion cost of recursive splitting, and LLM-based chunking at about ten times the cost of a basic strategy.

That might sound like a lot in isolation. Still, relative to the total cost of running the system, or the cost of wrong answers, it could be negligible.

I'd treat semantic chunking as an evaluated optimisation, not a starting point.

So, my recommendation is that for new workflows, you should start with recursive text splitting.

It's fast, simple, well-supported in both major frameworks, and competitive with more complex strategies when configured sensibly.

From there:

  1. Build an evaluation set: something like 100 test questions ranging from easy to hard, drawn from your actual documents.
  2. Run both chunking strategies on a small representative subset of your data.
  3. Compare the outputs and measure the difference.
  4. If semantic chunking shows a meaningful improvement on that subset, expand it. If not, recursive splitting is doing the job.

It's also worth thinking about a multi-step approach: first identify which document or section is relevant through a broader search, then drill into smaller chunks within that section.

That kind of layered retrieval can get you the context benefits of large chunks and the precision of small ones, without having to pick one.

Start with recursive or sentence-based splitting, tune it with a real evaluation, and only move to semantic chunking once you can prove it's giving you better results on your data.

References

  1. LangChain. (2024). Recursively split by character. LangChain Documentation. https://python.langchain.com/docs/how_to/recursive_text_splitter/
  2. LlamaIndex. (2024). Node Parsers / Text Splitters. LlamaIndex Documentation. https://developers.llamaindex.ai/typescript/framework/modules/data/ingestion_pipeline/transformations/node-parser/
  3. Vecta. (2026, February). We Benchmarked 7 Chunking Strategies on Real-World Data. Most Advice Was Wrong. Vecta Blog. https://www.runvecta.com/blog/we-benchmarked-7-chunking-strategies-most-advice-was-wrong
  4. Chroma Research. (2024). Evaluating Chunking Strategies for Retrieval. Chroma. https://www.trychroma.com/research/evaluating-chunking