RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured
RAG chunking strategies decide how documents are cut into pieces before they are embedded and retrieved. We split the same policy document five ways and measured what each method breaks: facts cut in half, headings separated from their text and extra tokens stored.
- RAG chunking strategies trade simplicity against keeping meaning intact. No single method won on every measure in our test.
- Plain fixed-size chunks of 100 tokens cut one of eight key facts in half: “Refunds are issued within | 5 business days”.
- 20 tokens of overlap rescued that fact but stored 545 tokens instead of 445, about 22% more.
- Recursive splitting kept every sentence whole but left 4 of 6 chunks ending with the next section’s heading.
Retrieval-augmented generation (RAG) works by finding the most relevant pieces of your documents and handing them to a language model. Those pieces are chunks, and the way you cut them quietly decides what the model can ever see. A fact split across two chunks may never be retrieved whole. A heading cut off from its paragraph makes a chunk ambiguous. Choosing among RAG chunking strategies is one of the cheapest ways to improve answer quality.
We implemented five common strategies in Python 3.13 and counted tokens with OpenAI’s tiktoken 0.14.0 using the o200k_base encoding. Every number below comes from that run. If RAG itself is new to you, read our what is RAG guide first.
What is chunking in RAG?
Chunking is the step that splits source documents into smaller passages before they are turned into embeddings and stored in a vector database. At question time, the system embeds the question, finds the closest chunks and pastes them into the prompt. Each chunk is retrieved, or ignored, as a unit.
Chunks must be small enough to be specific and to fit several into the model’s context window, and large enough to carry a complete thought. Every strategy is a different answer to where the cuts should go.
Why chunking matters
- Retrieval precision: a chunk mixing two topics matches questions about both, but answers neither well.
- Completeness: if a rule and its exception land in different chunks, the model may only see half the rule.
- Cost: overlap and large chunks mean more tokens to embed, store and send to the model.
- Citations: clean chunks map neatly to a section, which makes answers easier to verify.
Our test setup
We wrote a 368-word customer policy for a fictional online shop, with five Markdown sections (Returns, Shipping, Warranty, Accounts and Payments) and two paragraphs each. It came to 445 tokens. We chose eight facts a support bot must answer correctly, such as “return most items within 30 days” and “up to 10,000 rupees” for cash on delivery.
Each strategy used a target of 100 tokens per chunk, small enough to force cuts in a short document. For every result we measured:
- Chunks and average and maximum size in tokens.
- Stored tokens, the total across all chunks, which shows overlap cost.
- Cut mid-sentence: chunks that end part-way through a sentence.
- Orphan heading: chunks that end with a heading whose text is in the next chunk.
- Facts intact: how many of the eight facts appear whole in at least one chunk.
1. Fixed-size chunks
The simplest method counts tokens and cuts every N tokens, wherever that falls:
def fixed(size, overlap=0):
ids = enc.encode(doc)
step = size - overlap
out = []
for i in range(0, len(ids), step):
out.append(enc.decode(ids[i:i + size]))
if i + size >= len(ids):
break
return out
It produced 5 chunks with no wasted tokens, but 3 of them ended mid-sentence and one ended on an orphan heading. Worse, one fact was split down the middle:
chunk 1 ends: "... print the prepaid label. Refunds are issued within"
chunk 2 starts: " 5 business days after the warehouse receives and checks the"
A question about refund timing might retrieve chunk 1, which contains no number at all. Fixed-size chunking is fast and predictable, which makes it a fine baseline, but it ignores language entirely.
2. Fixed-size with overlap
Overlap repeats the last few tokens of each chunk at the start of the next, so text near a cut appears in both. We used 20 tokens, a 20% overlap. All eight facts were now intact somewhere, and the refund fact was rescued. The cost: 6 chunks and 545 stored tokens instead of 445. Cuts still landed mid-sentence, five times this time, and duplicate text can make two near-identical chunks crowd out other results. Overlap is a patch for arbitrary cuts, not a cure.
3. Sentence packing
This method splits text into sentences, then packs whole sentences into each chunk until the next one would exceed the limit:
def sentence_pack(size):
sents = [s for p in doc.split("\n\n") for s in sent_split(p) if s]
out, cur = [], ""
for s in sents:
if cur and T(cur + " " + s) > size:
out.append(cur)
cur = s
else:
cur = (cur + " " + s).strip()
return out + [cur]
It produced 5 chunks, no mid-sentence cuts and all eight facts intact, with 440 stored tokens. Two chunks still ended with an orphan heading, because a heading line looks like just another short sentence. Sentence splitting with a regular expression also stumbles on abbreviations such as “e.g.”; libraries such as spaCy or NLTK do this better.
4. Recursive splitting
Recursive splitting, made popular by frameworks such as LangChain, tries the biggest natural separator first (blank lines between paragraphs), and only falls back to smaller ones (line breaks, sentences, words) for pieces that are still too large. Our own implementation followed the same idea.
It kept every sentence whole and all eight facts intact, with the smallest chunks on average (74 tokens). But it revealed a subtle flaw: 4 of its 6 chunks ended with the next section’s heading. For example, one chunk ended “…checks the parcel. ## Shipping”, while the shipping rules sat in the following chunk. The heading attached itself to the wrong text, which can mislead retrieval for questions that mention “shipping”.
Tip: always print a sample of your chunks and read them. Our orphan heading problem was invisible in the averages and obvious after reading six chunks.
5. Heading-based chunks
Structured documents carry their own boundaries. Splitting on Markdown headings gave one chunk per section: 5 chunks, no mid-sentence cuts, no orphan headings and all facts intact. Each chunk starts with its own heading, which gives the embedding useful context.
The catch is size control. One section reached 113 tokens, over our 100-token target, and real documents have sections ranging from a line to several pages. In practice you combine methods: split by headings first, then apply recursive or sentence splitting inside any section that is too large, and repeat the heading at the top of each piece.
RAG chunking strategies compared
| Strategy | Chunks | Avg / max tokens | Stored tokens | Cut mid-sentence | Orphan heading | Facts intact |
|---|---|---|---|---|---|---|
| Fixed 100 | 5 | 89 / 100 | 445 | 3 | 1 | 7 of 8 |
| Fixed 100 + 20 overlap | 6 | 91 / 100 | 545 | 5 | 0 | 8 of 8 |
| Sentence packing | 5 | 88 / 100 | 440 | 0 | 2 | 8 of 8 |
| Recursive | 6 | 74 / 84 | 441 | 0 | 4 | 8 of 8 |
| By heading | 5 | 89 / 113 | 445 | 0 | 0 | 8 of 8 |
This is one short document, so treat the table as an illustration of failure modes rather than a benchmark. The pattern is still instructive: the more a method respects the document’s own structure, the fewer meaning-breaking cuts it makes.
What about semantic chunking?
Semantic chunking embeds each sentence and starts a new chunk where the meaning shifts, measured by a drop in similarity between neighbouring sentences. It needs no headings and can follow topic changes in plain prose. It also costs an embedding call per sentence, and published comparisons disagree on whether it beats simpler methods once the whole RAG pipeline is measured. We did not run it here. Try it only after a structure-aware baseline, and keep it only if your own evaluation shows a gain.
How to choose chunk size and overlap
- Start with structure. Split by headings or document sections when they exist.
- Cap the size. A common starting range is a few hundred tokens per chunk, around 200 to 500, then tune.
- Prefer clean boundaries over overlap. Add 10 to 20% overlap only if facts still straddle cuts.
- Attach metadata. Store the title, section heading and source URL with each chunk for filtering and citations.
- Measure retrieval. Write 20 to 50 real questions with known answers and check whether the right chunk ranks in the top results. Change one setting at a time.
- Count tokens properly with the tokenizer your models use, as explained in our LLM tokens guide.
Store the chunks in a database that supports metadata filters, as covered in our vector database guide.
Final thoughts
RAG chunking strategies are where many retrieval problems begin, and where they are cheapest to fix. Our test showed each method’s typical failure: fixed-size cuts facts in half, overlap costs storage, sentence packing and recursive splitting can strand headings, and heading-based chunks need a size cap. Start from your documents’ structure, fall back to sentence-aware splitting, read your chunks, and let a small evaluation set decide the rest.
Frequently asked questions
What is the best chunking strategy for RAG?
There is no universal best. For structured documents, split by headings and then cap size with recursive or sentence splitting. Validate the choice with your own retrieval tests.
What chunk size should I use for RAG?
A few hundred tokens, often around 200 to 500, is a common starting point. Smaller chunks are more precise, larger ones carry more context; tune with real questions.
Does chunk overlap help?
It can rescue facts that straddle a cut. In our test, 20 tokens of overlap kept all eight facts intact but stored about 22% more tokens and still cut sentences.
What is recursive chunking?
It splits on the largest separator first, such as blank lines, and only uses smaller separators like sentences or words for pieces that are still too large.
Is semantic chunking better than fixed-size chunking?
Not reliably. It follows topic shifts but costs extra embedding calls, and published results are mixed. Compare it against a simpler baseline on your own data.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.