1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies decide how documents are cut into pieces before they are embedded and retrieved. We split the same policy document five ways and measured what each method breaks: facts cut in half, headings separated from their text and extra tokens stored.

9 min read
RAG chunking strategies explained and measured

Key takeaways

  • RAG chunking strategies trade simplicity against keeping meaning intact. No single method won on every measure in our test.
  • Plain fixed-size chunks of 100 tokens cut one of eight key facts in half: “Refunds are issued within | 5 business days”.
  • 20 tokens of overlap rescued that fact but stored 545 tokens instead of 445, about 22% more.
  • Recursive splitting kept every sentence whole but left 4 of 6 chunks ending with the next section’s heading.

Retrieval-augmented generation (RAG) works by finding the most relevant pieces of your documents and handing them to a language model. Those pieces are chunks, and the way you cut them quietly decides what the model can ever see. A fact split across two chunks may never be retrieved whole. A heading cut off from its paragraph makes a chunk ambiguous. Choosing among RAG chunking strategies is one of the cheapest ways to improve answer quality.

We implemented five common strategies in Python 3.13 and counted tokens with OpenAI’s tiktoken 0.14.0 using the o200k_base encoding. Every number below comes from that run. If RAG itself is new to you, read our what is RAG guide first.

What is chunking in RAG?

Chunking is the step that splits source documents into smaller passages before they are turned into embeddings and stored in a vector database. At question time, the system embeds the question, finds the closest chunks and pastes them into the prompt. Each chunk is retrieved, or ignored, as a unit.

Chunks must be small enough to be specific and to fit several into the model’s context window, and large enough to carry a complete thought. Every strategy is a different answer to where the cuts should go.

Why chunking matters

  • Retrieval precision: a chunk mixing two topics matches questions about both, but answers neither well.
  • Completeness: if a rule and its exception land in different chunks, the model may only see half the rule.
  • Cost: overlap and large chunks mean more tokens to embed, store and send to the model.
  • Citations: clean chunks map neatly to a section, which makes answers easier to verify.

Our test setup

We wrote a 368-word customer policy for a fictional online shop, with five Markdown sections (Returns, Shipping, Warranty, Accounts and Payments) and two paragraphs each. It came to 445 tokens. We chose eight facts a support bot must answer correctly, such as “return most items within 30 days” and “up to 10,000 rupees” for cash on delivery.

Each strategy used a target of 100 tokens per chunk, small enough to force cuts in a short document. For every result we measured:

  • Chunks and average and maximum size in tokens.
  • Stored tokens, the total across all chunks, which shows overlap cost.
  • Cut mid-sentence: chunks that end part-way through a sentence.
  • Orphan heading: chunks that end with a heading whose text is in the next chunk.
  • Facts intact: how many of the eight facts appear whole in at least one chunk.

1. Fixed-size chunks

The simplest method counts tokens and cuts every N tokens, wherever that falls:

def fixed(size, overlap=0):
    ids = enc.encode(doc)
    step = size - overlap
    out = []
    for i in range(0, len(ids), step):
        out.append(enc.decode(ids[i:i + size]))
        if i + size >= len(ids):
            break
    return out

It produced 5 chunks with no wasted tokens, but 3 of them ended mid-sentence and one ended on an orphan heading. Worse, one fact was split down the middle:

chunk 1 ends:   "... print the prepaid label. Refunds are issued within"
chunk 2 starts: " 5 business days after the warehouse receives and checks the"

A question about refund timing might retrieve chunk 1, which contains no number at all. Fixed-size chunking is fast and predictable, which makes it a fine baseline, but it ignores language entirely.

2. Fixed-size with overlap

Overlap repeats the last few tokens of each chunk at the start of the next, so text near a cut appears in both. We used 20 tokens, a 20% overlap. All eight facts were now intact somewhere, and the refund fact was rescued. The cost: 6 chunks and 545 stored tokens instead of 445. Cuts still landed mid-sentence, five times this time, and duplicate text can make two near-identical chunks crowd out other results. Overlap is a patch for arbitrary cuts, not a cure.

3. Sentence packing

This method splits text into sentences, then packs whole sentences into each chunk until the next one would exceed the limit:

def sentence_pack(size):
    sents = [s for p in doc.split("\n\n") for s in sent_split(p) if s]
    out, cur = [], ""
    for s in sents:
        if cur and T(cur + " " + s) > size:
            out.append(cur)
            cur = s
        else:
            cur = (cur + " " + s).strip()
    return out + [cur]

It produced 5 chunks, no mid-sentence cuts and all eight facts intact, with 440 stored tokens. Two chunks still ended with an orphan heading, because a heading line looks like just another short sentence. Sentence splitting with a regular expression also stumbles on abbreviations such as “e.g.”; libraries such as spaCy or NLTK do this better.

4. Recursive splitting

Recursive splitting, made popular by frameworks such as LangChain, tries the biggest natural separator first (blank lines between paragraphs), and only falls back to smaller ones (line breaks, sentences, words) for pieces that are still too large. Our own implementation followed the same idea.

It kept every sentence whole and all eight facts intact, with the smallest chunks on average (74 tokens). But it revealed a subtle flaw: 4 of its 6 chunks ended with the next section’s heading. For example, one chunk ended “…checks the parcel. ## Shipping”, while the shipping rules sat in the following chunk. The heading attached itself to the wrong text, which can mislead retrieval for questions that mention “shipping”.

Tip: always print a sample of your chunks and read them. Our orphan heading problem was invisible in the averages and obvious after reading six chunks.

5. Heading-based chunks

Structured documents carry their own boundaries. Splitting on Markdown headings gave one chunk per section: 5 chunks, no mid-sentence cuts, no orphan headings and all facts intact. Each chunk starts with its own heading, which gives the embedding useful context.

The catch is size control. One section reached 113 tokens, over our 100-token target, and real documents have sections ranging from a line to several pages. In practice you combine methods: split by headings first, then apply recursive or sentence splitting inside any section that is too large, and repeat the heading at the top of each piece.

RAG chunking strategies compared

Strategy Chunks Avg / max tokens Stored tokens Cut mid-sentence Orphan heading Facts intact
Fixed 100 5 89 / 100 445 3 1 7 of 8
Fixed 100 + 20 overlap 6 91 / 100 545 5 0 8 of 8
Sentence packing 5 88 / 100 440 0 2 8 of 8
Recursive 6 74 / 84 441 0 4 8 of 8
By heading 5 89 / 113 445 0 0 8 of 8

This is one short document, so treat the table as an illustration of failure modes rather than a benchmark. The pattern is still instructive: the more a method respects the document’s own structure, the fewer meaning-breaking cuts it makes.

What about semantic chunking?

Semantic chunking embeds each sentence and starts a new chunk where the meaning shifts, measured by a drop in similarity between neighbouring sentences. It needs no headings and can follow topic changes in plain prose. It also costs an embedding call per sentence, and published comparisons disagree on whether it beats simpler methods once the whole RAG pipeline is measured. We did not run it here. Try it only after a structure-aware baseline, and keep it only if your own evaluation shows a gain.

How to choose chunk size and overlap

  1. Start with structure. Split by headings or document sections when they exist.
  2. Cap the size. A common starting range is a few hundred tokens per chunk, around 200 to 500, then tune.
  3. Prefer clean boundaries over overlap. Add 10 to 20% overlap only if facts still straddle cuts.
  4. Attach metadata. Store the title, section heading and source URL with each chunk for filtering and citations.
  5. Measure retrieval. Write 20 to 50 real questions with known answers and check whether the right chunk ranks in the top results. Change one setting at a time.
  6. Count tokens properly with the tokenizer your models use, as explained in our LLM tokens guide.

Store the chunks in a database that supports metadata filters, as covered in our vector database guide.

Final thoughts

RAG chunking strategies are where many retrieval problems begin, and where they are cheapest to fix. Our test showed each method’s typical failure: fixed-size cuts facts in half, overlap costs storage, sentence packing and recursive splitting can strand headings, and heading-based chunks need a size cap. Start from your documents’ structure, fall back to sentence-aware splitting, read your chunks, and let a small evaluation set decide the rest.

Frequently asked questions

What is the best chunking strategy for RAG?

There is no universal best. For structured documents, split by headings and then cap size with recursive or sentence splitting. Validate the choice with your own retrieval tests.

What chunk size should I use for RAG?

A few hundred tokens, often around 200 to 500, is a common starting point. Smaller chunks are more precise, larger ones carry more context; tune with real questions.

Does chunk overlap help?

It can rescue facts that straddle a cut. In our test, 20 tokens of overlap kept all eight facts intact but stored about 22% more tokens and still cut sentences.

What is recursive chunking?

It splits on the largest separator first, such as blank lines, and only uses smaller separators like sentences or words for pieces that are still too large.

Is semantic chunking better than fixed-size chunking?

Not reliably. It follows topic shifts but costs extra embedding calls, and published results are mixed. Compare it against a simpler baseline on your own data.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
Software Development

HTTP Methods Explained: GET, POST, PUT, PATCH, DELETE, Tested on a Live Server

HTTP methods explained with a small server we ran: GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS, safe vs idempotent,…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Software Development

JavaScript Sort Array: Numbers, Objects, Strings and Why 10 Comes Before 5

How to sort an array in JavaScript correctly: numbers with a compare function, objects by one or more fields, strings…

Read article →
Software Development

Python String Methods: Cheat Sheet, Real Examples and the strip() Trap

Python string methods with tested examples: split, join, strip, replace, find, startswith, is-methods, casefold, partition and translate, plus the strip()…

Read article →
Software Development

Python zip() Explained: Loop Two Lists at Once, Unzip, strict=True and Traps

Python zip() pairs items from several lists in one loop. Tested examples for dicts, unzipping with *, strict=True, zip_longest, sliding…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
Software Development

UTF-8 Encoding Explained: How Text Becomes Bytes and How to Fix Garbled Text

UTF-8 encoding explained with byte-level examples we ran in Python: how characters become 1 to 4 bytes, why café turns…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Databases & SQL

Delete Duplicate Rows in SQL: 5 Safe Methods, Tested Step by Step

How to delete duplicate rows in SQL safely: find them, preview them, then remove them with ROW_NUMBER, MIN(id), EXISTS, rowid…

Read article →
Databases & SQL

SQL RANK vs DENSE_RANK vs ROW_NUMBER: Differences with Tested Examples

SQL RANK vs DENSE_RANK vs ROW_NUMBER in one table: how each handles ties, top-N per group, the second-highest salary trap…

Read article →
Software Development

JavaScript Optional Chaining (?.) Explained: 8 Examples, ?? Defaults and Traps

JavaScript optional chaining stops “Cannot read properties of undefined” errors. 8 tested examples for properties, arrays and methods, ?? vs…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Databases & SQL

SQL COALESCE Explained: Defaults, NULLIF, IFNULL vs ISNULL and Traps, Tested

SQL COALESCE explained with queries we ran in SQLite 3.50: first non-null values, empty strings, NULL in maths, NULLIF for…

Read article →
Databases & SQL

SQL Self Join Explained: Employee-Manager, Pairs and Row Comparisons, Tested

SQL self join explained with queries we ran in SQLite 3.50: employee and manager, earning more than the manager, unique…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →
Artificial Intelligence

Prompt Injection Explained: How AI Apps Get Tricked and How to Defend Them

Prompt injection explained for builders: direct and indirect attacks, why keyword filters fail (our test blocked 1 of 4), and…

Read article →
DevOps

DNS Records Explained: A, AAAA, CNAME, MX, TXT and TTL With Real Lookups

DNS records explained with lookups we ran: how resolution works, A and AAAA, CNAME rules, MX and null MX, TXT…

Read article →
DevOps

Feature Flags Explained: Safe Releases, Gradual Rollouts and Flag Hygiene

Feature flags explained with a rollout simulation we ran: flag types, percentage rollouts with stable hashing, kill switches, allow lists,…

Read article →
Test Automation

Test Data Generation With Python Faker: Realistic, Repeatable and Safe Data

Test data generation with Python Faker, run and measured: seeding for repeatable data, locales, unique values, domain-shaped records, edge cases…

Read article →
Software Testing

Shift-Left Testing Explained: Find Bugs Earlier Without Slowing the Team

Shift-left testing explained: what it means, why early defects are cheaper, practices from requirement reviews to CI checks, the tester's…

Read article →
Artificial Intelligence

LLM Structured Output Explained: Getting Reliable JSON From AI Models

LLM structured output explained with validation we ran in Pydantic: JSON mode and schemas, constrained decoding, the ways model output…

Read article →
Test Automation

API Contract Testing Explained: Catch Breaking Changes Before They Ship

API contract testing explained with a schema check we ran: what a contract is, consumer-driven contracts, Pact, OpenAPI, which changes…

Read article →
Software Testing

Risk-Based Testing Explained: Score Risks, Build a Matrix and Test What Matters

Risk-based testing explained step by step: identifying risks, scoring likelihood and impact, building a risk matrix, mapping risk to test…

Read article →