Semantic chunking

Semantic chunking cuts long documents into pieces that each cover one topic, before an AI assistant searches them. Such an assistant does not read your whole handbook for every question: it looks up the few pieces that match the question. If a piece stops halfway through a rule, the assistant can miss the part that matters. Semantic chunking places the cuts where the subject changes.

How does it work?

A RAG system splits documents into chunks, turns each chunk into an embedding and retrieves the chunks closest in meaning to the question. The simplest splitter cuts after a fixed number of tokens or characters, often with some overlap between neighbouring chunks.

Semantic chunking in the narrow sense comes from Greg Kamradt's tutorial 5 Levels of Text Splitting, a video from January 2024 with a notebook:

1. Split the text into sentences.
2. Embed each sentence together with its neighbours, a window of three sentences.
3. Compute the cosine distance between each window and the next.
4. Start a new chunk wherever the distance is above the 95th percentile of all distances. That percentile is the one parameter to tune.

LlamaIndex adopted the method as SemanticSplitterNodeParser, and LangChain as SemanticChunker. Broader variants cut at headings and paragraphs, group sentences with similar embeddings even when they are not adjacent, or let a language model choose the cut points.

Does it retrieve better?

The measurements are mixed:

• Qu et al. (Vectara, October 2024) compared fixed-size chunking with two semantic chunkers on document retrieval, evidence retrieval and answer generation. Semantic chunking helped on artificially stitched documents with many topics, but "on non-synthetic datasets that better reflect real-world documents, fixed-size chunking often performed better". The extra computation was not justified by consistent gains, and the quality of the embeddings often mattered more than the chunking.
• In Chroma's evaluation (July 2024), strategies differed by up to 9% in recall, the share of the relevant text that retrieval finds. Kamradt's chunker with default settings scored slightly below average, with a recall of 83.6% and chunks of about 660 tokens on average. A simple recursive splitter with 200-token chunks and no overlap was consistently strong. The highest recall came from two new methods: a chunker that groups small pieces of text by embedding similarity (91.3%) and one that asks GPT-4o where to cut (91.9%).
• The unit you index matters in itself. In Dense X Retrieval (Chen et al., 2023), indexing short self-contained statements, called propositions, retrieved better than indexing passages.

Where does it go wrong?

• Cost. Every sentence window needs an embedding before anything is indexed, and an LLM chunker needs model calls: Chroma notes that its LLM chunker can take tens of minutes on a corpus.
• Tuning. LlamaIndex warns that its sentence splitting works mainly for English and that the percentile threshold may need tuning.
• Uncontrolled size. The threshold is relative, a percentile of all distances, so you do not set the chunk length. Chroma notes that this can lead to larger chunks in bigger corpora, while each chunk still has to fit what the embedding model accepts.
• No general answer. Kamradt himself writes that success is domain-specific and that chunking should be tested in retrieval evaluations.

Example

Suppose an accounting firm puts its terms and conditions into an AI assistant. A splitter that cuts every 300 words ends one chunk in the middle of the article on late payment, so the interest rate lands in the next chunk. A question about interest on late invoices may then retrieve the half without the rate. A splitter that follows the article headings keeps the article together, and so does a semantic splitter if the change of topic crosses its threshold. Which approach retrieves better on the firm's own documents is something to test with real client questions.