semchunk

Using a sophisticated set of heuristics, semchunk breaks long documents into smaller chunks while preserving as much context as possible. This allows high-information chunks to be passed to AI models for processing, including models that either do not support long context windows or perform worse when overloaded with too much context.

semchunk prioritizes whitespace first, then sentence-ending punctuation such as periods and question marks, and then punctuation commonly used to separate dependent clauses. As a result it often produces chunks that keep whole paragraphs and sentences intact, while remaining relatively inexpensive to run.

You can customize how chunking is performed by setting chunk size and chunk overlap ratio parameters in our API — or use semchunk directly in your own Python stack.

Heuristic chunking can struggle with higher-level document structures, such as sections within a statute, chapters within a book, or even simple headings. This is where AI chunking and Kanon 2 Enricher come in.

semchunk’s AI chunking mode uses Kanon 2 Enricher to identify spans that correspond to structural elements in a document. Those spans are then merged and decomposed as needed, producing chunks that stay within a specified chunk size.

semchunk is the world’s most popular semantic chunking algorithm, with over a million monthly downloads, and is used in Microsoft’s Intelligence Toolkit and Docling by IBM.

Over time, Isaacus has continued to improve the accuracy and efficiency of semchunk. A standard consumer PC can now chunk the entire Gutenberg Corpus — 18 books and 3 billion tokens — in around three seconds.