TokenPad

Recipe · RAG

Chunk text for retrieval

Split documents by real tokens with overlap and sentence boundaries, instead of by characters — which silently produces chunks that exceed the embedding limit.

The problem

You need to split documents into passages small enough to embed usefully and large enough to interpret on their own.

Splitting by character count produces chunks of wildly varying token length, some of which exceed the embedding model’s limit and get truncated — with no error and a wrong vector.

Pack sentences into token-sized chunks with overlappython
import re
import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

def split_sentences(text: str) -> list[str]:
    # Good enough for prose. For legal or medical text, use a real segmenter.
    return [s for s in re.split(r"(?<=[.!?])\s+(?=[A-Z0-9\"'(\[])", text) if s.strip()]

def chunk_text(text: str, size: int = 300, overlap: int = 45) -> list[str]:
    """Pack whole sentences up to a token ceiling, carrying overlap forward.

    Segmenting first and packing greedily protects the boundary, which matters
    more than uniform chunk size: a chunk starting mid-clause embeds badly and
    reads badly when it is later shown back to the model.
    """
    units = [(s, len(encoding.encode(s))) for s in split_sentences(text)]

    chunks: list[str] = []
    current: list[tuple[str, int]] = []
    current_tokens = 0

    for sentence, tokens in units:
        # A single sentence larger than the ceiling cannot be packed.
        if tokens > size:
            if current:
                chunks.append(" ".join(s for s, _ in current))
                current, current_tokens = [], 0
            chunks.append(sentence)
            continue

        if current_tokens + tokens > size:
            chunks.append(" ".join(s for s, _ in current))

            # Carry trailing sentences back in until the overlap budget is used.
            carried: list[tuple[str, int]] = []
            carried_tokens = 0
            for s, t in reversed(current):
                if carried_tokens + t > overlap:
                    break
                carried.insert(0, (s, t))
                carried_tokens += t

            current, current_tokens = carried, carried_tokens

        current.append((sentence, tokens))
        current_tokens += tokens

    if current:
        chunks.append(" ".join(s for s, _ in current))

    return chunks
Check the chunks before embedding thempython
def audit(chunks: list[str], size: int) -> None:
    """Catch the defects that make retrieval fail silently."""
    for i, chunk in enumerate(chunks):
        tokens = len(encoding.encode(chunk))

        if tokens > size:
            print(f"[{i}] {tokens} tokens — exceeds the ceiling, will be truncated")
        if tokens < 40:
            print(f"[{i}] {tokens} tokens — too small to interpret alone")
        if chunk[:1].islower():
            print(f"[{i}] starts mid-sentence — boundary cut through a clause")
        if re.match(r"^(this|that|it|they|the (former|latter|above))\b", chunk, re.I):
            print(f"[{i}] opens with an unresolvable reference")

Why it is written this way

Overlap is not optional

A sentence split across two chunks appears complete in neither, so neither embedding represents it and it becomes effectively unsearchable — a silent hole in the index. Ten to twenty percent overlap closes it.

Boundaries beat uniformity

Slicing at exact token indices gives perfectly even chunks that cut through clauses. Accepting variance in size to keep sentences intact is almost always the better trade.

Use the source structure where it exists

Markdown headings, HTML sections and code function boundaries are better split points than anything inferred from the text — the author already told you where the ideas begin.

What breaks the naive version

  • Splitting by characters. The embedding limit is in tokens, and a 1,000-character chunk is 250 tokens of prose or 400 of JSON.
  • Overlap above about twenty percent. Near-duplicate chunks start crowding each other out of results.
  • Indexing pointer-only chunks like "see above". They occupy a slot in every result set and say nothing.

Check your numbers