Split Long Text into Chunks for AI APIs

By the end of this guide you will have a single chunk_text() function, roughly sixty lines of plain Python, that turns any long document into a list of chunk records. Each record carries the text, a token count, the source filename, its position in the sequence, and the character range it came from — enough to feed a model and still cite the original. It takes about thirty minutes to build and needs no framework, only tiktoken and Python's built-in re module.

Chunking is a data-preparation job, so it sits in the Data Cleaning for AI section alongside Cleaning CSV Data with Pandas for AI. Clean your text first — strip HTML, collapse stray whitespace — then chunk it. Chunking messy text just gives you messy chunks.

Prerequisites

You need Python 3.10 or newer inside a virtual environment (an isolated folder holding this project's packages). If you have not made one yet, Create a Python Virtual Environment for AI walks through it. Then install the two packages this guide uses:

pip install "tiktoken>=0.7" "openai>=1.40" "python-dotenv>=1.0"

tiktoken is OpenAI's tokenizer library: it converts text into the exact numeric pieces the model bills you for, so you can count them locally without sending anything. You only need an API key for the final step, where chunks are actually sent somewhere. Put it in a .env file:

OPENAI_API_KEY=sk-your-key-here

Then add that file to .gitignore before your first commit, so the key never reaches a public repository:

echo ".env" >> .gitignore

Finally, save a long plain-text file next to your script as report.txt. Anything over a few thousand words works — a transcript, a policy document, a scraped article.

Step 1 — Measure the text in tokens, not characters

A token is the unit models actually read: a common word is usually one token, a rare or compound word splits into two or three, and punctuation often gets its own. In English prose one token averages about three-quarters of a word, but that average collapses the moment your text contains code, URLs, tables or non-English names, where a single "word" can cost six or seven tokens.

That is why character counts mislead. Two 1,000-character passages can differ by fifty per cent in token cost. Since the model's input limit and your bill are both measured in tokens, measure in tokens too. Start by pinning down a counter you will reuse everywhere:

import tiktoken

try:
    ENC = tiktoken.encoding_for_model("gpt-4o-mini")
except KeyError:
    ENC = tiktoken.get_encoding("o200k_base")


def count_tokens(text: str) -> int:
    """Exact token count for the model family above."""
    return len(ENC.encode(text))


if __name__ == "__main__":
    from pathlib import Path

    document = Path("report.txt").read_text(encoding="utf-8")
    print("characters:", len(document))
    print("tokens:    ", count_tokens(document))
    print("ratio:     ", round(len(document) / count_tokens(document), 2))

Run it and note the ratio for your own material. If it lands near 4.0 characters per token you have ordinary prose; if it drops toward 2.5 your text is dense with symbols and you should budget more conservatively. Keep this number handy — it is the same measurement used in Count Tokens in Python Before You Send and in Estimate OpenAI API Costs with Python.

Step 2 — Split on natural boundaries, not a fixed count

The instinct is to slice every N characters. Resist it. A blind slice cuts words in half, orphans a heading from the paragraph it introduces, and can end a chunk on "The refund policy does not apply when" — a fragment that means the opposite of the full sentence.

Instead, work in three tiers. Split the document on blank lines to get paragraphs, because an author already marked those as complete thoughts. If a paragraph is still too large, split it on sentence endings. If a single sentence somehow exceeds your budget — a wall of comma-separated items, say — fall back to a word-count slice as the last resort. Then pack those pieces back together until each group is as close to your target size as it can get without going over.

Here are the splitting tiers and the packing loop. Add them below count_tokens:

import re


def split_paragraphs(text: str) -> list[str]:
    """Blank lines usually mark a complete thought."""
    return [p.strip() for p in re.split(r"\n\s*\n", text.strip()) if p.strip()]


def split_sentences(paragraph: str) -> list[str]:
    """Break after . ! or ? followed by whitespace."""
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", paragraph.strip()) if s.strip()]


def to_units(text: str, max_tokens: int) -> list[str]:
    """Smallest pieces that each fit inside max_tokens, in document order."""
    units: list[str] = []
    for para in split_paragraphs(text):
        if count_tokens(para) <= max_tokens:
            units.append(para)
            continue
        for sentence in split_sentences(para):
            if count_tokens(sentence) <= max_tokens:
                units.append(sentence)
            else:
                words = sentence.split()
                for i in range(0, len(words), 40):
                    units.append(" ".join(words[i:i + 40]))
    return units

The sentence pattern is deliberately simple. It will split "Dr. Alvarez" in the wrong place, which costs you nothing in practice: the two halves land in the same chunk almost every time, and even when they do not, the overlap you add next covers the seam.

Step 3 — Add overlap so facts survive the boundary

Suppose a chunk boundary lands between "The warranty covers parts and labour for two years." and "It does not cover accidental damage." Ask "does the warranty cover accidental damage?" and retrieval finds the second chunk, which never names the warranty, or the first, which never mentions damage. The answer is technically present in your document and unreachable in your index.

Overlap fixes this by repeating the tail of each chunk at the head of the next. Ten to fifteen per cent of your chunk size is the useful range: about 50 tokens on a 400-token chunk, or 150 on a 1000-token chunk. Below ten per cent the seam is still exposed; above about twenty per cent you are paying to store and embed the same sentences several times for very little extra recall.

Implement overlap as a helper that hands back the trailing pieces of the chunk you just closed, so the next chunk starts with them already in place:

def overlap_tail(units: list[str], overlap_tokens: int) -> list[str]:
    """Return the last whole units that fit inside overlap_tokens."""
    kept: list[str] = []
    total = 0
    for unit in reversed(units):
        cost = count_tokens(unit)
        if total + cost > overlap_tokens:
            break
        kept.insert(0, unit)
        total += cost
    return kept

Because it keeps only whole paragraphs or sentences, the repeated text always reads as language rather than a severed fragment. That strictness has a catch worth knowing about: if every piece in the chunk is larger than the overlap budget, this function returns nothing and you get no overlap at all. Long-paragraph documents hit that case constantly, so the finished version in the next step adds a fallback that carries the trailing sentences of the last piece instead.

One more side effect: a chunk can finish slightly above max_tokens, since the carried text is already sitting in the buffer when the next piece arrives. Leave ten per cent of headroom below the model's real limit and this never bites.

Step 4 — Keep metadata so you can cite the original

A bare list of strings is almost useless three weeks later. Once chunks are embedded and searched, you need to answer "where did this come from?" — to show a page reference, to re-chunk one file after it changes, or to drop every chunk from a document you deleted. Store four fields alongside the text: the source name, the chunk's position in the sequence, the character range in the original file, and the token count.

The character range is the one people skip and later regret. With it, you can highlight the exact passage in the source document instead of showing a paraphrase. Here is the complete function. Each piece is now a (text, start, end) tuple, and carry_over is the Step 3 helper with the sentence fallback added:

from dataclasses import dataclass

Piece = tuple[str, int, int]  # text, start character, end character


@dataclass
class Chunk:
    text: str
    index: int
    source: str
    start_char: int
    end_char: int
    n_tokens: int


def locate(units: list[str], document: str) -> list[Piece]:
    """Attach each unit's character range, scanning forward through the source."""
    located, cursor = [], 0
    for unit in units:
        start = document.find(unit, cursor)
        if start == -1:
            start = cursor
        end = start + len(unit)
        cursor = end
        located.append((unit, start, end))
    return located


def carry_over(pieces: list[Piece], overlap_tokens: int) -> list[Piece]:
    """Trailing pieces that fit the overlap budget, or the last piece's final sentences."""
    kept, total = [], 0
    for piece in reversed(pieces):
        cost = count_tokens(piece[0])
        if total + cost > overlap_tokens:
            break
        kept.insert(0, piece)
        total += cost
    if kept:
        return kept

    text, start, end = pieces[-1]
    sentences, total = [], 0
    for sentence in reversed(split_sentences(text)):
        cost = count_tokens(sentence)
        if total + cost > overlap_tokens:
            break
        sentences.insert(0, sentence)
        total += cost
    if not sentences:
        return []
    offset = text.rfind(sentences[0])
    return [] if offset == -1 else [(text[offset:], start + offset, end)]


def build_chunk(pieces: list[Piece], index: int, source: str) -> Chunk:
    text = "\n\n".join(p[0] for p in pieces)
    return Chunk(
        text=text,
        index=index,
        source=source,
        start_char=pieces[0][1],
        end_char=pieces[-1][2],
        n_tokens=count_tokens(text),
    )


def chunk_text(document: str, source: str, max_tokens: int = 500,
               overlap_ratio: float = 0.12) -> list[Chunk]:
    """Split a document into overlapping, boundary-aware, metadata-carrying chunks."""
    overlap_tokens = int(max_tokens * overlap_ratio)
    pieces = locate(to_units(document, max_tokens), document)

    chunks: list[Chunk] = []
    current: list[Piece] = []
    running = 0

    for piece in pieces:
        cost = count_tokens(piece[0])
        if current and running + cost > max_tokens:
            chunks.append(build_chunk(current, len(chunks), source))
            current = carry_over(current, overlap_tokens)
            running = sum(count_tokens(p[0]) for p in current)
        current.append(piece)
        running += cost

    if current:
        chunks.append(build_chunk(current, len(chunks), source))
    return chunks


if __name__ == "__main__":
    from pathlib import Path

    doc = Path("report.txt").read_text(encoding="utf-8")
    for chunk in chunk_text(doc, source="report.txt", max_tokens=500):
        preview = chunk.text[:60].replace("\n", " ")
        print(f"[{chunk.index:>3}] {chunk.n_tokens:>4} tok "
              f"chars {chunk.start_char}-{chunk.end_char}  {preview}...")

Run it and read the output. Token counts should sit in a tight band just under your target rather than swinging wildly, and each chunk's start_char should be a little lower than the previous chunk's end_char — that gap is your overlap, visible as a number.

Step 5 — Match chunk size to the model and the task

Two separate limits pull chunk size in opposite directions. The context window is the total tokens a model will accept in one request, and it caps how many chunks you can send at once: with a 500-token chunk and ten retrieved results, you are spending 5,000 tokens before the question is even asked. That pressure pushes chunks smaller.

Retrieval quality pushes the other way, but not linearly. Small chunks are precise — one idea each, so a matching search hits it cleanly — yet they lose the surrounding argument, and an answer that needs two paragraphs of reasoning now has to reassemble it from fragments. Large chunks preserve reasoning but dilute the match, because the one relevant sentence is averaged in with a thousand tokens of neighbours.

Once the size is settled, sending chunks is ordinary API work. This loop summarises each one and keeps the metadata attached to the result:

import os
from pathlib import Path

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()  # reads .env, which is listed in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

doc = Path("report.txt").read_text(encoding="utf-8")
summaries = []

for chunk in chunk_text(doc, source="report.txt", max_tokens=500):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarise the passage in one sentence."},
            {"role": "user", "content": chunk.text},
        ],
    )
    summaries.append({
        "index": chunk.index,
        "chars": (chunk.start_char, chunk.end_char),
        "summary": response.choices[0].message.content,
    })

for item in summaries:
    print(item["index"], item["chars"], item["summary"])

Because every summary still knows its character range, you can trace any sentence in the final output back to the paragraph that produced it. That is the same pattern used in Summarize Long PDFs with Python and Chunking.

Quick reference: chunk size and what it suits

The first three rows assume you will retrieve several chunks per question; the last assumes you process every chunk exactly once.

Troubleshooting

ModuleNotFoundError: No module named 'tiktoken' — the package went into a different interpreter than the one running your script. Confirm your virtual environment is active, then reinstall with python -m pip install tiktoken so pip and Python are guaranteed to be the same installation.

KeyError: 'Could not automatically map ... to a tokeniser' — your tiktoken version predates the model name you passed. The try/except KeyError in Step 1 already handles this by falling back to a named encoding; if you removed it, either restore it or upgrade with pip install -U tiktoken.

This model's maximum context length is ... tokens — your chunk plus the system prompt plus the reply exceeded the window. Overlap can push a chunk above max_tokens, so lower the target and leave headroom for the model's own output. Fix the Context-Length-Exceeded Error in Python covers the arithmetic in detail.

Chunks come back with wildly different token counts — your document has no blank lines, so split_paragraphs returns one giant piece and everything falls through to the sentence tier. Normalise line endings first with document.replace("\r\n", "\n"), and if the source is HTML, convert it to text with real paragraph breaks before chunking.

When to use this vs. alternatives

• Use this hand-written function when you want to see and tune the boundaries, when you need custom metadata fields, or when you are chunking a format with structure worth respecting, such as Markdown headings or transcript timestamps. Sixty lines you understand beat a black box you do not.
• Use a framework's built-in splitter when chunking is incidental to a larger pipeline you are already building on that framework, and its defaults are good enough. The trade is convenience for control: you inherit its boundary rules and its metadata shape.
• Skip chunking entirely when your whole document comfortably fits the model's context window and you only ask one question of it. Chunking exists to work around a limit; if you are not near the limit, sending the full text is simpler and often gives better answers.

Chunking is the step that decides how good every later answer can be, so it is worth the half hour. Once your chunks are clean and consistent, the natural next moves are removing near-identical pieces — see Remove Duplicate Records with Embeddings in Python — and wiring the result into a retrieval system, as in Connect a Chatbot to Your Docs with RAG. Back to Data Cleaning for AI.

Related guides

• Remove Duplicate Records with Embeddings in Python — strip near-identical chunks before you embed them twice.
• Count Tokens in Python Before You Send — the token measurement this guide depends on, explained on its own.
• Cleaning CSV Data with Pandas for AI — tidy the text before you cut it into pieces.
• Build a Document Question-Answering Tool — put these chunks to work answering questions over your own files.
• Fix the Context-Length-Exceeded Error in Python — what to do when a chunk still overflows the window.