Classify Incoming Documents with Python and AI

By the end of this guide you will have a script that watches one folder, reads the first page of every PDF that lands in it, decides whether it is an invoice, a contract, a receipt, a CV or something else, moves the file into the matching subfolder, and writes one line to a log for every decision it made. Budget about an hour, and you will spend more of it choosing your labels than writing code.

This is one guide in AI Document Processing with Python, which covers what to do with a folder of PDFs once your business starts receiving more of them than one person can open. Classification is the first job in that chain: once a file knows what it is, the specialist scripts can take over — invoices go to Extract Invoice Data from PDFs with Python and AI, long reports go to Summarize Long PDFs with Python and Chunking.

Prerequisites

You need Python 3.10 or newer inside a virtual environment. If that phrase is new, Create a Python Virtual Environment for AI walks through it. Beyond the tools the main guide already uses, this script needs a token counter so you can measure the snippet you send:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install "openai>=1.40" "pypdf>=5.0" "tiktoken>=0.7" "python-dotenv>=1.0"
pip freeze > requirements.txt

Put your API key in a .env file next to the script:

OPENAI_API_KEY=sk-your-key-here

Then add that file to .gitignore before your first commit, because a key in a public repository gets found and spent by strangers:

echo ".env" >> .gitignore

Finally, create the folders the script will use. inbox is where new files arrive; sorted is where they end up:

mkdir -p inbox sorted

Steps 1 to 4 build a single file, sort_inbox.py, in the order they appear: the label set first, then the text reader, then the model call, then the routing and logging. Each block adds to the ones above it, and where a block ends with if __name__ == "__main__": it replaces the previous one so you can run the file at every stage with python sort_inbox.py.

Step 1 — Write down the label set and give it an escape hatch

A classifier is only as good as its list of choices. Write the list in Python, not in your head, and give each label a one-sentence definition — that sentence goes straight into the prompt, so it is the actual instruction the model follows.

Two rules make the difference between a classifier you can trust and one you cannot. First, labels must not overlap: if "receipt" and "invoice" both mean "a document about money", the model will flip between them and your accuracy will look random. Say what distinguishes them — an invoice asks for payment, a receipt confirms payment already happened. Second, always include an other label. A model asked to choose from four options with no way out will always choose one of the four, even for a parking permit, and it will sound sure about it. The escape hatch converts a silent wrong answer into an obvious one you can act on.

LABELS = {
    "invoice": "a request for payment that is not yet paid; has a due date, an amount owed and a bill-to party",
    "contract": "an agreement between named parties with clauses, obligations or signature blocks",
    "receipt": "proof that a payment already happened; shows a paid total, a card or transaction reference",
    "cv": "a person's career history: name, roles, dates, education, skills",
    "other": "anything that is not clearly one of the labels above, or text too garbled to judge",
}

Keep the set small. Five labels is a comfortable ceiling for a single pass; if you genuinely need fifteen, group them into five families first and run a second, narrower classification inside whichever family the file landed in. Adding a label later costs you one line here and one re-run of the accuracy check in Step 5.

Step 2 — Send the first page only, not the whole document

Feeding a forty-page contract to the model to learn that it is a contract wastes money on every single file, and the cost grows with the length of the document. It is also unnecessary. Documents announce themselves at the top: the letterhead, the title line, the bill-to block and the first paragraph carry nearly all the signal about what kind of document you are holding.

So extract text from the first page or two, cut it to a fixed token budget, and send that. A token is the unit that language models bill in — a chunk of text averaging about three-quarters of a word. Capping the snippet at 800 tokens means a one-page receipt and a two-hundred-page tender document cost you exactly the same to classify.

from pathlib import Path

import tiktoken
from pypdf import PdfReader

ENCODER = tiktoken.encoding_for_model("gpt-4o-mini")


def first_tokens(path: Path, budget: int = 800) -> str:
    """Return the opening ~800 tokens of a PDF's text, or '' if there is none."""
    reader = PdfReader(path)
    pages = reader.pages[:2]                      # page one usually decides it
    text = "\n".join(page.extract_text() or "" for page in pages)
    text = " ".join(text.split())                 # collapse the ragged whitespace
    tokens = ENCODER.encode(text)
    return ENCODER.decode(tokens[:budget])


if __name__ == "__main__":
    sample = next(Path("inbox").glob("*.pdf"))
    snippet = first_tokens(sample)
    print(f"{sample.name}: {len(ENCODER.encode(snippet))} tokens")
    print(snippet[:300])

Run that on a handful of your own files before going further. If a file prints 0 tokens, it is a scan — an image of a page rather than text — and no amount of prompting will fix it; Step 5's troubleshooting section covers what to do with those. The same trimming idea, applied to whole documents rather than first pages, is covered in Split Long Text into Chunks for AI APIs.

Step 3 — Ask for a label and a confidence in JSON

Ask for prose and you get prose you have to parse. Ask for JSON — a plain data format of keys and values that Python reads with one function — and you get something your script can branch on. The OpenAI SDK has a switch for this: response_format={"type": "json_object"} forces valid JSON out of the model, which removes a whole category of parsing bugs.

Request three fields. The label is the decision. The confidence is a number from 0 to 1 that tells you how much the model wants to be argued with. The reason is a handful of words naming the evidence — "due date and amount owed" — and it is what makes the log readable to a colleague six months later.

Never trust the reply blindly, even in JSON mode. The model can return a label you never defined or a confidence as a string. Validate both, and fall back to other with a confidence of zero rather than letting a surprise value flow into your folder logic.

import json
import os

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()  # reads .env, which is in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

LABEL_LINES = "\n".join(f"- {name}: {meaning}" for name, meaning in LABELS.items())

SYSTEM_PROMPT = f"""You sort business documents. You will be shown the opening text of one file.
Choose exactly one label from this list:
{LABEL_LINES}

Reply with JSON only, in this shape:
{{"label": "one label from the list", "confidence": 0.0, "reason": "under 8 words"}}

Set confidence to how sure you are, from 0.0 to 1.0.
Use "other" whenever two labels fit equally well, or the text is unreadable.
Never invent a label that is not on the list."""


def classify(snippet: str) -> dict:
    """Return {'label': str, 'confidence': float, 'reason': str} for one snippet."""
    if not snippet.strip():
        return {"label": "other", "confidence": 0.0, "reason": "no extractable text"}

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": snippet},
        ],
    )
    data = json.loads(response.choices[0].message.content)

    label = data.get("label", "other")
    if label not in LABELS:                      # the model invented something
        label = "other"
    try:
        confidence = float(data.get("confidence", 0.0))
    except (TypeError, ValueError):
        confidence = 0.0

    return {
        "label": label,
        "confidence": max(0.0, min(1.0, confidence)),
        "reason": str(data.get("reason", ""))[:80],
    }

temperature=0 asks for the least adventurous answer the model has, which is what you want when the same file should always sort the same way. If you want to know why that single argument matters so much here, What Is Temperature in an LLM API? explains it, and Write System Prompts that Control Output Format covers writing instructions that hold their shape across thousands of calls.

Step 4 — Route on the confidence score, then file and log

Here is where most home-made classifiers go wrong: they act on every answer as if it were equally good. Your script has a number that tells it which answers to doubt, so use it. Pick two thresholds — one above which the file is filed silently, one below which no automatic action is taken at all — and treat the band in between as "filed, but flagged for a spot check".

The other label short-circuits all of that. Whatever confidence the model reports, an other file goes to the review queue, because "I could not tell" is a request for a human, not a filing instruction.

The log is not optional paperwork. It is the only way to answer "why is this contract in the receipts folder?" without re-running anything, and it is the raw material for tuning your thresholds later. Write one CSV row per file, with the timestamp, the original name, the label, the score, the model's reason and where the file went.

import csv
import shutil
from datetime import datetime, timezone
from pathlib import Path

INBOX = Path("inbox")
SORTED_ROOT = Path("sorted")
LOG_FILE = Path("classification_log.csv")

AUTO_FILE = 0.85       # at or above this, no human ever sees it
NEEDS_REVIEW = 0.60    # below this, no automatic filing at all

FIELDS = ["ts", "source", "label", "confidence", "reason", "action", "destination"]


def route(label: str, confidence: float) -> tuple[str, Path]:
    """Decide what happens to a file, given the model's answer."""
    if label == "other" or confidence < NEEDS_REVIEW:
        return "review", SORTED_ROOT / "_review"
    if confidence < AUTO_FILE:
        return "flagged", SORTED_ROOT / label
    return "auto", SORTED_ROOT / label


def log_decision(row: dict) -> None:
    is_new = not LOG_FILE.exists()
    with LOG_FILE.open("a", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=FIELDS)
        if is_new:
            writer.writeheader()
        writer.writerow(row)


def process(path: Path) -> dict:
    result = classify(first_tokens(path))
    action, folder = route(result["label"], result["confidence"])
    folder.mkdir(parents=True, exist_ok=True)

    target = folder / path.name
    if target.exists():                                   # two files, same name
        stamp = datetime.now(timezone.utc).strftime("%H%M%S")
        target = folder / f"{stamp}-{path.name}"
    shutil.move(str(path), str(target))

    row = {
        "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "source": path.name,
        "label": result["label"],
        "confidence": round(result["confidence"], 2),
        "reason": result["reason"],
        "action": action,
        "destination": str(target),
    }
    log_decision(row)
    return row


if __name__ == "__main__":
    for pdf in sorted(INBOX.glob("*.pdf")):
        row = process(pdf)
        print(f"{row['source'][:34]:<36} {row['label']:<9} {row['confidence']:.2f}  {row['action']}")

Drop a dozen files into inbox, run it, and you have sorted folders plus classification_log.csv. Notice that nothing is ever deleted and nothing is overwritten — a same-named file gets a time prefix. When you later run this on a schedule, Schedule Python AI Jobs with GitHub Actions shows how, and Log and Monitor AI API Calls in Production covers keeping the log useful once it has thousands of rows.

Step 5 — Measure accuracy on twenty documents you already know

You cannot tune a classifier you have not measured, and "it looked right when I tried it" is not a measurement. Twenty files is enough to expose the problems that matter — a whole label that never fires, a pair of types the model keeps swapping — without turning into a research project.

Gather twenty real documents, roughly four per label, and write the answers down yourself in a two-column CSV called labels.csv:

file,expected
samples/acme-march.pdf,invoice
samples/nda-2026.pdf,contract
samples/taxi-0412.pdf,receipt
samples/j-okafor-cv.pdf,cv
samples/parking-permit.pdf,other

Then run every file through the same classify function your live script uses. Save this second script as check_accuracy.py next to sort_inbox.py, so it imports the exact prompt and label set you are running in production:

import csv
from collections import Counter
from pathlib import Path

from sort_inbox import classify, first_tokens


def accuracy_check(csv_path: str = "labels.csv") -> None:
    with open(csv_path, newline="", encoding="utf-8") as handle:
        rows = list(csv.DictReader(handle))

    correct, mistakes = 0, Counter()
    wrong_scores = []

    for row in rows:
        result = classify(first_tokens(Path(row["file"])))
        got, expected = result["label"], row["expected"]
        if got == expected:
            correct += 1
        else:
            mistakes[f"{expected} -> {got}"] += 1
            wrong_scores.append(result["confidence"])
            print(f"MISS {row['file']}: wanted {expected}, got {got} "
                  f"({result['confidence']:.2f}) — {result['reason']}")

    print(f"\n{correct}/{len(rows)} correct")
    for pair, count in mistakes.most_common():
        print(f"  confused {pair}: {count}x")
    if wrong_scores:
        print(f"  highest confidence on a wrong answer: {max(wrong_scores):.2f}")


if __name__ == "__main__":
    accuracy_check()

That last line is the most valuable number the check produces. It tells you where to put AUTO_FILE: set the threshold above the highest score the model gave a wrong answer, and those mistakes stop being filed silently. If the misses all fall on one pair — receipts read as invoices, say — do not raise the threshold, sharpen that pair of definitions in LABELS and run the twenty files again. Re-run this check whenever you add a label, change the prompt or switch models; it takes a minute and it is the only honest comparison between gpt-4o-mini and gpt-4o for your own documents.

Confidence bands and routing

Troubleshooting

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) — the model answered with prose, or with JSON wrapped in a code fence. The cause is almost always a missing response_format={"type": "json_object"} on the call. Add it, and keep the word "JSON" in your system prompt, which the API requires when that mode is on. Fix JSONDecodeError with AI API Responses in Python covers the stubborn cases.

pypdf.errors.PdfReadError: EOF marker not found — the PDF is truncated or was still being copied when your script picked it up. Wrap the first_tokens call in a try block, log the failure, and move the file to sorted/_review/ rather than letting one bad file stop the whole run.

Every file comes back as other with a low score — check what first_tokens actually returned. Scanned documents are images of pages, so extract_text() returns an empty string and the model correctly says it cannot tell. Those files need optical character recognition before classification; route them to the review folder and handle them as a separate batch.

openai.RateLimitError: Error code: 429 — you are sending files faster than your account allows. Add a short time.sleep between calls or retry with a growing delay; Fix the 429 Rate-Limit Error in Python has a ready-made backoff loop you can paste in.

When to use this vs. alternatives

• Filename or keyword rules beat AI when your documents come from a handful of known senders with stable templates — matching INV- in a filename is free, instant and perfectly accurate for those. They fall apart the moment a supplier redesigns a template or a new sender appears, which is exactly the situation this script is built for.
• Embeddings with a nearest-neighbour lookup suit a large, fixed set of categories where you have many labelled examples already; the per-file cost is lower than a chat call. You lose the plain-language reason, though, and you cannot add a category by typing one sentence.
• A dedicated document-understanding service is worth the integration work when you need page-level bounding boxes, handwriting or table structure. For "which of five buckets does this belong in", it is more setup than the job requires.

In practice the two combine well. Run your cheap filename rules first, and only call the model for the files no rule matched — you keep the low cost on the predictable majority and the flexibility on the awkward remainder. Once files are sorted, the rest of this series takes over: send the invoices onward for field extraction, and point Build a Document Question-Answering Tool at the contracts folder so people can ask questions of what you have filed. Back to AI Document Processing with Python.

Related guides

• Extract Invoice Data from PDFs with Python and AI — the natural next step for everything that lands in the invoice folder.
• Build a Document Question-Answering Tool — let colleagues ask questions of the documents you just sorted.
• Rename and Sort Files with Python and AI — the same idea applied to messy file names rather than document types.
• Count Tokens in Python Before You Send — measure your 800-token snippet before it becomes a bill.
• Python Script to Automate Email Sorting — classify messages arriving in an inbox instead of files in a folder.