Find Content Gaps with Python and AI

In about thirty minutes you will have a script that reads your sitemap and a competitor's, turns both into lists of topics, finds every topic of theirs that has no close match on your site, asks a model how relevant each one is to your readers, and writes a ranked CSV you can open in a spreadsheet and triage over coffee. No SEO subscription, no scraping hundreds of pages, one modest API bill.

The trick is to compare meaning rather than words. If they publish "shipping rates for small parcels" and you published "how much does postage cost", a word-matching script calls that a gap and wastes your afternoon. An embedding — a list of numbers that represents what a phrase means — puts those two titles close together, so the script keeps quiet. Only the topics with nothing similar on your side survive to the shortlist.

This guide sits inside SEO Keyword Research with Python. It picks up where Python Script for Competitor Keyword Analysis stops: that guide works phrase by phrase inside page copy, this one works page by page across whole sites.

Prerequisites

You need Python 3.10 or newer in a virtual environment. If that phrase is unfamiliar, Create a Python Virtual Environment for AI sets one up in five minutes, then come back.

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install "httpx>=0.27" "openai>=1.40" "numpy>=1.26" "python-dotenv>=1.0"
pip freeze > requirements.txt

There is no XML library to install — xml.etree.ElementTree ships with Python. httpx fetches the sitemaps, numpy does the similarity maths in one line, and the openai SDK handles both the embeddings and the scoring call.

Put your key in a .env file beside the script:

OPENAI_API_KEY=sk-your-key-here

Then keep it out of version control before you write another line:

echo ".env" >> .gitignore

You also need two sitemap URLs. Most sites expose theirs at /sitemap.xml, and /robots.txt names it if they do not. Check the competitor's robots.txt for crawl rules while you are there and honour them.

Step 1 — Build two lists of topics from sitemap.xml

A sitemap is plain XML: a <urlset> full of <url> elements, each holding a <loc> with one page address. Big sites split theirs into a <sitemapindex> that points at several child sitemaps, so the fetcher below follows one level of nesting and stops. Every element is namespaced, which is why each findall call carries the sm: prefix and a namespace dictionary.

Rather than fetching every page for its <title>, we read the topic straight out of the URL slug. It costs nothing, it never annoys anyone's server, and slugs are written by humans to describe the page. Save this as gaps.py:

import time
import xml.etree.ElementTree as ET
from urllib.parse import urlparse

import httpx

SITEMAP_NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
HEADERS = {"User-Agent": "content-gap-finder/1.0 (contact: you@example.com)"}


def fetch_sitemap_urls(sitemap_url: str, limit: int = 300) -> list[str]:
    """Return page URLs from a sitemap, following one level of sitemap index."""
    urls: list[str] = []
    queue = [sitemap_url]
    with httpx.Client(headers=HEADERS, timeout=30.0, follow_redirects=True) as client:
        while queue and len(urls) < limit:
            response = client.get(queue.pop(0))
            response.raise_for_status()
            root = ET.fromstring(response.content)
            if root.tag.endswith("sitemapindex"):
                queue += [n.text.strip() for n in root.findall("sm:sitemap/sm:loc", SITEMAP_NS)]
            else:
                urls += [n.text.strip() for n in root.findall("sm:url/sm:loc", SITEMAP_NS)]
            time.sleep(1.0)  # be a polite visitor
    return urls[:limit]


def slug_to_topic(url: str) -> str:
    """Turn https://site.com/blog/cold-email-tips/ into 'cold email tips'."""
    path = urlparse(url).path.strip("/")
    if not path:
        return ""
    slug = path.split("/")[-1]
    for suffix in (".html", ".php", ".htm"):
        slug = slug.removesuffix(suffix)
    return slug.replace("-", " ").replace("_", " ").strip()


def topics_for(sitemap_url: str) -> list[tuple[str, str]]:
    """Return (topic, url) pairs, skipping empty slugs and duplicates."""
    seen: set[str] = set()
    pairs: list[tuple[str, str]] = []
    for url in fetch_sitemap_urls(sitemap_url):
        topic = slug_to_topic(url)
        if topic and topic not in seen:
            seen.add(topic)
            pairs.append((topic, url))
    return pairs


if __name__ == "__main__":
    mine = topics_for("https://your-site.example/sitemap.xml")
    theirs = topics_for("https://competitor.example/sitemap.xml")
    print(f"you: {len(mine)} topics, them: {len(theirs)} topics")
    print("sample of theirs:", [t for t, _ in theirs[:5]])

Run python gaps.py and check the sample output before going further. If the topics read like sentences, the slugs are good. If they read like p 4417, the site uses numeric URLs and you will need real page titles instead — fetch each page and read its <title> tag, or skip that competitor. Everything downstream treats a topic as a short string, so the source does not matter after this point.

Here is the whole pipeline you are about to build, so you can see where each step lands.

Step 2 — Embed every title once and cache the result

An embedding turns a phrase into a fixed-length list of numbers. Phrases that mean similar things get similar numbers, which is what lets you compare "invoice template" with "billing document example" and get a sensible answer. You send text, you get vectors back, and the vectors are what you compare from then on.

Two habits keep this cheap. Send titles in batches of a hundred rather than one at a time, and write the vectors to a JSON file so a second run of the script costs nothing. The same batching idea appears in Group Keywords with Python and Embeddings, and if you want to know what a run will cost before you press enter, Estimate OpenAI API Costs with Python shows the arithmetic.

Add this to gaps.py:

import json
import os
from pathlib import Path

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()  # reads .env, which is in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

EMBED_MODEL = "text-embedding-3-small"
CACHE = Path("embeddings.json")


def embed_all(texts: list[str]) -> list[list[float]]:
    """Embed a list of strings in batches of 100, reusing a local cache."""
    cache: dict[str, list[float]] = json.loads(CACHE.read_text()) if CACHE.exists() else {}
    missing = [t for t in texts if t not in cache]
    for start in range(0, len(missing), 100):
        batch = missing[start:start + 100]
        response = client.embeddings.create(model=EMBED_MODEL, input=batch)
        for text, item in zip(batch, response.data):
            cache[text] = item.embedding
        print(f"embedded {start + len(batch)}/{len(missing)} new titles")
    if missing:
        CACHE.write_text(json.dumps(cache))
    return [cache[t] for t in texts]

The cache is keyed by the exact title string, so re-running after adding one new competitor page charges you for one title. Delete embeddings.json if you ever switch embedding models — vectors from two different models are not comparable, and mixing them produces confident nonsense.

Step 3 — Match each of their topics to your nearest one

Cosine similarity measures the angle between two vectors and returns a number from -1 to 1, where 1 means "as close to identical in meaning as this model can tell". If you normalise every vector to length one first, cosine similarity is just a dot product, so the whole comparison becomes a single matrix multiplication instead of a nested loop over thousands of pairs.

For each of their topics you want two answers: the highest score against anything on your site, and which page of yours produced it. argmax gives you the index, max gives you the score, and both run along the same axis:

import numpy as np


def normalise(vectors: list[list[float]]) -> np.ndarray:
    matrix = np.array(vectors, dtype="float32")
    return matrix / np.linalg.norm(matrix, axis=1, keepdims=True)


def nearest_matches(their_topics: list[str], my_topics: list[str]):
    """For each of their topics, return (topic, best score, your closest topic)."""
    theirs = normalise(embed_all(their_topics))
    mine = normalise(embed_all(my_topics))
    similarity = theirs @ mine.T          # rows: theirs, columns: yours
    best_index = similarity.argmax(axis=1)
    best_score = similarity.max(axis=1)
    return [
        (their_topics[i], float(best_score[i]), my_topics[best_index[i]])
        for i in range(len(their_topics))
    ]

Now the judgement call: where do you draw the line? A best match of 0.92 means you have written that article. A best match of 0.31 means the topic is foreign to your site. The interesting band is the middle, where you have something adjacent but not equivalent. Print twenty rows sorted by score and read them before you commit to a number — the right cutoff depends on how narrow your niche is, and a site about one product will score everything higher than a general blog does.

Step 4 — Score the surviving gaps for relevance

The script now hands you every topic with no close match — and that list is longer than it should be, because "nothing like it on your site" and "worth your time" are different questions. A competitor's careers page, their pricing calculator, and their case study about a client in an industry you never serve will all sail through the similarity filter.

So ask a model, once per batch, in a strict format. Number the topics, describe your audience in one sentence, and demand JSON back. Setting temperature=0 makes repeated runs agree with each other, and response_format={"type": "json_object"} stops the model wrapping its answer in explanation. If you want more control over reply shapes, Write System Prompts that Control Output Format goes deeper.

SCORING_SYSTEM = (
    "You rate content ideas for a specific audience. "
    "You reply with JSON only and never add commentary."
)


def score_gaps(topics: list[str], audience: str, model: str = "gpt-4o-mini") -> list[dict]:
    """Score a batch of topics 1-5 for relevance to one audience."""
    numbered = "\n".join(f"{i}. {t}" for i, t in enumerate(topics, 1))
    user_prompt = (
        f"Audience: {audience}\n\n"
        "Rate each topic from 1 to 5 for how useful an article on it would be to that "
        "audience. 5 means they would act on it this week; 1 means it is irrelevant. "
        "Give a reason of at most eight words.\n"
        'Reply exactly as {"items": [{"n": 1, "score": 4, "reason": "..."}]}\n\n'
        f"Topics:\n{numbered}"
    )
    response = client.chat.completions.create(
        model=model,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SCORING_SYSTEM},
            {"role": "user", "content": user_prompt},
        ],
    )
    items = json.loads(response.choices[0].message.content)["items"]
    scored = []
    for item in items:
        index = int(item["n"]) - 1
        if 0 <= index < len(topics):
            scored.append({
                "topic": topics[index],
                "score": int(item["score"]),
                "reason": str(item["reason"]),
            })
    return scored

Two details in there earn their keep. The model is asked to echo the number n rather than the topic text, which means a rewritten or truncated title cannot silently reattach a score to the wrong row. And the loop ignores any index outside the batch, so a hallucinated item 47 in a batch of 20 is dropped instead of crashing the run. Keep batches to about twenty topics — long numbered lists are where models start skipping entries.

Step 5 — Write a CSV a human can triage

The output is a decision aid, not a verdict, so give the reader everything they need to overrule the script in one glance: the missing topic, its URL so they can go read it, the similarity score, your closest existing page, the relevance score, and the model's one-line reason. Sort by relevance first and similarity second, so the most obviously missing, most obviously useful topics rise to the top.

import csv


def write_report(rows: list[dict], path: str = "content_gaps.csv") -> None:
    fields = ["relevance", "reason", "topic", "their_url", "similarity", "your_closest"]
    rows.sort(key=lambda r: (-r["relevance"], r["similarity"]))
    with open(path, "w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)
    print(f"wrote {len(rows)} gaps to {path}")


def main() -> None:
    audience = "solo marketers running a small ecommerce shop"
    threshold = 0.55

    mine = topics_for("https://your-site.example/sitemap.xml")
    theirs = topics_for("https://competitor.example/sitemap.xml")
    my_topics = [t for t, _ in mine]
    url_of = {t: u for t, u in theirs}

    matches = nearest_matches([t for t, _ in theirs], my_topics)
    gaps = [m for m in matches if m[1] < threshold]
    print(f"{len(gaps)} of {len(matches)} topics look missing")

    rows: list[dict] = []
    gap_topics = [g[0] for g in gaps]
    lookup = {g[0]: g for g in gaps}
    for start in range(0, len(gap_topics), 20):
        for item in score_gaps(gap_topics[start:start + 20], audience):
            topic, similarity, closest = lookup[item["topic"]]
            rows.append({
                "relevance": item["score"],
                "reason": item["reason"],
                "topic": topic,
                "their_url": url_of.get(topic, ""),
                "similarity": round(similarity, 3),
                "your_closest": closest,
            })
    write_report(rows)


if __name__ == "__main__":
    main()

Replace the two example domains and the audience string with real values and run python gaps.py. On a competitor with a few hundred pages the whole thing finishes in a couple of minutes, most of which is the polite one-second pause between sitemap requests. Open content_gaps.csv, read the top twenty rows, and delete every row you would not defend to a colleague. If you want this to run every month without you, Schedule Python AI Jobs with GitHub Actions turns it into a cron job that mails you the CSV.

Not every gap is worth filling

The script is deliberately generous about what counts as missing, because a false alarm costs you ten seconds of reading and a missed topic costs you an article. That trade-off only works if you actually reject things. A gap is a fact about two sitemaps; it is not evidence that anyone wants the article. Your competitor may have published it for a partnership, for a client who churned, or because someone on their team had a hunch that did not pay off.

Four questions separate a real opportunity from a distraction. Run each shortlisted row past them before it enters your plan.

There is a second failure mode worth naming. If you run this against five competitors and write everything the union of them covers, you end up with their site instead of yours. The gaps that matter most are usually the ones only one competitor has spotted, or the ones none of them have — and the second kind will never show up in this report, because the report can only compare you against what already exists.

Tuning reference

Four settings change the character of the output. Start with these values and adjust one at a time.

Troubleshooting

• xml.etree.ElementTree.ParseError: syntax error: line 1, column 0 — the URL returned HTML, usually a 404 page or a consent wall, not XML. Print response.text[:200] to confirm, then find the real sitemap path listed in the site's robots.txt.
• httpx.HTTPStatusError: Client error '403 Forbidden' — the site is refusing an unknown client. Keep the descriptive User-Agent header already in the script, slow the loop down, and if it still refuses, respect that and pick another competitor. If the request hangs instead of failing, Fix Connection and Timeout Errors with AI APIs covers timeouts and retries.
• json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) — the scoring call returned prose instead of JSON. Confirm response_format={"type": "json_object"} is present and that the word "JSON" appears in your prompt; Fix JSONDecodeError with AI API Responses in Python has the full checklist.
• openai.RateLimitError: Rate limit reached for requests — you fired the embedding batches faster than your account tier allows. Add a short pause between batches or wrap the call in a retry with exponential backoff, as shown in Fix the 429 Rate-Limit Error in Python.
• Every single topic is reported as a gap — your own list is empty or malformed. Print len(my_topics) and the first five entries; a sitemap index that you parsed as a urlset returns zero URLs and makes everything look missing.

When to use this vs. alternatives

• Use this sitemap comparison when you want a map of what a competitor covers and you do not, at the level of whole articles, without paying for a research tool or crawling anyone's site aggressively.
• Use phrase-level analysis instead when you already know the topic and want the exact wording to target inside it — Python Script for Competitor Keyword Analysis reads the copy on individual pages rather than the shape of a whole site.
• Use a paid research tool as well when the decision hinges on search volume or ranking difficulty. This script tells you what is missing and whether it fits your readers; it cannot tell you how many people search for it each month, because that data is not in anyone's sitemap.

Once the CSV is triaged and a few topics are approved, the rest of this series takes over: Generate Blog Posts with the OpenAI API drafts them and Generate Meta Descriptions in Bulk with Python handles the metadata. Run the gap script quarterly rather than weekly — sitemaps do not change fast enough to reward more attention than that, and the discipline of writing the approved topics matters more than finding new ones.

Back to SEO Keyword Research with Python.

Related guides

• Python Script for Competitor Keyword Analysis — go one level deeper and compare the phrases inside individual pages.
• Group Keywords with Python and Embeddings — turn a long gap list into named themes before you plan.
• Remove Duplicate Records with Embeddings in Python — the same similarity technique applied to deduplicating any list.
• Estimate OpenAI API Costs with Python — price a run before you embed a competitor's entire site.
• Generate Blog Posts with the OpenAI API — draft the articles your triaged CSV approved.