Find Duplicate Sitemap Pages with Python
Zusammenfassung
Pixel shares a practical Python guide for finding identical and near-duplicate pages across a sitemap using only the standard library. It covers HTML and main-content hashing, difflib similarity scoring, duplicate clusters, canonical checks, CSV reporting, and CI-friendly exit codes—helping the community spot SEO issues efficiently.
DEV Community
Find Duplicate and Near-Duplicate Pages Across Your Sitemap with Python (Standard Library Only): Content Hashing, difflib Similarity, CSV Report
Most duplicate pages are not copied on purpose. A CMS keeps serving the old URL after a page is renamed. A newsletter link with a tracking parameter ends up in the sitemap. A print view, a demo copy of the home page, or a set of city pages built from one template quietly becomes a second (or fifth) version of the same content. Each page loads fine on its own. You only see the problem when you compare pages with each other. This post builds a small duplicate finder in Python using only the standard library (urllib, html.parser, xml.etree, hashlib, difflib, csv, concurrent.futures). It reads a sitemap or sitemap index, fetches every page, strips the shared template, and compares pages three ways: identical HTML, identical main content, and near-identical main content scored with difflib. Matching pages are grouped into clusters, each cluster gets a canonical check and a suggested keeper URL, and the result is a CSV, a console summary and an exit code that works in CI. Google Search Central's What is canonicalization (showing "Last updated 2026-08-20 UTC"), How to specify a canonical URL with rel="canonical" and other methods (showing "Last updated 2026-07-10 UTC") and Fix canonicalization issues (showing "Last updated 2026-08-21 UTC") were opened on 14 Nov 2026, and every Google statement below comes from those three pages. Appie free-tool descriptions are as of 14 Nov 2026, checked on the tool pages that day. The script was tested on Python 3.13.5 against a local fixture site with planted duplicates; the output below is real, from a run on 14 Nov 2026. Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them. Where this fits This is another script in a small standard-library series. An earlier post audited canonical tags and noindex directives across a sitemap, which checks what each page says its canonical is. Another found missing and duplicate H1 tags. This one asks the question underneath both: which pages actually are the same, judged by their content, and do their canonical signals agree about which one should win? What Google actually says (and what it does not) The canonicalization overview defines the result of the process: "a canonical URL is the URL of a page that Google chose as the most representative from a set of duplicate pages." It then lists common reasons a site ends up with duplicates, which double as a checklist for what to plant in a test fixture: "Region variants: for example, separate USA and UK URLs with hreflang, but essentially the same content in the same language" "Device variants: for example, a page with both a mobile and a desktop version" "Protocol variants: for example, the HTTP and HTTPS versions of a site" "Site functions: for example, the results of sorting and filtering functions of a category page" "Accidental variants: for example, the demo version of the site is accidentally left accessible to crawlers" The same page is clear that duplicates are not a penalty in themselves: "Some duplicate content on a site is normal and it's not a violation of Google's spam policies." The reasons it gives for caring are practical. Having the same content at many URLs "can be a bad user experience" and "may make it harder for you to track how your content performs in search results." It also describes how Google handles them, and this is the part the script imitates. Google determines "the primary content (or centerpiece) of each page," and "If Google finds multiple pages that seem to be the same or the primary content very similar, it clusters them together." It then picks one page as canonical, and "duplicates are crawled less frequently." Your own signals help but do not decide: "indicating a canonical preference is a hint, not a rule." The page on specifying a canonical URL ranks the signals "in order of how strongly they can influence canonicalization": Redirects: "A strong signal that the target of the redirect should becom...

Powered by Readybot.io