Floorcheck: Catch False-Pass Accessibility Scans

DEV Community

My accessibility scanner said "clean". It was a 403 page.

I build websites for small businesses and institutions, and I run automated accessibility checks on most of them. A while ago I scanned a few hundred homepages in one go and counted how many came back with zero failures. The count was wrong. Some of those "clean" pages were 403 errors. Others were Cloudflare's "Just a moment..." screen, and one was a maintenance notice. A page with nothing on it has nothing to fail, so the scanner happily reported them as perfect. That's not really a bug in axe-core. axe checks whatever page it's given. But it made me look at everything else that a typical scan setup quietly gets wrong, and I ended up building a small CLI around axe-core to fix the parts that kept biting me. It's called floorcheck, because an automated scan is the floor, not the audit. What it does differently It checks that the page is the page. Before trusting a result, floorcheck looks at what actually loaded: the HTTP status, bot-challenge titles and markup, maintenance notices, and pages that are nearly empty. If it's not the real page, the result is "not scanned" and the run exits with code 2. It never counts as a pass. It keeps three piles apart. axe returns violations, "incomplete" results it couldn't decide, and best-practice rules that aren't WCAG requirements. Mixed together, a report is either terrifying or easy to ignore. floorcheck keeps WCAG A/AA failures, things a person has to check, and general advice in separate sections, and only the first one can fail your CI build. It checks reflow at 320px. axe has no rule for WCAG 1.4.10. floorcheck loads the page at 320 CSS pixels and, if it scrolls sideways, names the element that causes it. Code blocks, tables and canvases go to "needs review" instead of failing, because WCAG allows two-dimensional scrolling when the layout really needs it. It tells you when an overlay is running. If the page loads UserWay, accessiBe or a WordPress accessibility toolbar, that's reported as context. The overlay rewrites the page while the scan runs, so you're looking at results for the page with the overlay switched on. Cookies stay with your site. If you scan pages behind a login with --header "Cookie: ...", the cookie is only sent to the site you're scanning, not to the analytics and font hosts the page also loads. That one surprised me: when you override request headers in a Playwright route, a Cookie header is ignored, so cookies have to go through the browser's cookie jar instead. Trying it You need Node 20+ and Chrome or Edge. npx floorcheck https://example.com
npx floorcheck --sitemap https://example.com/sitemap.xml --max-pages 100
npx floorcheck --self-test
The self-test scans a page where every problem is planted on purpose and checks each one is found in the right section. If axe ever silently stops running, that's how you'd notice. In GitHub Actions it's one step. It writes a summary on the run page, and with your source folder it points failures at the file and line, so they show up as code scanning alerts in the pull request: - uses: Pryzmastudio/floorcheck@v1
with:
urls: http://localhost:4173/
project: .
- uses: github/codeql-action/upload-sarif@v4
if: always()
with:
sarif_file: floorcheck.sarif
Checking the checker I also ran floorcheck against the W3C ACT Rules test cases, the shared test suite accessibility tools use to show how their results line up with WCAG. On the approved rules, 30 come out consistent, the same number as in axe-core's own published report, and floorcheck's own focus check comes out partially consistent, which is what you'd expect from a check that can only say "can't tell" rather than "fail". npm run act-report in the repo regenerates that report. What it can't do No automated tool can tell you a site is accessible. axe-core has rules for 23 of the 55 WCAG 2.2 A and AA success criteria. A page with zero failures still needs someone to go through it with a keyboard and a screen reader. I wrote the code with a lot of help fr...

Powered by Readybot.io