Q18 of 24 · Accessibility
How do you integrate accessibility testing into CI/CD at scale with meaningful failure thresholds?
Short answer
Short answer: Run axe-core assertions in your E2E test suite on critical paths (gated — fail the build on violations). Add a Lighthouse CI step with a minimum accessibility score to catch broader regressions. Run Pa11y or a spider-based scan against staging on a schedule. Manual audits run quarterly — not as CI.
Detail
A layered approach at scale:
Layer 1 — Component-scoped axe checks in E2E tests (PR gate): for each critical user journey (sign-in, checkout, account settings), add an axe scan scoped to the relevant component or page. These run on every PR. A violation fails the build immediately. Scope the scan to your product's components, not the entire page — third-party widgets should be excluded so their failures don't block your team.
Layer 2 — Lighthouse CI (branch or nightly): add the Lighthouse CI action to a branch gate or nightly job. Set a minimum accessibility score (e.g. 90) as the threshold. Lighthouse provides a broader check including some things axe doesn't cover (heading order, link descriptions, form label heuristics). Don't use Lighthouse as your only CI check — it's a score, and a score can obscure individual high-severity failures.
Layer 3 — Pa11y CI or custom spider (staging, weekly): crawl your staging environment's key pages with Pa11y CI or a custom script that runs axe on a URL list. This catches pages that your E2E tests don't visit but real users do (terms pages, help articles, error pages).
Layer 4 — Manual quarterly audit: one full audit per quarter. CI catches regressions; the quarterly audit catches new issues introduced by design decisions, content changes, or features that CI doesn't exercise.
Handling existing violations: if you inherit a codebase with existing violations, baseline them with an explicit allowlist and a ticket reference per entry. The CI check fails only on new violations — enforcing the rule that you don't make things worse, and tracking paydown of existing debt.
// EXAMPLE
a11y.ci.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test('checkout: no new axe AA violations', async ({ page }) => {
await page.goto('/checkout');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
.exclude('#third-party-chat-widget') // known external violation
.analyze();
expect(results.violations).toEqual([]);
});