Q13 of 24 · Accessibility
Why does automated accessibility testing only catch around 30-40% of real issues?
Short answer
Short answer: Automation can verify structure — presence of alt text, label associations, contrast ratios, ARIA syntax — but it cannot evaluate meaning. Whether alt text is meaningful, whether the reading order makes sense, and whether a custom widget is usable with a screen reader all require human judgment.
Detail
The 30-40% figure comes from research by WebAIM and Deque. Automated tools like axe-core evaluate rules that have deterministic answers: is this contrast ratio above 4.5:1? Is this input labelled? Does this element have a valid role? These checks pass or fail without needing to understand the content.
What automation cannot evaluate:
- Alt text quality: an image of a customer signing a contract with alt text "image123" passes the axe check (alt attribute is present and non-empty) but is meaningless to a screen reader user.
- Keyboard interaction flow: axe can check that elements are focusable, but it can't verify whether the Tab order makes logical sense, whether arrow key navigation within a custom widget behaves correctly, or whether focus is returned to the right place after closing a modal.
- Screen reader announcement quality: axe can verify that a button has an accessible name, but it can't determine whether that name is clear, distinct from other buttons, and contextually meaningful.
- Reading order: the visual layout and the DOM order may diverge — visually logical but confusing when read linearly. Automated tools rarely catch this.
- Cognitive accessibility: plain language, clear instructions, consistent navigation patterns. These require human judgment about meaning, not structure.
The right way to present this in an interview: automated testing is a regression net — it catches regressions in structural accessibility. It is not a substitute for a manual audit, keyboard testing, and screen reader testing.