Automated Accessibility Testing: What It Can and Cannot Detect

Automated Accessibility Testing Guide: What It Can and Cannot Detect

Automated accessibility testing tools catch many surface problems fast. They can find missing attributes, contrast failures and simple markup errors. They cannot replace manual checks, user testing or judgement about meaning, context and usability.

What automated tools typically detect

Most scanners and linters are good at structural, machine-checkable issues. Expect these checks to be reliably reported.

• Missing alternative text for images and some non-text content (WCAG 1.1.1).
• Document structure problems such as missing or out-of-order heading elements and duplicate IDs.
• Basic ARIA problems like invalid attributes, misused roles and unlabeled ARIA controls (gives hints about WCAG 4.1.2).
• Colour contrast failures that do not meet numeric ratios, for example 4.5:1 for normal text and 3:1 for large text (WCAG 1.4.3).
• Missing form labels and inputs without an associated label or accessible name.
• Empty links or buttons and elements with no accessible name.
• Unreachable or hidden content such as elements with display:none that still receive focus.
• Common keyboard focus issues that are detectable by DOM inspection, for example tabindex=“-1” used incorrectly.

What automated tools cannot reliably detect

Tools cannot read intent or judge content quality. Many WCAG requirements need human interpretation or manual testing with assistive technology.

• Quality of alternative text. A tool can say alt text is present or missing. It cannot tell if alt text conveys the right meaning for the image.
• Logical reading order and visual meaning. Complex layouts, CSS positioning and content that relies on visual order require manual review or screen reader testing.
• Keyboard interaction quality. Whether controls are operable in a usable order, whether shortcuts conflict, or whether focus moves logically needs manual checks.
• Colour as sole means of conveying information. A tool can flag contrast but not whether colour is the only signal for meaning.
• Cognitive accessibility and content clarity. Comprehension, instruction clarity and plain language must be evaluated by people and target users.
• Multimedia accessibility quality. Presence of captions or transcripts can be detected if marked up. Quality of captions, accuracy of transcripts and adequacy of audio descriptions need review.
• CAPTCHAs and interactive widgets. Automated checks usually cannot exercise or validate external widgets used for authentication.
• False positives and contextual ARIA use. Tools often flag ARIA as risky but cannot know if the ARIA is required to support a complex custom control.

Examples of false positives and false negatives

• False positive: a decorative image with alt attribute “” flagged as missing alt when the scanner expects text content.
• False negative: a visually ordered table rendered via CSS grid that a scanner reports as correct but a screen reader presents incorrectly.

Practical step-by-step testing workflow

This workflow shows how to use automated checks and where to add manual work. Use it on every feature branch and release candidate.

1. Run an automated scan locally. Use a tool that checks HTML, ARIA and contrast and that can run in the browser and headless CI.
2. Filter results by confidence and severity. Group clear coding errors separately from policy or design issues.
3. Map each issue to a WCAG success criterion and to a code location. Note if it is a potential false positive.
4. Fix code-level issues first. Re-run the scanner until high-confidence failures are resolved.
5. Perform manual checks: keyboard-only navigation; screen reader tests with NVDA or VoiceOver; mobile viewport and touch checks; caption and transcript review for media.
6. If possible, run a short assisted test with one or two users of assistive technology for the most critical flows like checkout and signup.
7. Add regression tests and run scans in continuous integration. Fail builds only on high-severity regressions to avoid noisy blocking.

Quick decision framework: when automated tests are enough

Automated tests are good for catching regressions and fast surface checks. They are not enough to approve a release for accessibility.

• Use automated tests only for structural checks, regressions and developer feedback.
• Require manual testing for any user-facing flow that relies on semantics, interaction or multimedia.
• Treat automated passes as a gate to reduce manual effort, not as final acceptance.

Table: common WCAG checks and detectability

Check	Detectability	Notes
Presence of alt attribute (1.1.1)	Automatable	Tool can detect presence but not quality.
Contrast ratio (1.4.3)	Automatable	Measured numerically for static colours. Dynamic or canvas text may need manual checks.
Label associations for form controls (1.3.1)	Mostly automatable	Detects missing label element or aria-label. Complex custom widgets may need manual checks.
Keyboard operability (2.1.x)	Partial	DOM-level issues spotted; usability and tab order require human testing.
Logical reading order (1.3.2)	Partial	Tools can flag anomalies but cannot fully determine user experience with assistive tech.
Caption accuracy and media descriptions (1.2.x)	Partial to manual	Detects presence of elements but not accuracy or quality.
Use of ARIA roles and properties (4.1.2)	Automatable	Tools can detect invalid or missing roles but cannot judge correct semantic use in complex widgets.

Integrating automated tests into CI and your backlog

Automated scans belong in pull request checks and nightly builds. Keep failure rules sensible so developers trust the tool.

• Run lightweight checks on pull requests and full scans on merge or scheduled runs.
• Annotate failing tests with code links and recommended fixes to speed triage.
• Use baselining to ignore pre-existing low-value failures while prioritising new issues.
• Prioritise fixes by user impact rather than raw counts. Track trends over time.

Common mistakes and how to avoid them

• Stopping after automated fixes. Add manual review for critical flows.
• Blindly trusting tool severity labels. Triage each item against user impact.
• Using ARIA to patch broken HTML. Prefer semantic elements first.
• Failing to test on real devices and assistive tech. Emulators miss important behaviours.

Simple triage priority rules

• P0: Accessibility break that blocks a core task for assistive tech users, for example a keyboard trap or missing label on a required field. Fix before release.
• P1: Significant issue that affects many users, for example missing alt on informative images. Fix in the next sprint.
• P2: Cosmetic or low-impact failures, for example minor contrast issues in low-use UI. Schedule into backlog grooming.

Post-scan checklist

• Confirm each high-severity automated failure is fixed and re-scanned.
• Do keyboard-only walkthroughs of key flows.
• Run a screen reader on critical pages and check reading order and labels.
• Verify media has captions or a transcript and that captions match the audio.
• Plan at least one quick user test with an assistive tech user for each release with substantive UI changes.

Automated accessibility testing is a powerful tool when used correctly. Treat it as the first filter and the regression guard. Add human checks and user testing for the gaps that only people can find.