Generating Accessibility and Visual Reports

A failing test that nobody can read is a test that goes ignored. Visual regressions need pixel diffs the reviewer can scan; accessibility violations need a structured report with rule ids and affected elements. This last lesson of the chapter wires the two together — saving structured a11y JSON to disk, generating browsable HTML dashboards with pytest-html, attaching screenshots and reports to Allure, and combining visual and a11y checks in one test so a single PR check covers both. By the end you'll have the reporting infrastructure that lets a non-engineer (PM, designer, accessibility specialist) review the suite's output and act on it.

Saving a11y reports to disk

The structured output of axe.run(page) lives on results.response. Dump it to JSON for review:

import json
from pathlib import Path
from playwright.sync_api import Page
 
 
def test_a11y_audit_homepage(page: Page, axe_scanner):
    page.goto("/")
    results = axe_scanner.run(page)
 
    report_dir = Path("reports/a11y")
    report_dir.mkdir(parents=True, exist_ok=True)
    with open(report_dir / "homepage.json", "w") as f:
        json.dump(results.response, f, indent=2)
 
    high_impact = [v for v in results.violations if v["impact"] in ["critical", "serious"]]
    assert len(high_impact) == 0

Each scanned page produces a JSON file with the full axe output — every violation, every passing rule, every node, every help URL. The team can grep, diff, and trend these files over time. CI uploads reports/a11y/ as an artefact; reviewers download and inspect on demand.

Multi-page accessibility dashboard via parametrize

Combine the reporting pattern with parametrize to scan an entire site in one test function:

@pytest.mark.a11y
@pytest.mark.parametrize("path", ["/", "/products", "/login", "/checkout", "/about"])
def test_a11y_per_page(page: Page, axe_scanner, path: str):
    page.goto(path)
    results = axe_scanner.run(page)
 
    # Persist the report regardless of pass/fail
    safe_name = path.replace("/", "_") or "_home"
    Path("reports/a11y").mkdir(parents=True, exist_ok=True)
    with open(f"reports/a11y/{safe_name}.json", "w") as f:
        json.dump(results.response, f, indent=2)
 
    critical = [v for v in results.violations if v["impact"] in ["critical", "serious"]]
    assert len(critical) == 0, (
        f"Critical/serious a11y issues on {path}:\n"
        + "\n".join(f"  [{v['impact']}] {v['id']}: {v['description']}" for v in critical)
    )

One test function runs five times, produces five JSON reports, and fails per-page with a per-page assertion message. Adding a sixth route is one line in the parametrize list.

pytest-html — the simple HTML dashboard

pytest-html turns any pytest run into a browsable HTML report. Install and use:

pip install pytest-html
pytest --html=reports/index.html --self-contained-html

--self-contained-html inlines all CSS and images so the report is a single file you can email or upload. The report shows every test, its duration, its outcome, captured stdout/stderr, and any attached files.

Attach screenshots from a test:

import pytest
 
@pytest.fixture(autouse=True)
def attach_screenshot_on_failure(request, page):
    yield
    if request.node.rep_call.failed:
        screenshot_bytes = page.screenshot()
        # Attach via pytest-html's extra mechanism
        if hasattr(request.config, "_html"):
            from pytest_html import extras
            request.node.user_properties.append(("screenshot", screenshot_bytes))

When a test fails, the pytest-html report shows the screenshot alongside the failure message. Reviewers see the broken UI without re-running locally.

Allure — the production-grade reporting tool

Allure is heavier than pytest-html but produces a much richer dashboard — feature/severity hierarchies, trend graphs, attachment galleries, history tracking. Install:

pip install allure-pytest

Run with the Allure adapter:

pytest --alluredir=reports/allure
allure serve reports/allure

The first command writes JSON results; the second launches a local web server with the rendered dashboard. CI typically uploads reports/allure/ as a build artefact and a downstream job publishes the HTML.

Allure attachments — screenshots and structured data

Allure's killer feature is attachments — anything you call allure.attach(...) on appears under the relevant test in the dashboard:

import allure
import json
from playwright.sync_api import Page
 
 
@allure.feature("Visual Testing")
def test_homepage_visual(page: Page):
    page.goto("/")
 
    screenshot = page.screenshot()
    allure.attach(
        screenshot,
        name="Homepage",
        attachment_type=allure.attachment_type.PNG,
    )
 
    expect(page).to_have_screenshot()

The @allure.feature decorator groups the test under a "Visual Testing" feature in the dashboard. The allure.attach(...) call embeds the PNG into the test's report card — clickable, zoomable, downloadable.

For accessibility, attach the JSON results directly:

@allure.feature("Accessibility")
def test_a11y_homepage(page: Page, axe_scanner):
    page.goto("/")
    results = axe_scanner.run(page)
 
    allure.attach(
        json.dumps(results.response, indent=2),
        name="Accessibility Report",
        attachment_type=allure.attachment_type.JSON,
    )
 
    high_impact = [v for v in results.violations if v["impact"] in ["critical", "serious"]]
    assert len(high_impact) == 0

Now every a11y test in the report has a clickable JSON viewer with the full axe response, plus a structured breakdown of violations.

Combining visual and a11y in one test

The two scans are cheap and complementary — pair them in tests that care about both:

import allure
import json
from playwright.sync_api import Page, expect
 
 
@allure.feature("Page Quality")
def test_products_page_quality(page: Page, axe_scanner):
    page.goto("/products")
 
    # Visual check — captures or compares a snapshot
    expect(page).to_have_screenshot(
        "products.png",
        animations="disabled",
        max_diff_pixel_ratio=0.01,
    )
 
    # Accessibility check — asserts no high-impact violations
    results = axe_scanner.run(page)
    allure.attach(
        json.dumps(results.response, indent=2),
        name="A11y results",
        attachment_type=allure.attachment_type.JSON,
    )
    high_impact = [v for v in results.violations if v["impact"] in ["critical", "serious"]]
    assert len(high_impact) == 0

One test, two assertions, both surfaces covered. If either fails, you know which one and why — pixel diff for visuals, structured JSON for a11y. The Allure dashboard groups them under "Page Quality" so PMs and designers see them together.

The reporting pipeline

Five steps, three artefact types (visual diffs, a11y JSON, run report), and the result is a self-service dashboard the whole team can use without re-running tests locally.

Per-test report metadata with Allure decorators

Allure exposes a rich set of decorators for organising the dashboard:

import allure
 
 
@allure.feature("Authentication")
@allure.story("Login")
@allure.severity(allure.severity_level.CRITICAL)
def test_admin_login(page, axe_scanner):
    allure.dynamic.title("Admin can log in and reach dashboard")
    page.goto("/login")
    # ...

• @allure.feature — the top-level grouping (use for large product areas).
• @allure.story — sub-grouping under a feature (use for scenarios within an area).
• @allure.severity — BLOCKER / CRITICAL / NORMAL / MINOR / TRIVIAL. Drives report sorting and filtering.
• allure.dynamic.title(...) — runtime override of the test name. Useful for parametrized tests where you want a custom display name per case.

The dashboard surfaces these as filters and breakdowns — "show all critical Auth tests that failed in the last 5 runs" is a one-click query.

Coming from Playwright TypeScript?

The TS course covers this material with HTML reporter and Allure as well — the difference is install commands and import statements:

• TS npx playwright test --reporter=html → Python pytest --html=reports/index.html
• TS npx playwright test --reporter=allure-playwright → Python pytest --alluredir=reports/allure
• TS await testInfo.attach('name', { body, contentType }) → Python allure.attach(body, name="name", attachment_type=...)

The conceptual model is identical — capture artefacts during the test, render to a dashboard after the run, publish the dashboard from CI.

Cross-reference: the Manual Software Testing course

The Manual Software Testing course covers the human side of accessibility — screen-reader testing, keyboard-only navigation, the WCAG criteria automated tools can't check. Use that course's content to build the audit checklist; use this lesson's automation to gate the easy half. Together they cover ~80% of the WCAG surface; the last 20% always requires manual review.

⚠️ Common mistakes

• Treating Allure as a build artefact upload only. Allure shines when its history feature is configured — --allure-features-history-style flags let one run see the trend of the previous N runs. Without history, every run looks the same; with it, you see flake patterns and regressions across time. Set up history persistence in CI on day one.
• Attaching huge artefacts to every test. A 5MB full-page screenshot on every test inflates report sizes into hundreds of MB. Attach screenshots only on failure (via an autouse fixture that checks the test outcome), or use Allure's step mechanism to attach intermediate evidence only when relevant.
• Skipping the Path.mkdir(parents=True, exist_ok=True) step before writing reports. The first time you run on a fresh CI runner, the reports directory doesn't exist and open(..., "w") raises FileNotFoundError. Always create the parent directory before writing — parents=True handles nested paths, exist_ok=True makes it idempotent.

🎯 Practice task

Wire up reporting end-to-end. 30-40 minutes.

1. Install reporting tools:

   Copy

   pip install pytest-html allure-pytest

2. Add a "page quality" test that combines visual + a11y + Allure attachments. In tests/test_quality.py:

   Copy

   import json
   import allure
   import pytest
   from playwright.sync_api import Page, expect
   from axe_playwright_python.sync_playwright import Axe
    
   @pytest.fixture
   def axe_scanner():
       return Axe()
    
   @allure.feature("Page Quality")
   @pytest.mark.parametrize("path,name", [
       ("/", "login"),
       ("/inventory.html", "inventory"),
   ])
   def test_page_quality(page: Page, axe_scanner, path, name, standard_user_state, browser):
       # Use auth state for paths that need it; reuse from chapter 5
       if path != "/":
           context = browser.new_context(
               storage_state=standard_user_state,
               base_url="https://www.saucedemo.com",
           )
           page = context.new_page()
    
       page.goto(f"https://www.saucedemo.com{path}")
    
       # Visual
       expect(page).to_have_screenshot(f"{name}.png", animations="disabled")
    
       # A11y
       results = axe_scanner.run(page)
       allure.attach(
           json.dumps(results.response, indent=2),
           name=f"A11y for {name}",
           attachment_type=allure.attachment_type.JSON,
       )
       high_impact = [v for v in results.violations if v["impact"] in ["critical", "serious"]]
       assert len(high_impact) == 0

3. Generate visual baselines: pytest tests/test_quality.py --update-snapshots.

4. Run with both reporters:

   Copy

   pytest tests/test_quality.py \
     --html=reports/visual.html --self-contained-html \
     --alluredir=reports/allure

   Open reports/visual.html to see the pytest-html dashboard.

5. Generate the Allure report: allure serve reports/allure. A browser opens with the Allure dashboard. Click into a test — the JSON a11y attachment appears in the test detail.

6. Force a failure. Modify a baseline (capture a fresh one, then alter its visual content via custom CSS or by loading a different --browser). Re-run. Both the HTML and Allure reports show the failure with the diff and a11y attachments still attached.

7. Stretch: wire it into a .github/workflows/test.yml (or your CI config). The job runs pytest, uploads reports/ as an artefact, and prints the artefact URL in the job summary. Reviewers download reports/visual.html from the action run page; for Allure, set up actions/upload-artifact for reports/allure and a follow-up peaceiris/actions-gh-pages step to publish to GitHub Pages.

You've completed the visual testing and accessibility chapter. The next chapter takes the suite into CI/CD territory — running on GitHub Actions, parallelising with pytest-xdist, Docker images, and the Allure/HTML reporting integration that publishes results to the team automatically.