TestNG Listeners and Reporting

A test passes — TestNG does almost nothing visible. A test fails — TestNG prints a stack trace. That's not enough for a real suite. You want screenshots on failure, a Slack message when the regression suite breaks, a structured HTML report with severity tags, and the ability to retry the most flaky tests automatically. All of these come from one place: TestNG listeners — small classes that hook into every important event in the test lifecycle. This lesson covers ITestListener (the workhorse), screenshot-on-failure (the single most-asked feature), the retry-analyzer, and how to plug in Allure/ExtentReports when the default report runs out of road.

ITestListener — five hooks, one interface

ITestListener is the listener you'll write 90% of the time. It defines five methods that fire at corresponding points:

package com.mycompany.tests.listeners;
 
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
 
public class ConsoleListener implements ITestListener {
 
    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("▶ STARTING: " + result.getName());
    }
 
    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("✅ PASSED:   " + result.getName());
    }
 
    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("❌ FAILED:   " + result.getName());
        System.out.println("   Reason:   " + result.getThrowable().getMessage());
    }
 
    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println("⏭ SKIPPED:  " + result.getName());
    }
 
    @Override
    public void onFinish(ITestContext context) {
        System.out.println("\nSuite finished. Passed: "
            + context.getPassedTests().size()
            + ", Failed: " + context.getFailedTests().size()
            + ", Skipped: " + context.getSkippedTests().size());
    }
}

The ITestResult argument carries everything you'll want — getName(), getThrowable() (the exception), getInstance() (the test class instance), getTestContext() (suite-level context). Read the JavaDoc once; you won't need to again.

The listener flow

Registering a listener

Two ways. Pick one, stay consistent:

// 1. Annotation on the test class
@Listeners({ConsoleListener.class, ScreenshotListener.class})
public class LoginTest extends BaseTest { ... }

<!-- 2. testng.xml — applies to every test in the suite -->
<suite>
    <listeners>
        <listener class-name="com.mycompany.tests.listeners.ConsoleListener"/>
        <listener class-name="com.mycompany.tests.listeners.ScreenshotListener"/>
    </listeners>
    <test>...</test>
</suite>

For cross-cutting concerns (every test takes a screenshot on failure), put them in testng.xml so a new test class doesn't need to remember the annotation. For class-specific listeners (a Slack-only-on-checkout-tests listener), use the annotation.

Screenshot on failure — the must-have listener

This single listener earns more goodwill from QA teams than any other piece of test infrastructure. Wire it up on day one:

package com.mycompany.tests.listeners;
 
import com.mycompany.tests.base.BaseTest;
import org.apache.commons.io.FileUtils;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;
 
import java.io.File;
import java.io.IOException;
import java.nio.file.Paths;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
 
public class ScreenshotListener implements ITestListener {
 
    @Override
    public void onTestFailure(ITestResult result) {
        BaseTest baseTest = (BaseTest) result.getInstance();
        WebDriver driver = baseTest.getDriver();
        if (driver == null) return;     // setup may have failed before driver creation
 
        File source = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        String timestamp = LocalDateTime.now().format(DateTimeFormatter.ofPattern("HHmmss"));
        File destination = Paths.get(
            "target", "screenshots",
            result.getName() + "_" + timestamp + ".png"
        ).toFile();
 
        try {
            FileUtils.copyFile(source, destination);
            System.out.println("📸 Saved: " + destination.getAbsolutePath());
        } catch (IOException e) {
            System.err.println("Could not save screenshot: " + e.getMessage());
        }
    }
}

Three small but real-world touches:

• Cast result.getInstance() to your BaseTest so you can pull the driver out via getDriver(). The casting is brittle — every test class must extend BaseTest. That's why chapter 6 makes that mandatory.
• if (driver == null) return; handles the case where @BeforeMethod blew up before the driver was assigned. Without the guard, the listener itself crashes during teardown.
• Save under target/screenshots/ so screenshots are co-located with Surefire reports and CI can publish them as artefacts in one step.

To use Apache Commons IO's FileUtils.copyFile, add to pom.xml:

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.16.1</version>
    <scope>test</scope>
</dependency>

The default report — and when to outgrow it

Out of the box, every mvn test produces files under target/surefire-reports/:

• index.html — a clickable summary by suite/test/method.
• emailable-report.html — a single-file HTML you can attach to an email.
• testng-results.xml — machine-readable XML that CI tools parse for pass/fail metrics.

For a small project this is enough. As your suite grows, the default report becomes painful — no screenshots inline, no failure trends over time, no severity tags. Two ecosystem libraries are the standard upgrades:

• Allure (open-source, strong in Java ecosystems) — io.qameta.allure:allure-testng. Generates trends, severity, retries, and gorgeous HTML. Use the official guide for setup details.
• ExtentReports (commercial-ish, popular at enterprises) — com.aventstack:extentreports. Rich HTML with screenshots embedded inline.

Both wire up via a ITestListener (or in Allure's case, a built-in TestNG hook) — same onTestFailure you've already written, additionally calling Allure's or Extent's API.

Retry analyzer — the safety net for genuinely flaky tests

When a test fails because of an environmental hiccup (network blip, browser launch glitch), it's tempting to retry. TestNG supports it cleanly via IRetryAnalyzer:

package com.mycompany.tests.listeners;
 
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
 
public class RetryAnalyzer implements IRetryAnalyzer {
    private int attempts = 0;
    private static final int MAX = 2;     // max two retries — total 3 runs
 
    @Override
    public boolean retry(ITestResult result) {
        if (attempts < MAX) {
            attempts++;
            System.out.println("🔁 Retrying " + result.getName() + " (attempt " + (attempts + 1) + ")");
            return true;
        }
        return false;
    }
}

Apply per-test:

@Test(retryAnalyzer = RetryAnalyzer.class)
public void shouldHandleOccasionallyFlakyFlow() { ... }

Or globally via IAnnotationTransformer — apply it to every @Test automatically. Beware the temptation: retry hides flakes. The right policy is "retry for known-environmental categories only, and treat any test that needs retries as debt to fix." Don't retryAnalyzer the entire suite as a way to make CI green.

A complete listener test

package com.mycompany.tests.tests;
 
import com.mycompany.tests.base.BaseTest;
import com.mycompany.tests.listeners.ConsoleListener;
import com.mycompany.tests.listeners.ScreenshotListener;
import org.openqa.selenium.By;
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
 
@Listeners({ConsoleListener.class, ScreenshotListener.class})
public class ListenerDemoTest extends BaseTest {
 
    @Test
    public void shouldPassQuickly() {
        driver.get("https://www.saucedemo.com");
        Assert.assertTrue(driver.findElement(By.id("user-name")).isDisplayed());
    }
 
    @Test
    public void shouldFailAndProduceScreenshot() {
        driver.get("https://www.saucedemo.com");
        Assert.assertTrue(
            driver.findElement(By.id("login-button")).getText().equals("THIS WILL NEVER MATCH")
        );
    }
}

Run it. Console prints the lifecycle; one PNG appears in target/screenshots/ for the failing test. That single screenshot has saved hundreds of debugging hours across QA teams worldwide.

The TestNG cheat sheet covers all the listener interfaces; the Selenium tool entry has the screenshot-related APIs.

⚠️ Common mistakes

• Casting result.getInstance() without a guaranteed parent class. If half your tests extend BaseTest and half don't, the cast in your listener throws ClassCastException on the latter — masking the real test failure with a listener crash. Make BaseTest mandatory; a getDriver() method on a common base is the cleanest contract.
• Catching RuntimeException inside a listener and printing a stack trace. The listener's job is to react to events; if it crashes, TestNG can swallow the failure quietly. Instead: log to a file or stderr, and don't let your listener throw. A robust listener degrades gracefully.
• Treating retry analyzer as a flake fix. Re-running a flaky test until it passes makes CI green and the actual problem invisible. The right use is: retry only for categorised, known-transient failures (network unreachable, browser failed to start), and track every retry — when retries grow, the suite is in trouble.

🎯 Practice task

Wire up real listener infrastructure. 35–45 minutes.

1. Add BaseTest, ConsoleListener, and ScreenshotListener to your project. Make every existing test class extend BaseTest. Confirm the suite still runs green.
2. Run ListenerDemoTest (with the deliberately failing test). Confirm a PNG appears under target/screenshots/ named with the test method and a timestamp. Open it — it should show the login page exactly as the test saw it when it failed.
3. Hook in Allure. Add io.qameta.allure:allure-testng:2.25.0 to pom.xml and the allure-maven plugin. Run mvn clean test then mvn allure:report. Open target/site/allure-maven-plugin/index.html. Compare it to Surefire's default report. The trends, history, and severity tags are why teams switch.
4. Retry analyzer. Apply RetryAnalyzer to a test that would otherwise pass. Force a 70%-flaky test by asserting on a value with Math.random(). Run @Test(retryAnalyzer = RetryAnalyzer.class, invocationCount = 10) and watch some attempts retry once or twice before passing. Then track how often the retries fire — that count is the flake budget.
5. Slack-on-suite-failure (mock). Write a SlackListener that, in onFinish, prints "Would post to Slack: N tests failed". Real Slack integration is two lines beyond that — HttpClient.send(...) to a webhook URL. Don't actually wire to a real channel during practice — the mocked print is enough to prove the listener structure works.
6. Stretch — listener at suite level. Move both listeners from @Listeners(...) annotations on each class into testng.xml's <listeners> block. Remove the annotations. Run the suite. Confirm the listeners still fire — they now apply globally.

Chapter 5 is done. You can write structured TestNG suites with lifecycle hooks, groups, dependencies, data-driven tests, and rich reporting. Chapter 6 is where everything you've learned comes together: the Page Object Model. We'll factor today's repetitive findElement calls into reusable, testable, type-safe page classes — the design pattern every Selenium codebase converges on.