Retry and Self-Healing Strategies

A test that fails 1 time out of 10 runs, with no code change between runs, is flaky. Flaky tests are the most corrosive problem in a mature test suite — not because they fail, but because of what the team does in response. First they re-run. Then they add a retry. Then the suite "goes green" and nobody investigates. Six months later, 30% of tests are retrying silently, the suite reports green while hiding dozens of real instabilities, and engineers have stopped trusting test results. This lesson covers retry strategies — how they work, when they're legitimate, and how to use them without letting them become a mask over problems that need fixing.

Why tests are flaky

Before choosing a strategy, understand the cause. Flaky failures cluster into a handful of categories:

Timing issues. An element isn't clickable yet. An AJAX call hasn't completed. An animation is still running. The test acts before the app is ready. Root cause: missing or insufficient waits.

Shared state between tests. Test A leaves the database, browser, or file system in a state that causes Test B to fail when run in that order. Root cause: missing test isolation.

Network instability in test environments. A third-party API the app depends on has occasional timeouts. A staging server is slow. Root cause: environment, not code.

Parallel data collisions. Two tests create a user with the same email simultaneously. Root cause: non-unique test data.

Genuine application bugs. A race condition in the application code that only triggers occasionally. Root cause: a real defect that testing is revealing intermittently.

The first four are framework problems — fixable with better waits, isolation, data, and environment stability. The fifth is a real bug — retrying it makes it invisible. Retry strategies must be applied with this taxonomy in mind.

Smart waits — the first line of defence

Before reaching for any retry mechanism, ask: can an explicit wait fix this? The vast majority of timing-based flakiness is solved by waiting for the right condition before acting.

// Wrong: implicit wait or Thread.sleep
Thread.sleep(2000);  // always wrong
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));  // blunt instrument
 
// Right: explicit wait for the specific condition
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
wait.until(ExpectedConditions.elementToBeClickable(By.id("submit")));
driver.findElement(By.id("submit")).click();

// Playwright — built-in auto-waiting, extend with custom conditions
await page.waitForSelector("[data-testid='submit']:not([disabled])");
await page.click("[data-testid='submit']");
 
// Or wait for a network call to complete
await page.waitForResponse(resp => resp.url().includes("/api/orders") && resp.status() === 200);

# Python/pytest — EC or polling with custom condition
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
 
wait = WebDriverWait(driver, timeout=10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[data-testid='submit']")))
driver.find_element(By.CSS_SELECTOR, "[data-testid='submit']").click()

If a smart wait fixes the flakiness, you're done. No retry needed.

Action-level retry — for transient DOM exceptions

Some failures are transient at the action level, not at the test level. StaleElementReferenceException happens when an element reference is invalidated between locating the element and acting on it — common in SPAs that re-render components after JavaScript updates. Re-locating the element and retrying the action fixes this cleanly:

public void clickWithRetry(By locator, int maxAttempts) {
    for (int i = 0; i < maxAttempts; i++) {
        try {
            driver.findElement(locator).click();
            return;
        } catch (StaleElementReferenceException e) {
            if (i == maxAttempts - 1) throw e;
            log.warn("StaleElement on click attempt {}/{} — retrying: {}", i + 1, maxAttempts, locator);
        }
    }
}

This belongs in BasePage or a utility helper. Tests call clickWithRetry(submitButton, 3) and never see the retry logic.

Test-level retry — the measured last resort

Test-level retry re-runs the entire test when it fails. This is appropriate for failures caused by external transience (network timeouts, third-party service blips) that no amount of smart waiting can prevent.

TestNG — IRetryAnalyzer:

public class RetryAnalyzer implements IRetryAnalyzer {
    private int retryCount = 0;
    private static final int MAX_RETRY = 2;
 
    @Override
    public boolean retry(ITestResult result) {
        if (retryCount < MAX_RETRY) {
            log.warn("Retrying test '{}' — attempt {}/{}", result.getName(), retryCount + 1, MAX_RETRY);
            retryCount++;
            return true;
        }
        return false;
    }
}

Apply via annotation or listener — listener-based is cleaner because it applies globally without touching each @Test:

public class RetryListener implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation, Class testClass,
                          Constructor testConstructor, Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

Playwright — built-in retry:

// playwright.config.ts
retries: process.env.CI ? 2 : 0,  // retry only in CI, not locally

pytest — pytest-rerunfailures:

pytest --reruns 2 --reruns-delay 1

The chart reflects a real-world pattern: the vast majority of flakiness is fixable without retries. Timing issues — the dominant category — are resolved with explicit waits. Data collisions are resolved with isolated test data. Only external service instability genuinely warrants retry, because no framework change can make a third-party API reliable.

Tracking flakiness — making the problem visible

A retry without tracking is a hidden problem. A retry with tracking is a metric you can act on.

// Log every retry with enough context to find the root cause
@Override
public boolean retry(ITestResult result) {
    if (retryCount < MAX_RETRY) {
        log.warn("[FLAKY] Test: {} | Failure: {} | Retry: {}/{}",
            result.getName(),
            result.getThrowable().getMessage(),
            retryCount + 1,
            MAX_RETRY);
        retryCount++;
        return true;
    }
    log.error("[FLAKY-FAIL] Test: {} has failed after {} retries — needs investigation",
        result.getName(), MAX_RETRY);
    return false;
}

Aggregate these logs over time. A test that retries on 20% of runs is a P1 engineering task. A test that retries 1% of runs is environment noise. Without the data, you can't tell them apart.

Self-healing locators — a specialist tool

Self-healing frameworks (Healenium for Java/Selenium, Testim's built-in healing) attempt to find elements when their locators break — using ML models trained on DOM snapshots to suggest or automatically apply replacement locators.

Use self-healing with extreme caution:

• It can mask real breakages. If the login button was renamed because the login flow fundamentally changed, self-healing that silently finds "the most similar button" may make the test pass incorrectly — the test validates the wrong element.
• It adds significant infrastructure complexity. Healenium requires a separate service, a database for snapshot storage, and a CI-compatible deployment.
• Explicit wait fixes are almost always better. If a locator breaks because of a DOM timing issue, a more robust selector + explicit wait is cleaner than a healing algorithm.

Where self-healing is genuinely useful: suites with hundreds of legacy tests that break frequently due to minor UI renames, where the cost of manually updating 50 page objects per sprint exceeds the risk of occasional false passes.

⚠️ Common mistakes

• Setting MAX_RETRY = 3 globally without a flakiness investigation policy. Every test now retries silently. The suite reports green. In three months, nobody knows which tests are genuinely reliable and which pass only on the second or third attempt. Cap retries at 1–2, log every retry, and review the retry log weekly.
• Using Thread.sleep instead of an explicit wait and calling the result "fixed." Adding Thread.sleep(3000) before a click that was previously timing out makes the test pass but adds 3 seconds to every run — even when the element loads in 200ms. Explicit waits wait the minimum necessary time and fail fast when the condition genuinely doesn't occur.
• Not differentiating retry counts by root cause. External service calls should retry more (they can timeout for legitimate reasons); locator-based failures should retry zero times (they indicate a code problem). Use different retry policies for different test categories by tagging them.

🎯 Practice task

Implement and audit retry in your framework — 35 minutes.

1. Explicit wait audit. Find every Thread.sleep in your test codebase. For each one, identify what it's waiting for. Replace it with an ExpectedConditions wait (Selenium), waitForSelector (Playwright), or wait_for_condition (Python). Run the suite — timing-based failures should disappear.
2. Implement RetryAnalyzer. Create the IRetryAnalyzer implementation with MAX_RETRY = 1 and logging. Apply via IAnnotationTransformer listener. Register in testng.xml.
3. Force a flaky condition. Artificially introduce a flaky test: add if (Math.random() > 0.5) throw new AssertionError("Flaky!"). Run the suite 5 times. Verify the retry logic fires and the test eventually passes (or fails after max retries).
4. Build a flakiness report. Add a counter to RetryAnalyzer that increments a static AtomicInteger per test name. In @AfterSuite, log all test names with retry counts above zero. Run the suite 10 times in CI. Review the log: which tests retried and how often?
5. Stretch — triage one flaky test. Take the test in your suite most commonly retried. Reproduce the failure deterministically (increase speed, reduce waits, run in parallel). Identify the root cause from the taxonomy: timing, shared state, network, data collision, or real bug. Fix it. Confirm the test passes 20 consecutive times without retry.

Next lesson: test independence and isolation — the principle that every test must be runnable in any order, in any environment, without depending on another test's side effects.