Running TestNG Suites in Jenkins and GitHub Actions

A test suite that only runs on a developer's laptop is not a CI test suite — it's a local script. The value of automation is continuous: tests run on every push, every pull request, every night without anyone pressing a button. This lesson wires your TestNG suite into both GitHub Actions and Jenkins, covers the configuration decisions that matter (headless mode, thread counts, which suite to run when), and shows the pattern for passing environment secrets into your tests without committing credentials.

GitHub Actions

Create .github/workflows/testng.yml in your project root:

name: TestNG Suite
 
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'   # nightly regression at 02:00 UTC
 
jobs:
  smoke:
    name: Smoke Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: 'maven'       # cache ~/.m2 between runs
 
      - name: Run smoke suite
        run: mvn clean test -DsuiteXmlFile=smoke.xml -Dbrowser=chrome -Dheadless=true
        env:
          BASE_URL:     ${{ secrets.STAGING_BASE_URL }}
          ADMIN_PASS:   ${{ secrets.ADMIN_PASSWORD }}
 
      - name: Upload test reports
        if: always()           # upload even when tests fail
        uses: actions/upload-artifact@v4
        with:
          name: smoke-reports-${{ github.run_number }}
          path: |
            test-output/
            reports/
            target/surefire-reports/
          retention-days: 14
 
  regression:
    name: Nightly Regression
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: 'maven'
 
      - name: Run regression suite
        run: mvn clean test -DsuiteXmlFile=regression.xml -Dbrowser=chrome -Dheadless=true
        env:
          BASE_URL:   ${{ secrets.STAGING_BASE_URL }}
          ADMIN_PASS: ${{ secrets.ADMIN_PASSWORD }}
 
      - name: Upload regression reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: regression-reports-${{ github.run_number }}
          path: |
            test-output/
            reports/
          retention-days: 30

Key decisions in this workflow:

if: always() on the upload step. Without it, the upload is skipped when Maven exits with a non-zero code (i.e., when tests fail). You need the report most when tests fail — if: always() ensures it is always uploaded regardless of test outcome.

cache: 'maven' caches ~/.m2/repository between runs. The first run downloads everything; subsequent runs restore the cache and skip network I/O. On a large suite with many dependencies this saves 1–3 minutes per run.

Dheadless=true passed as a system property. Your BaseTest reads this:

@BeforeMethod
@Parameters("browser")
public void setup(@Optional("chrome") String browser) {
    boolean headless = Boolean.parseBoolean(
        System.getProperty("headless", "false"));
    ChromeOptions opts = new ChromeOptions();
    if (headless) {
        opts.addArguments("--headless=new", "--no-sandbox",
                          "--disable-dev-shm-usage");
    }
    DriverManager.initDriver(browser, opts);
}

--no-sandbox and --disable-dev-shm-usage are required on GitHub Actions' Ubuntu containers — Chrome crashes without them because the container has limited /dev/shm memory.

Secrets go in GitHub → repository → Settings → Secrets and variables → Actions. Reference them as ${{ secrets.SECRET_NAME }}. They are never logged, never visible in PR forks from external contributors.

Jenkins pipeline

A Jenkinsfile at the project root:

pipeline {
    agent any
 
    tools {
        maven 'Maven-3.9'
        jdk   'JDK-21'
    }
 
    environment {
        BASE_URL   = credentials('staging-base-url')
        ADMIN_PASS = credentials('admin-password')
    }
 
    triggers {
        // Nightly regression at 01:00
        cron('0 1 * * *')
    }
 
    stages {
        stage('Smoke') {
            steps {
                sh 'mvn clean test -DsuiteXmlFile=smoke.xml -Dbrowser=chrome -Dheadless=true'
            }
        }
 
        stage('Regression') {
            when { triggeredBy 'TimerTrigger' }
            steps {
                sh 'mvn test -DsuiteXmlFile=regression.xml -Dbrowser=chrome -Dheadless=true'
            }
        }
    }
 
    post {
        always {
            // TestNG plugin — produces trend graphs in Jenkins UI
            testNG reportFilenamePattern: '**/testng-results.xml'
 
            // Archive Extent or Allure report as a linked artefact
            publishHTML(target: [
                reportDir:   'reports',
                reportFiles: 'extent-report.html',
                reportName:  'Extent Report',
                keepAll:     true,
                allowMissing: true
            ])
 
            // Archive raw test-output for download
            archiveArtifacts artifacts: 'test-output/**,reports/**',
                             allowEmptyArchive: true
        }
        failure {
            mail to: 'qa-team@mycompany.com',
                 subject: "Build ${BUILD_NUMBER} failed — ${JOB_NAME}",
                 body: "See: ${BUILD_URL}"
        }
    }
}

Install the TestNG Plugin in Jenkins to get the testNG post-build step. It parses testng-results.xml and renders a per-run table and trend graph in the Jenkins job UI. The HTML Publisher Plugin provides publishHTML for linking ExtentReports.

Credentials in Jenkins come from Manage Jenkins → Credentials, referenced with credentials('credential-id'). They are masked in build logs automatically.

Thread counts in CI

Local development can use thread-count="4". CI runners have different constraints:

• GitHub Actions (ubuntu-latest): 2 vCPUs — use thread-count="2" for Selenium, thread-count="3" for pure API tests
• Jenkins (shared agent, 4 cores): thread-count="2" to leave headroom for the Maven process and Chrome itself
• Jenkins (dedicated agent, 8 cores): thread-count="4" is safe; thread-count="6" if tests are short-lived

Set thread-count as a property so you can override from CI without editing XML:

<suite name="Regression" parallel="classes"
       thread-count="${threadCount:-2}">

mvn test -DthreadCount=4

The CI pipeline flow

Scheduling nightly regression

GitHub Actions cron:

on:
  schedule:
    - cron: '0 2 * * *'     # 02:00 UTC every night

Jenkins cron (Triggersblock in Jenkinsfile or UI):

H 1 * * *    # 01:00, H = hash-based minute to spread load

Nightly regression runs the full suite, including slow groups excluded from the PR smoke run. A typical pattern:

• On every PR: smoke.xml — 5–15 tests, under 5 minutes, gates merge
• On merge to main: regression.xml — full coverage, 10–30 minutes
• Nightly: regression.xml + extended suites — slow tests, cross-browser, performance gates

⚠️ Common mistakes

• Missing --no-sandbox on GitHub Actions. Chrome on a GitHub-hosted Ubuntu runner crashes silently without this flag. The symptom is all Selenium tests failing with SessionNotCreatedException or a zero-byte screenshot. Add --no-sandbox --disable-dev-shm-usage to ChromeOptions when headless=true is set.
• Uploading artefacts only on success. CI reports are most valuable when tests fail. if: always() in GitHub Actions (or post { always {} } in Jenkins) ensures the report is available for every run. Without it you have no evidence to investigate when the nightly run fails.
• Committing secrets in testng.xml. A <parameter name="adminPassword" value="real-password"/> in testng.xml is committed to the repository. Use System.getProperty("adminPass") instead and inject the value via -DadminPass=${{ secrets.ADMIN_PASS }} — the value never touches source control.

🎯 Practice task

Get your suite running in CI. 30–45 minutes.

1. Create .github/workflows/testng.yml from the template. Push to a GitHub repository. Watch the Actions tab — confirm the workflow triggers, Java is installed, and mvn test runs (even if tests fail due to missing credentials).
2. Add if: always() to the upload step and confirm the report artefact appears in the run summary after both a passing and a failing run.
3. Wire secrets. Go to repository Settings → Secrets → Actions. Add STAGING_BASE_URL with your test site's URL. Update the workflow to pass -DbaseUrl=${{ secrets.STAGING_BASE_URL }}. Confirm the workflow reads it (echo a masked version in the logs).
4. Add headless mode. Update BaseTest.setup() to read System.getProperty("headless") and add the Chrome flags when true. Pass -Dheadless=true in the workflow. Confirm tests run (and pass) in the headless container.
5. Set up the nightly cron. Add schedule: - cron: '0 2 * * *' and a separate regression job that only triggers on schedule. Manually trigger it via "Run workflow" in the Actions UI to test it without waiting until 02:00.
6. Stretch — Jenkins. If you have a local Jenkins instance, add the Jenkinsfile from this lesson. Install the TestNG and HTML Publisher plugins. Run the pipeline. Confirm the TestNG trend graph appears after two runs.

Next chapter: the capstone project. Everything from this course lands in one framework.