Q11 of 38 · CI/CD & DevOps

What is a quality gate and what should it block on?

CI/CD & DevOpsMidci-cdquality-gatecoveragepolicy

Short answer

Short answer: A quality gate is a CI step that fails the build (or blocks merge/deploy) on objective criteria. Block on: failing tests, test pass rate below threshold, coverage drop on new code, critical-severity security findings, breaking schema changes. Don't block on subjective things.

Detail

A gate has three properties:

  1. Objective — pass/fail driven by data, not opinion.
  2. Owned — someone is on the hook when it fails.
  3. Actionable — the failure tells the dev exactly what to do.

Things to block on:

  • Failing tests — the obvious one. No retries to make green.
  • Coverage drop on changed lines — not absolute coverage. "Don't decrease the coverage of code you touched." Tools: SonarQube, Codecov diff coverage.
  • Critical security findings — Snyk/Dependabot/SAST flagging high-severity CVEs. Mid/low: warn, don't block.
  • Breaking API changes — oasdiff or a contract test suite catching schema regressions.
  • Linter errors (not warnings) — broken syntax, banned imports, unused variables. Style nits should be auto-formatted, not gated.
  • Performance regression beyond threshold — k6 thresholds breaching.
  • Migration safety checks — squawk, gh-ost dry-run for risky DDL.

Things NOT to block on:

  • Subjective code style — auto-format with Prettier/Black, don't argue.
  • Coverage absolute % (e.g. "must be above 80%") — incentivises low-quality tests written for line coverage.
  • Flaky tests — quarantine and triage; don't punish PRs for unrelated flakiness.
  • Warnings from new tooling rules — bulk-fix and then enable as a gate.

Granularity matters: PR gate (every change), pre-merge gate (final check), pre-deploy gate (before staging or prod). Each tier checks different things.

Override discipline: humans must be able to override in genuine emergencies (production fire). But every override is logged, reviewed weekly, and a Pareto distribution emerges — the same gate gets overridden repeatedly, signalling either it's wrong or the team needs help fixing the underlying issue.

The real test of a gate: does it block bad code without blocking good code? If devs work around it routinely, it's the wrong gate.

// WHAT INTERVIEWERS LOOK FOR

Defining gates objectively, naming what to block / not block, granularity by stage, and an override-with-audit policy. Bonus for the diff-coverage insight.

// COMMON PITFALL

Gating on absolute coverage percentage. Devs add no-op tests to hit the bar; real coverage of the changed code stays poor.