What Makes a Test Flaky
A flaky test is one that produces different outcomes (pass vs fail) across runs without any code change, usually caused by race conditions between the test and the application—an element that hasn't finished animating, a network request that hasn't resolved, or a WebSocket update arriving after the assertion already ran. Flakiness erodes trust in a test suite faster than outright failures because engineers start ignoring red builds, assuming 'it's probably just flaky,' which is exactly the failure mode that lets real regressions slip through unnoticed.
Cricket analogy: A flaky test is like a fielder who drops an easy catch on one over and takes a screamer the next — the same skill level producing inconsistent outcomes because of a timing lapse, not a change in ability.
Configuring Retries
Playwright's retries config option (set in playwright.config.ts or per-project) reruns a failing test up to N additional times before marking it as failed, and on CI the default is 2 retries while local runs default to 0 so developers see real failures immediately during development. Each retry starts fresh—new browser context, cleared state—and the HTML report labels retried tests distinctly (flaky, shown in yellow) versus tests that failed on every attempt, which lets teams triage true bugs separately from timing-sensitive tests worth investigating.
Cricket analogy: Configuring retries is like the third-umpire review system giving a batter's dismissal a second look via DRS before the decision is final, but a captain wouldn't want unlimited reviews burning through the innings.
// playwright.config.ts
export default defineConfig({
retries: process.env.CI ? 2 : 0,
projects: [
{
name: 'checkout-flow',
retries: 1, // override per project for a known-sensitive area
},
],
});Auto-Waiting vs Retries: Fixing Root Causes
The correct fix for most flakiness isn't retries at all but Playwright's auto-waiting and web-first assertions: methods like expect(locator).toBeVisible() or page.getByRole('button').click() automatically poll the DOM for up to the configured timeout instead of asserting once immediately, so a button that appears 200ms after a fetch resolves is handled correctly without an explicit waitForTimeout. Retries mask symptoms by giving a flaky interaction more chances to get lucky, whereas fixing the underlying race—waiting on the right condition—makes the test reliably pass on the first attempt.
Cricket analogy: Auto-waiting is like a batter waiting for the ball to actually arrive before playing the shot instead of swinging blindly at where they guessed it would be — timing the action to the real event, not a fixed clock.
Web-first assertions like expect(locator).toBeVisible() retry the underlying check internally until it passes or the timeout expires, which is different from a plain assert that checks the DOM exactly once. Prefer these over manual waitForTimeout() calls, which guess at a delay rather than waiting on a real condition.
Test Annotations for Known Flakiness
For genuinely known-unstable scenarios, Playwright offers test.fixme() to skip a test while flagging it as broken (rather than silently deleting it), test.slow() to triple its timeout for legitimately slow operations, and testInfo.retry inside a test body to branch logic on which attempt is currently running—useful for clearing browser storage or logging extra diagnostics only on retries. Overusing these annotations to paper over real application bugs, however, just relocates the flakiness problem into a growing pile of technical debt that nobody circles back to fix.
Cricket analogy: test.fixme is like officially flagging an injured player as unavailable on the team sheet instead of pretending they're fit and hoping they perform anyway.
test('checkout completes after slow payment gateway', async ({ page }, testInfo) => {
if (testInfo.retry > 0) {
// Only clear storage on a retry, to rule out stale-state causes
await page.context().clearCookies();
}
await page.goto('/checkout');
await expect(page.getByRole('button', { name: 'Pay now' })).toBeEnabled();
});
test.fixme('legacy export button is unreliable pending backend fix', async ({ page }) => {
// Skipped but tracked, not silently deleted
});Retries are a safety net, not a fix. A test that only passes on the second or third attempt is telling you there's a real race condition in either the app or the test — silencing it with retries just delays the pain until the flake rate creeps high enough to slow down the whole team.
- Flaky tests produce inconsistent results from race conditions, not code changes, and erode trust in the suite.
- The retries config option reruns failing tests, defaulting to 2 on CI and 0 locally.
- The HTML report labels a test that passed after retrying as 'flaky', distinct from a hard failure.
- Web-first assertions like toBeVisible() auto-poll the DOM instead of asserting once immediately.
- Fixing the real race condition is better than relying on retries to mask it.
- test.fixme() and test.slow() flag known-unstable tests explicitly instead of deleting or ignoring them.
- testInfo.retry lets a test branch logic based on which attempt is currently executing.
Practice what you learned
1. What primarily causes a test to be flaky?
2. What is Playwright's default retries value on CI versus locally?
3. How does the HTML report distinguish a test that failed once but passed on retry from one that failed every attempt?
4. Why is fixing auto-waiting conditions preferred over relying on retries?
5. What does test.fixme() do?
Was this page helpful?
You May Also Like
Reporters and HTML Reports
How Playwright's built-in reporters work, how to configure and combine them, and how to build a custom reporter for bespoke workflows.
Playwright in CI/CD Pipelines
How to configure Playwright to run reliably on CI, including GitHub Actions setup, Docker images, and CI-aware config adjustments.
Parallel Workers and Sharding
How Playwright Test runs suites faster using isolated worker processes locally and distributes them across multiple CI machines with sharding.