If a browser test suddenly starts failing after a deploy, the first instinct is often to blame the test. Sometimes that is correct, but when the failure appears only after a CDN purge, a cache-control tweak, or a new asset hash in the build output, the app itself may be fine. The real problem is that the test is loading a different version of CSS, JavaScript, or an image than the one the app expected.

This is one of those failure modes that can waste hours because it looks like a frontend bug, a timing issue, or a locator problem. In reality, the browser is doing exactly what you told it to do, it is just being served stale static assets, mismatched bundles, or partially propagated content. If you have ever seen a selector disappear after a seemingly harmless cache change, or watched a screenshot diff explode because the page still references an older stylesheet, you have probably met this class of bug.

This guide is a practical war-story style walkthrough for diagnosing why browser tests break after CDN or cache changes, how to prove the app is healthy, and how to harden your test and delivery pipeline so the next deploy is boring.

What this failure mode actually looks like

Browser automation failures caused by delivery-layer changes usually fall into a few recognizable patterns:

  • A page loads, but styles are broken or incomplete.
  • A click target exists in the DOM, but layout is wrong because CSS did not match the markup version.
  • JavaScript errors appear only in CI or only in certain regions.
  • Screenshot or visual regression tests fail with large diffs after a deploy.
  • A test passes locally, then fails in a clean CI environment, or vice versa.
  • A test intermittently sees old assets after a rollout, especially when a CDN edge has not fully invalidated.

The important clue is that the DOM may be present, but the assets behind it are inconsistent. In software testing terms, this is not just a functional defect, it is a delivery consistency problem that leaks into the automation layer. For a good general overview of testing and automation concepts, the Wikipedia pages on software testing, test automation, and continuous integration are useful reference points, but the real trick is knowing how these concepts interact with CDN behavior and browser caches.

When the markup is new but the CSS or JS is old, automation tends to fail in misleading ways. The app did not necessarily regress, the delivery path changed underneath it.

Why asset delivery changes break tests

Modern frontend stacks usually split responsibilities across several layers:

  • The app server generates HTML.
  • The build pipeline emits versioned static assets.
  • A CDN caches and serves those assets closer to users.
  • The browser caches some resources locally.
  • Service workers, if present, may cache assets independently of both the browser and the CDN.

A browser test can fail when any one of those layers serves an unexpected version. Common triggers include:

1. Asset hash changes

Many builds emit filenames like app.9f2a1c.js or styles.31c8d.css. That is good practice because it makes cache busting explicit. But if the HTML page, CDN, and deployed bundle are not in sync, the page may reference an old hash or a new hash that is not yet available everywhere.

Typical symptoms:

  • 404s for JS or CSS files.
  • ChunkLoadError in single-page apps.
  • Missing styles that make element coordinates change.
  • An app shell that loads, but a feature chunk never arrives.

2. Cache-Control header changes

A cache header such as Cache-Control: max-age=31536000, immutable is excellent for hashed assets, but dangerous if you apply it to resources that are not actually versioned. A change from short-lived caching to aggressive caching can make browsers or CDNs keep serving old files much longer than expected.

Typical symptoms:

  • A deployed fix does not appear in tests.
  • One test runner sees new code, another sees the old version.
  • A stale script keeps an old API contract alive.

3. CDN invalidation delays or partial propagation

Even when you invalidate correctly, edges do not update atomically. One region may get the new file instantly while another briefly serves old content. That is enough to make tests flaky, especially in parallel CI, distributed browsers, or cross-region execution.

Typical symptoms:

  • The same test passes and fails without code changes.
  • Failures cluster by geography or CI node.
  • Retry sometimes fixes the issue, which is a strong hint the problem is outside the test logic.

4. HTML and asset deployment drift

If the HTML is deployed before the matching assets, or vice versa, browsers can request a bundle that does not exist yet. This is especially common in multi-step release processes or blue-green deployments where the app server and CDN are updated separately.

Typical symptoms:

  • Short windows of failure after release.
  • Tests fail during rollout, then recover later.
  • Workers or users pinned to one version see different behavior than fresh sessions.

First question to ask, is the app broken or is the delivery broken?

Before changing selectors or adding waits, determine whether the app code is actually at fault. The fastest path is usually to inspect the network and the loaded assets in the failing test run.

Look for these signals:

  • 404 or 403 responses for JS, CSS, fonts, or images.
  • Mixed versions of assets from different deploys.
  • Old ETags or Last-Modified values that do not match the deploy time.
  • A browser console error referencing missing chunks, MIME type mismatches, or integrity failures.
  • A visible page that is obviously unstyled or partially initialized.

If the app is fine in a fresh manual browser session but a CI browser keeps failing, delivery inconsistency is more likely than a product defect.

A practical debugging habit is to capture the actual resource list from the browser during the failure. In Playwright, for example, you can log failed requests and page errors:

import { test } from '@playwright/test';

test('debug asset failures', async ({ page }) => {
  page.on('requestfailed', request => {
    console.log('FAILED', request.url(), request.failure()?.errorText);
  });

page.on(‘pageerror’, error => { console.log(‘PAGE ERROR’, error.message); });

await page.goto(‘https://example.com’); });

That snippet will not fix the issue, but it can quickly confirm whether you are dealing with a stale asset or a true UI defect.

A debugging checklist that saves time

When a browser test starts failing after a cache or CDN change, use a checklist instead of guessing.

1. Confirm the exact asset versions in the failing run

Open the network tab or log request URLs and response headers. You want to know:

  • Which HTML was loaded.
  • Which JS and CSS files were requested.
  • Whether the filenames include hashes.
  • Whether any request was redirected.
  • Whether any request returned an error page with a 200 status.

If your automation framework supports it, compare the asset URLs in the failing run with a passing run. A mismatch often reveals the issue immediately.

2. Check response headers for cache behavior

Inspect headers on the HTML and static assets:

  • Cache-Control
  • ETag
  • Age
  • Vary
  • Surrogate-Control
  • CDN-specific cache headers if available

A common anti-pattern is long-lived caching on HTML. HTML usually needs shorter caching than hashed assets because it is the pointer to the current bundle set. If HTML is cached too aggressively, the browser may keep loading references to old asset hashes.

3. Look for deploy-order problems

Ask whether the release process uploads bundles before publishing HTML, or the reverse. If both happen through separate jobs, there may be a short but real inconsistency window.

This is where browser tests in CI often discover the bug first, because they run during the rollout window and across multiple environments. A human clicking around later may never notice.

4. Reproduce with a clean browser context

Use a brand-new browser profile or an incognito session. Clear caches when necessary. If the failure disappears, that is a clue that the browser cache, service worker, or persistent storage is involved.

5. Reproduce from multiple vantage points

Try the same test from:

  • Local machine with cache cleared
  • CI runner
  • Another region or network if possible
  • A container with a clean browser profile

If failures cluster by location, suspect CDN propagation or regional caching behavior rather than the test code itself.

The most common root causes, and how to recognize them

Stale static assets after an HTML deploy

This is the classic mismatch. HTML references app.abc123.js, but the CDN edge or browser still has a previous HTML document pointing at app.old999.js, or the referenced file was not available everywhere when the page loaded.

Recognition signs:

  • 404 on a hashed file.
  • Console errors like chunk load failures.
  • The page loads the shell, but feature widgets never render.

Fix direction:

  • Ensure atomic deploys of HTML and assets.
  • Use versioned directories or release manifests.
  • Make HTML cache short-lived.

Overly aggressive caching on HTML

HTML should usually be the freshest layer. If it is cached like a static image, the browser may keep using an old asset map long after deployment.

Recognition signs:

  • Users and tests keep seeing an old app shell.
  • Refreshing once or twice eventually “fixes” it.
  • CI failures disappear when a new runner is used.

Fix direction:

  • Reduce HTML TTL.
  • Add revalidation rather than long-lived freshness.
  • Keep asset hashes immutable and HTML volatile.

CDN invalidation did not finish

You invalidated the path, but not all edge nodes have updated yet. The app is healthy in origin and in one region, but stale in another.

Recognition signs:

  • Intermittent failures after deploy, then recovery.
  • Geography-sensitive behavior.
  • The same URL returns different content in separate runs.

Fix direction:

  • Avoid requiring invalidation for every asset by using content-addressed filenames.
  • If invalidation is necessary, understand its propagation window.
  • Consider rollout sequencing that avoids serving references before files exist.

Service worker interference

A service worker can cache old application assets even when the CDN and browser cache are correct. This often gets overlooked because the page seems to load normally.

Recognition signs:

  • Problems persist even after cache clearing in the browser UI.
  • Only some clients fail, usually those with an older SW version.
  • Network traffic is unexpectedly sparse, because the SW intercepts requests.

Fix direction:

  • Inspect and version the service worker carefully.
  • Make update logic explicit.
  • Test with the service worker disabled when debugging.

Mixed deploys in microfrontend or multi-bundle systems

If the page composes several independently deployed bundles, tests can fail when one bundle is newer than another.

Recognition signs:

  • One widget breaks while the rest of the page works.
  • Shared components misbehave due to version skew.
  • CSS class names or DOM structure changed in one bundle but not another.

Fix direction:

  • Coordinate release manifests.
  • Avoid unversioned shared contracts.
  • Add compatibility checks between bundles.

What to inspect in the browser, specifically

When debugging browser automation failures, the console and network panels are your best friends.

Network tab

Check the failed request chain and confirm:

  • The requested URL matches the current build.
  • The response is from the expected origin or CDN.
  • The status code is correct.
  • Content-Type is correct, especially for scripts and styles.
  • No unexpected redirects are occurring.

If a JS bundle is served with the wrong MIME type, the browser may refuse to execute it. That can look like a logic bug, but it is a delivery issue.

Console tab

Common clues include:

  • ChunkLoadError
  • Loading CSS chunk failed
  • Unexpected token < when HTML is returned instead of JS
  • Refused to apply style because...
  • CSP errors when the asset path or host changed

Application tab

If you suspect service workers or storage caches, inspect those too. A stale service worker is one of the more annoying reasons a test can keep loading the wrong version after multiple retries.

How to make browser tests less sensitive to delivery-layer drift

You do not want to hide real defects, but you also do not want every cache event to become a false alarm. There are a few practical hardening steps.

Prefer deterministic asset versioning

Hashed filenames are good, but only if the whole delivery chain honors them consistently. Build systems should emit a manifest that maps logical names to the exact deployed files. The app should consume that manifest or otherwise guarantee it is serving a coherent set.

A release should behave like a snapshot, not a loose collection of files.

Keep HTML short-lived, assets long-lived

This is the standard model for static asset delivery:

  • HTML: short cache or revalidated frequently.
  • Hashed JS/CSS/images: long cache, immutable if truly content-addressed.

That combination reduces stale shell problems while letting immutable assets benefit from caching.

Treat CDN invalidation as a migration step, not a magic reset

Invalidation is useful, but it is not a substitute for good cache design. If your deployment only works after a manual purge, the system is too dependent on synchronization timing.

Add asset integrity checks in CI

For important releases, it helps to verify that the published HTML references assets that actually exist and return the expected content type. A simple check can catch broken deploy sequencing before browser tests do.

For example, a CI job might fetch the page, parse asset URLs, and fail if any script or stylesheet returns a non-200 response.

curl -I https://example.com
curl -I https://example.com/assets/app.9f2a1c.js
curl -I https://example.com/assets/styles.31c8d.css

That is not a complete test, but it is a good early warning system.

Use browser tests that fail loudly on asset errors

Make sure your browser automation captures page errors and failed requests, not just assertion failures. A layout check that fails with “button not visible” is less useful than a test that also tells you the CSS bundle never loaded.

Stabilize selectors, not layout assumptions

If the visible issue is caused by stale CSS, tests that rely on coordinates or visual layout will be extra flaky. Prefer semantic locators when possible. That does not solve caching issues, but it reduces the number of unrelated failures when styles shift.

A practical triage workflow

When a test breaks after a CDN or cache change, follow this order:

  1. Capture the failure once with network and console logging enabled.
  2. Check whether the page loaded stale or missing assets.
  3. Compare asset URLs to the latest build output.
  4. Inspect cache headers on HTML and assets.
  5. Re-run in a clean browser profile.
  6. Test from another region or CI runner.
  7. Verify deploy sequencing and CDN invalidation timing.
  8. Check for service worker involvement.

This workflow prevents the common mistake of adding waits or retry logic to a delivery problem. Retries may mask the issue, but they do not explain it.

If a retry fixes the failure, that is not proof the test was wrong. It may simply mean the cache eventually caught up.

Example: diagnosing a flaky Playwright test after a release

Suppose a Playwright test fails because the settings drawer never opens. The DOM shows the button, but clicking it does nothing. You inspect the console and see a chunk load error. The network panel shows the page requested /assets/settings.4c1d2.js, but the CDN returned 404 for a short window after release.

In that situation, the right response is not to add a longer click timeout. The correct fix is to make sure the app does not publish HTML that references a chunk until the chunk is available everywhere, or to design the release so the old bundle remains valid until the new one is fully live.

That kind of failure is especially common in single-page apps with lazy-loaded routes. A route can work locally, but production lazily requests a chunk that was not propagated yet. In CI, the test lands on the route immediately after deploy and becomes the first witness.

Release engineering practices that prevent these failures

The best browser test is still a stable delivery process. A few habits pay off quickly:

  • Deploy assets and HTML as a coherent release unit.
  • Avoid changing cache headers casually, because they affect existing clients immediately.
  • Version build artifacts predictably.
  • Test the app through the same CDN path used in production, not just against origin.
  • Add smoke checks that confirm asset availability before declaring a release healthy.
  • If you use service workers, include them in release and rollback planning.

A lot of frontend flakiness disappears when you stop thinking of CSS and JS as “just files” and start treating them as part of the runtime contract.

When to suspect the test instead

Sometimes the test really is at fault. If network logs show the correct assets loaded, no console errors appear, and the app behaves correctly in a clean browser session, then the failure may be in the automation itself.

Suspect the test when you see:

  • Overly brittle selectors.
  • Hard-coded waits that assume timing instead of observing readiness.
  • Assertions tied to CSS classes that are expected to change.
  • Visual checks that are too sensitive to minor layout shifts.

But if the app is missing bundles, returning stale content, or serving mismatched versions, fix delivery first. A broken cache strategy will keep producing the same class of flakes until the root cause is removed.

Final takeaway

When browser tests break after CDN or cache changes, the fastest path is to stop treating the failure as a purely test-side problem. Asset delivery is part of the system under test. If stale static assets, cache-control headers, or CDN invalidation timing changed the version the browser sees, your automation is simply reporting the inconsistency back to you.

The practical mindset is this, prove which version the browser loaded, prove whether the assets match the deployed HTML, and only then decide whether to touch the test. Once you build that habit, browser automation failures become much easier to classify, and your deploys become a lot less mysterious.