Browser tests that pass on a laptop and fail in CI are frustrating enough. Add asset hash changes to the mix, and the failure often looks random: a missing bundle, a stale CSS file, a 404 on a script with a different filename, or a test that only fails when the build artifact changes between runs.

This is one of those problems where the surface symptom is misleading. The test may not be broken at all. The product may also not be broken. The real issue can live in the space between the app, the build system, the cache layer, and the browser runner. That is why the phrase browser tests fail in CI after asset hash changes is worth treating as a debugging category, not just an error message.

A hash in a filename is a hint that the asset is meant to be immutable. When a test sees a different hash, the first question is not “why did the test change?”, but “which layer served the wrong version?”

What asset hashes usually mean

Asset hashing is a cache-busting technique. Build tools such as Webpack, Vite, Rollup, and many framework-specific pipelines emit filenames like app.4f3a91c2.js or main.91d0b.css. The hash changes when the file content changes, which helps browsers and CDNs cache aggressively without serving stale bundles.

That is good for users, but it creates a few failure modes for automated browser tests:

  • The app shell references the wrong manifest or build output.
  • The browser or proxy cache serves a stale HTML page with old asset references.
  • CI and local environments build different artifacts, so tests target different bundle names.
  • A deploy race causes tests to hit a page that references assets from a newer or older release.
  • A service worker, CDN, or reverse proxy caches one layer longer than expected.

The key point is that the hash is not the bug. It is the clue.

Start by classifying the failure

Before changing selectors or rewriting waits, classify the failure by where it appears.

1. HTML loads, but JS or CSS assets 404

This is the most common case. The page loads, but a script or stylesheet request fails because the HTML points at a file name that is not present in the deployment bundle. Typical root causes include:

  • A stale HTML page in cache
  • Deployment order problems, where HTML is published before assets
  • Build output not fully uploaded to object storage or a CDN origin
  • A release directory mismatch between app server and static server

2. The page loads, but the app behaves as if old code is running

In this case, the network panel may show successful requests, but the UI still uses stale logic. Common causes include:

  • Service worker caching old assets
  • A CDN caching the HTML too long
  • Browser cache reusing a previous chunk
  • A test runner reusing a browser profile across runs

3. The test fails only in CI, not locally

This usually indicates environment drift rather than a product defect. Compare:

  • Build commands and environment variables
  • Node, package manager, and browser versions
  • Docker image and base OS
  • Cache configuration in the CI runner
  • Parallelism and timing differences

4. The failure moves around between different tests

That is a sign of shared-state contamination, often from cache, service workers, or test data. If one test primes the browser with a stale asset path, later tests may fail in places that look unrelated.

First collection of evidence, not first guess

The biggest mistake is to start changing timeouts or adding retries before collecting evidence. Instead, capture the exact asset path and the response chain.

For Playwright, a small amount of request logging can help identify whether the browser is asking for a file that never existed or one that existed locally but not in CI.

import { test, expect } from '@playwright/test';

test('home page loads assets', async ({ page }) => {
  page.on('response', response => {
    if (response.request().resourceType() === 'script') {
      console.log(response.status(), response.url());
    }
  });

await page.goto(‘http://localhost:3000’); await expect(page.locator(‘h1’)).toBeVisible(); });

For a failing CI run, the useful details are usually:

  • The exact asset URL requested
  • The status code
  • Whether the URL came from HTML, a manifest, or runtime code
  • Whether the same URL exists in the deployed artifact

If the hash in the request differs from the hash in the deploy output, the question becomes, where did the stale reference come from?

Compare the local and CI artifact, not just the test output

Browser tests often fail because the application artifact is different, even though the source code is the same. That is especially true when build metadata includes timestamps, commit IDs, or environment-specific config.

Check these items side by side:

  • The exact build command
  • Environment variables that affect asset paths, such as PUBLIC_URL, ASSET_PREFIX, base, or CDN host settings
  • Whether CI uses a production build while local testing uses a dev server
  • Whether source maps are enabled differently
  • Whether build caching is reusing a stale output directory

A practical comparison method is to archive the production build output from CI and inspect the generated manifest or HTML.

bash cat dist/manifest.json | jq ‘.””’

If your build system emits a manifest file, compare it with the HTML that the test actually loaded. If the HTML references a chunk that is not in the manifest, the bug may be in deploy sequencing or caching, not in the browser automation.

Inspect the network waterfall, not just the assertion

A failed assertion like “button not visible” often hides the real issue. If a JS bundle failed to load, the component never mounted, and the assertion is just collateral damage.

Look for these patterns in the browser network trace:

404 on chunk or asset files

This is the simplest static asset failure. The browser asked for a file and the server could not find it. In practice, this means one of the following:

  • The HTML references a hash that was not deployed
  • The asset was deployed under a different path
  • The base URL in CI is different from local

200 response, but old content

A proxy or CDN may respond with a cached older asset. The file name can even match if the HTML itself is stale. That is a classic cache busting bug.

Redirects to a fallback page

Some static hosting setups return index.html for missing assets. That can turn a chunk request into a misleading 200 response with HTML content, which then causes syntax errors in the browser console.

Mixed-origin requests blocked by policy

If CI tests run against a different host or port, asset URLs can trigger CORS or CSP issues that never show locally.

Use browser console logs and JS errors as evidence

Console errors are often more revealing than the test failure message. Look for messages such as:

  • Failed to load resource
  • ChunkLoadError
  • Unexpected token '<'
  • Refused to execute script
  • ServiceWorker failed to register

A ChunkLoadError usually means the runtime expected a hashed bundle that the server did not provide. Unexpected token '<' often means an HTML fallback was served where JavaScript was expected.

In Playwright, capture console output and uncaught exceptions when a CI run fails.

page.on('console', msg => console.log('console:', msg.type(), msg.text()));
page.on('pageerror', error => console.log('pageerror:', error.message));

If the browser is screaming about a missing chunk while the test is failing on a selector, fix the chunk problem first. The selector might be fine once the app actually loads.

Check for cache layers that outlive your test run

Asset hash changes and caching are tightly coupled. If a test fails only after a new release, inspect every cache layer that can preserve old references.

Browser cache

A reused browser profile can keep scripts, styles, and even service worker state from a previous run. In CI, this is often caused by shared user-data directories or persistent contexts.

If your test suite should start clean, use isolated browser contexts and avoid reusing profiles unless you intentionally need persisted state.

Service workers

Service workers can keep old app shells alive after deploys. They are useful in production and annoying in tests when they serve stale files.

For debugging, disable or unregister them in the test setup if they are not part of the scenario.

Reverse proxies and CDNs

If HTML is cached too long, the page can reference old chunk names after a release. If assets are cached incorrectly, the server may keep serving stale chunks even after a redeploy.

CI workspace caches

Build caches can cause a pipeline to reuse old output directories. That is particularly risky if the deploy step assumes a clean dist directory but the runner keeps previous files.

A simple diagnostic is to force a clean build in CI and compare outcomes. If the test passes after cleaning, you have narrowed the issue to build or cache state.

Look for environment drift, not only timing drift

Many teams are trained to blame timing when a browser test fails in CI. Timing is certainly a factor, but asset-hash failures often come from environment drift instead.

Environment drift includes:

  • Different Node versions
  • Different package lockfiles or transitive dependencies
  • Different build modes, such as development vs production
  • Different environment variables for asset paths
  • Different container images or browser versions
  • Different default ports or hostnames

A useful practice is to compare the actual runtime environment in CI with the one used locally. In GitHub Actions, for example, print the relevant variables and version info early in the job.

- name: Debug environment
  run: |
    node -v
    npm -v
    printenv | sort | grep -E 'PUBLIC_URL|ASSET|BASE|CI'

The goal is not to log everything forever. The goal is to capture enough state to explain why the local and CI browsers saw different asset URLs.

Make the test tell you what it is waiting for

If a browser test waits for visible UI, it can fail without telling you whether the page was broken, slow, or stale. Add assertions that check the asset-loading path directly when you suspect hash-related issues.

Examples of useful checks:

  • Wait for a specific request to finish with HTTP 200
  • Assert that the JS bundle was loaded from the expected host
  • Verify that the HTML contains the manifest-derived asset path
  • Check for absence of ChunkLoadError in the console

A Playwright example that waits for the main script request:

typescript

await page.goto('https://staging.example.com');
await page.waitForResponse(resp =>
  resp.url().includes('/assets/app.') && resp.status() === 200
);

This does not replace UI assertions. It adds a lower-level signal so you can distinguish a rendering problem from an asset delivery problem.

Reproduce CI conditions locally, then narrow the gap

A local pass is only interesting if the local environment matches CI closely enough to be meaningful. For asset-hash failures, try to reproduce the CI build and serving path.

A useful sequence is:

  1. Run the same build command used in CI.
  2. Serve the built output, not the dev server.
  3. Use the same browser version or container image if possible.
  4. Start from a clean cache.
  5. Open the same URL path the CI test uses.

A common discovery is that local testing uses a hot-reload server that resolves assets dynamically, while CI uses built files served from a static host. That difference alone can explain why a missing or stale chunk never appears locally.

Separate product regressions from deployment noise

Not every browser failure linked to asset hashes is a deployment issue. Sometimes the product really is broken. The task is to separate the two.

Here is a practical rule of thumb:

  • If the app source code references the wrong route, selector, or component logic, treat it as a product regression.
  • If the browser requests an asset that the deployment should have provided, treat it as a delivery or cache issue first.
  • If the same code passes against a clean local production build but fails in CI, focus on the pipeline and environment.

A good investigation keeps two hypotheses alive for as long as the evidence allows, product fault and environment fault. The job is to eliminate one with proof, not preference.

A debugging checklist you can apply quickly

When browser tests fail in CI after asset hash changes, move through this checklist in order:

  1. Capture the failing asset URL and response code.
  2. Check whether the requested hash exists in the deployed artifact.
  3. Verify the HTML was built and published from the same release as the asset bundle.
  4. Inspect browser console logs for ChunkLoadError, CSP errors, or HTML served as JS.
  5. Compare CI and local build commands, environment variables, and browser versions.
  6. Clear or bypass browser, CDN, service worker, and workspace caches.
  7. Re-run the test against a clean production build served locally.
  8. If the failure persists, reduce the problem to a single page load and a single network request.

That sequence tends to expose the fault line faster than randomly changing waits or adding retries.

What not to do

There are a few anti-patterns that make this class of problem harder to solve:

  • Adding arbitrary sleeps instead of checking network state
  • Re-running the same failing job until it passes
  • Masking the failure with test retries before understanding the cause
  • Deleting all caches without recording what changed
  • Changing locators when the app never loaded correctly
  • Assuming a local pass proves the app is healthy in CI

Retries can reduce noise, but they are not a substitute for understanding why a hash-based asset reference diverged.

When to change the test, and when to change the system

A good browser test suite does not try to paper over deployment defects. It should help reveal them.

Change the test when:

  • It depends on a dev-only server behavior
  • It assumes persistent browser state that CI does not provide
  • It fails to wait for the actual network condition relevant to the page
  • It needs clearer evidence, such as console logs or request tracing

Change the system when:

  • HTML and assets are deployed out of sync
  • Caching rules keep stale app shells alive
  • Service worker behavior conflicts with release flow
  • Build artifacts differ between environments in ways that should not happen

The healthiest outcome is not “the test passes.” It is “the test now distinguishes a real regression from a deployment artifact problem.”

Closing thought

Asset hashes exist to make caching safer for real users, but they expose weak points in build and deployment pipelines. When browser tests fail in CI after hash changes, the fastest path to a useful answer is usually not deeper waits or more retries. It is tracing the asset path from HTML to network request to deployed artifact, then comparing that path across local and CI environments.

That is investigative testing in practice: gather evidence, test competing explanations, and let the environment tell you where the mismatch lives. For software teams working with software testing and test automation, this is one of the clearest reminders that the test is often only the messenger.