How to Test `

`, Focus Traps, and Escape-Key Behavior Without Missing Modal Regressions

A modal bug is rarely just a visual bug. If Escape closes the dialog but focus lands somewhere random, or the overlay looks correct while the background is still clickable, the regression will survive most shallow tests. That is why the safest way to test modal behavior is to assert the interaction contract, not just the presence of a popup.

For modern apps, that contract usually has four parts:

1. The dialog opens and becomes the active interaction target.
2. Keyboard focus is trapped inside it while it is open.
3. Escape closes it only when the product rule says it should.
4. The rest of the page is inert, visually and interactively.

If you are writing browser automation for dialog flows, start there. The rest of this article shows what to assert, how native <dialog> differs from custom overlays, where flakiness comes from, and how to separate a real bug from a test harness problem.

First, define the modal you are testing

People say “modal” when they may mean three different implementations:

Native <dialog>

The browser provides the element and the behavior. When opened modally with showModal(), focus management, top-layer behavior, and close mechanics are part of the platform model. The dialog can also be closed with close() and can emit a cancel event when Escape is pressed, depending on how the page handles it.

Custom overlay

This is a div-based UI that looks like a modal but implements focus trapping, Escape handling, and backdrop blocking in app code. Many design systems use this pattern.

Hybrid modal

Some apps use <dialog> for semantics but layer custom focus logic, transitions, or analytics on top. These are the hardest to test because a browser-native rule and an app-specific rule can both affect the same interaction.

The test strategy changes with the implementation. If you write the same assertions for native <dialog> and a custom overlay, you will miss one class of regressions and create false failures in another.

What to assert for modal behavior

For browser dialog testing, treat the modal as an interaction boundary. The useful assertions are usually these:

1. The dialog is visible and active

Check that the modal is actually open, not just rendered in the DOM. For native <dialog>, that means verifying the open state and a visible accessible surface. For a custom overlay, verify its visible layer and the expected ARIA role if your component follows an accessible pattern.

2. Initial focus is correct

When the modal opens, focus should move to the right element. That may be the first actionable control, a heading with tabindex="-1", or a specific primary action. Do not assume the browser or library will always choose the same target unless the product rule says so.

3. Tab and Shift+Tab stay inside

This is the heart of focus trap testing. Repeated Tab presses should cycle within the modal, not escape to page content. Shift+Tab should wrap in the opposite direction.

4. Escape closes only when allowed

Escape key modal close behavior is often conditional. Some dialogs should close on Escape, others should block it when the user has unsaved changes, critical workflow steps, or an explicit “must confirm” requirement. Your test should assert the product rule, not a universal assumption.

5. Background interaction is blocked

If the modal is open, background buttons, links, and inputs should not respond. For native modal dialogs this is where inert behavior matters, and for custom overlays you may need to verify pointer-events, focus suppression, and aria-hidden handling separately.

6. Close restores focus predictably

After close, focus should return to the opener or another deterministic target. This is one of the most common regressions because it affects keyboard users and later test steps.

A compact decision table for what to test

How to test native <dialog> in browser automation

If your app uses showModal(), write tests against the browser-visible result and the keyboard contract. Here is a small Playwright example.

import { test, expect } from '@playwright/test';

test('modal dialog traps focus and closes with Escape', async ({ page }) => {
  await page.goto('/dialog-demo');

  await page.getByRole('button', { name: 'Open dialog' }).click();

  const dialog = page.locator('dialog[open]');
  await expect(dialog).toBeVisible();

  await expect(page.getByRole('heading', { name: 'Edit profile' })).toBeFocused();

  await page.keyboard.press('Tab');
  await expect(page.getByRole('button', { name: 'Save' })).toBeFocused();

  await page.keyboard.press('Escape');
  await expect(dialog).toBeHidden();
  await expect(page.getByRole('button', { name: 'Open dialog' })).toBeFocused();
});

A few notes about that test:

• It checks the open state through the element itself, not a CSS class.
• It uses keyboard navigation instead of only clicking buttons.
• It verifies focus returns after close, which catches a class of accessibility regressions that pure DOM assertions miss.

If your app listens for the cancel event to prevent closing, add a specific test for that branch:

test('escape does not close when unsaved changes are blocked', async ({ page }) => {
  await page.goto('/dialog-demo');
  await page.getByRole('button', { name: 'Open dialog' }).click();
  await page.getByRole('textbox', { name: 'Notes' }).fill('draft');

  await page.keyboard.press('Escape');
  await expect(page.locator('dialog[open]')).toBeVisible();
  await expect(page.getByText('Discard changes?')).toBeVisible();
});

That second test matters because “Escape closes the modal” is not a universal rule. It is a product rule.

Focus trap testing, without brittle tab-count assumptions

A lot of focus trap tests fail because they assume one Tab press equals one specific target forever. That breaks as soon as a dialog gets a new link, hidden sentinel node, or conditional control.

A better approach is to assert containment and order, not hard-coded tab counts.

Better checks

• Focus starts on a known element after open.
• Repeated Tab eventually cycles back to the start.
• Shift+Tab cycles in the reverse direction.
• Focus never lands on page content behind the modal.

Avoid these traps

• Counting exact Tab presses unless the dialog content is stable and intentionally small.
• Assuming the tab order is identical across browsers when the browser or platform may influence focus ring behavior.
• Using document.activeElement in a way that ignores shadow DOM or nested browsing contexts.

For custom overlays, it is worth checking that tabbable background elements are inaccessible while the modal is open. If the app uses a trap library, test the contract that the library is supposed to provide, not the internal implementation detail.

When a test fails on the second Tab press, the bug may be in the dialog, the test, or the focusable elements added by the browser shell. Inspect the actual active element before rewriting the component.

Testing inert background behavior

The background must be more than visually dimmed. It should not receive clicks, keyboard focus, or accidental scroll interaction.

For native <dialog>, the browser is expected to manage the blocking behavior when the dialog is shown modally. For custom overlays, you need to prove it yourself.

Useful assertions include:

• Clicking a background button does not trigger its handler.
• Tabbing does not move focus to the page behind the overlay.
• Screen-reader facing semantics are consistent with your component pattern, especially if the background is hidden with aria-hidden or the dialog uses aria-modal="true".

A straightforward Playwright check for pointer blocking looks like this:

test('background is blocked while modal is open', async ({ page }) => {
  await page.goto('/dialog-demo');
  await page.getByRole('button', { name: 'Open dialog' }).click();

  await page.getByRole('button', { name: 'Background action' }).click({ trial: true });
  await expect(page.getByText('Background clicked')).toHaveCount(0);
});

If your automation framework supports it, use a real click and verify the resulting state. A trial click is useful as a probe, but it should not be your only proof when you are validating the business rule.

Why modal tests get flaky

Most modal flakiness comes from timing, not from the modal itself.

1. Animation timing

If the dialog animates in or out, focus may be assigned before the animation completes, or the element may still intercept events while visually hidden. Either disable animations in test runs or wait for a stable state that the UI exposes intentionally.

2. Portal and overlay rendering

Custom modals often render outside the page’s main app root. If your test uses a brittle selector tied to the component tree, it may miss the real rendered node.

3. Deferred focus management

Framework code may use requestAnimationFrame, microtasks, or a setTimeout to move focus. That can make a naive “click then assert immediately” test fail. Prefer event-driven waiting, for example waiting for the open state, then checking focus.

4. Overlapping dismiss paths

Escape may close the modal, the backdrop may close the modal, and the close button may close the modal. If multiple paths update the same state, one of them can race the others and leave the DOM in a half-closed state.

5. Browser differences

Focus behavior, selection behavior, and accessibility tree timing can vary across engines. If a test only passes in one browser family, check whether you are depending on a browser quirk rather than a product contract.

A quick debugging checklist when a modal test fails

When a modal test breaks, answer these questions in order:

1. Is the dialog actually open, or just present in the DOM?
2. What is document.activeElement right after open?
3. Is Escape being handled by the page, the dialog component, or both?
4. Can the background still receive clicks or focus?
5. Did an animation or async state update delay the focus trap?
6. Are you testing native <dialog> semantics or a custom overlay implementation?

A tiny debugging helper can save time during triage:

await page.evaluate(() => ({
  activeTag: document.activeElement?.tagName,
  activeName: document.activeElement?.getAttribute('aria-label') || document.activeElement?.textContent,
  openDialogs: Array.from(document.querySelectorAll('dialog[open]')).length
}));

Use this kind of probe to understand state, then convert the insight into a stable assertion.

A practical rule set for real teams

If you need a simple standard for modal tests, this is a good baseline:

• Every modal gets one open-state test.
• Every keyboard-modal gets one focus-trap test.
• Every dismissible modal gets one Escape test.
• Every destructive or unsaved-changes modal gets one “Escape is blocked or confirmed” test.
• Every modal gets one close-and-restore-focus test.
• Native <dialog> and custom overlays are tested by the same user outcome, but not by the same internal mechanics.

That last point is important. The browser implementation and the app implementation may solve the same user problem differently, so the assertions should reflect the user-facing contract, not the code path.

When a failure is probably a real bug

Treat it as a product bug when:

• Background controls are still clickable while the modal is open.
• Tab reaches content behind the overlay.
• Escape closes a dialog that is explicitly supposed to block dismissal.
• Focus disappears after close or lands on an unrelated control.
• The dialog is visible but not the active interaction surface.

Treat it as a test or harness problem when:

• The failure only happens with animation enabled.
• The element is in the DOM but not open.
• The test assumes a specific tab sequence after the component changed.
• The framework reports a click failure because the dialog is intentionally blocking background interaction.

Closing thought

The best modal tests are not long. They are precise. If you can prove that open, focus trap, Escape behavior, inert background interaction, and focus restoration all work, you will catch the regressions that users notice first. That is enough to turn browser dialog testing from a flaky UI chore into a reliable contract test.

FAQ

How do I test dialog modal behavior in browser automation without brittle selectors?

Use role-based selectors where possible, then assert the state change that matters, such as dialog[open], focused element, and blocked background interaction. Avoid tying the test to implementation-specific CSS classes.

What is the difference between a focus trap and inert background testing?

A focus trap keeps keyboard focus inside the modal. inert background testing checks that the rest of the page cannot be interacted with while the modal is open. They solve related but different problems.

Should Escape always close a modal?

No. Escape should follow the product rule. Some dialogs must close, some must confirm, and some must ignore Escape until a required action is complete.

Why does a modal test pass locally but fail in CI?

Animation timing, async focus updates, and browser differences are the usual causes. Disable animations in test runs when possible, or wait for a stable dialog state before checking focus.

Do native <dialog> and custom overlays need the same tests?

They need the same user outcomes, but not the same internal assertions. Native <dialog> should be checked as a browser feature, while custom overlays need extra verification of the app’s focus and blocking logic.