e2e: agentic E2E tests for web and mobile

e2e is an open-source framework for agentic end-to-end testing of web and mobile apps (Apache-2.0, npm e2e). A test states the goal in plain English, an AI agent works out the clicks, and locators with expect check the result without asking a model. A verified agent step is recorded and replays with no model calls.

Everything here is read from the source at e2e@0.23.1. Docs: e2e.com/docs.

What an e2e test looks like

From examples/with-vite, an agent step with a deterministic check next to it:

// tests/agentic.e2e.ts
import { test } from '@e2e-dev/web';
import { expect } from 'e2e';

// Agent steps call the model in e2e.config.ts. Without a key these tests are skipped.
const skip = process.env.AI_GATEWAY_API_KEY ? false : 'set AI_GATEWAY_API_KEY to run agent tests';

test('the agent gets a greeting', { skip }, async ({ app, agent, screen }) => {
  await app.open('/');

  // The agent works out which field and button to use.
  await agent.act('get the app to greet {name}', { params: { name: 'Grace' } });

  // Pair each agent step with a check that does not depend on the model.
  await expect(screen.getByRole('status')).toHaveText('hello, Grace!');
  await agent.assert('the app greets Grace by name');
});
CallWhat it doesModel
agent.actthe agent drives the app to the goalfirst run, replayed after
agent.assertjudges the current screen onceevery run
agent.waitForpolls until the claim holdsevery run
agent.extractstructured data off the screen, typed by a schemaevery run
screen.getBy*locators, like Playwright’snever
expectdeterministic assertionsnever

The model is your choice: an API key or AI gateway, a ChatGPT / GitHub Copilot / OpenCode Console / SuperGrok login (npx e2e login openai), a local OpenAI-compatible server through @ai-sdk/openai-compatible, or your Claude Code or Codex session over the Agent Client Protocol with @e2e-dev/acp. A test without agent steps needs no model at all.

e2e architecture: one runner, engines behind JSON-RPC

e2e architecture diagram. Your tests and e2e.config.ts go into the e2e npm package: cli, config, collect, scheduler, worker child processes, and the fixtures app, screen and agent. An agent step runs on an executor (the built-in AI SDK tool loop, @e2e-dev/acp for Claude Code or Codex, or @e2e-dev/decision), which calls a model. The package reaches engines only through the engine contract: JSON-RPC 2.0 with LSP framing over a loopback, spawn or socket transport. Engines: @e2e-dev/web on Playwright (Chromium, Firefox, WebKit) and @e2e-dev/mobile on agent-device (iOS simulators, Android emulators). Optional providers plug into an engine: kernel, browser-run, lightpanda, smol, testmu and e2b for browsers; testmu, browserstack, limrun, eas and e2b for devices. Side doors: the e2e mcp server for coding agents, and reporters (list, json, junit, markdown, html, github). e2e: one runner, engines behind a wire protocol the core knows the engine contract, never an engine e2e npm package: SDK, runner, CLI dashed: optional side door · no provider: local browsers, simulators or emulators your projecttests/*.e2e.ts · e2e.config.ts e2e mcpcoding agent, live session clie2e run · init · mcp confige2e.config.ts collectregister · select schedulerunits → workers workerschild processes fixturesapp · screen · agent executorAI SDK loop · @e2e-dev/acp · decision modelsAPI key · subscription · local server engine contractJSON-RPC 2.0 with LSP framingloopback · spawn · socket reporterslist · json · junitmarkdown · html · github @e2e-dev/webPlaywright: Chromium · Firefox · WebKit @e2e-dev/mobileagent-device: iOS · Android BrowserProviderkernel · browser-run · lightpandasmol · testmu · e2b DeviceProvidertestmu · browserstack · limruneas · e2b
e2e architecture: one runner talks to every engine over a JSON-RPC contract, and providers plug into an engine, not into the core.
e2e architecture diagram. Your tests and e2e.config.ts go into the e2e npm package: cli, config, collect, scheduler, worker child processes, and the fixtures app, screen and agent. An agent step runs on an executor (the built-in AI SDK tool loop, @e2e-dev/acp for Claude Code or Codex, or @e2e-dev/decision), which calls a model. The package reaches engines only through the engine contract: JSON-RPC 2.0 with LSP framing over a loopback, spawn or socket transport. Engines: @e2e-dev/web on Playwright (Chromium, Firefox, WebKit) and @e2e-dev/mobile on agent-device (iOS simulators, Android emulators). Optional providers plug into an engine: kernel, browser-run, lightpanda, smol, testmu and e2b for browsers; testmu, browserstack, limrun, eas and e2b for devices. Side doors: the e2e mcp server for coding agents, and reporters (list, json, junit, markdown, html, github). e2e architecture engines behind a wire protocol e2e npm package: SDK, runner, CLI dashed: optional side door no provider: local browsers, simulators, emulators your projecttests/*.e2e.ts · e2e.config.ts e2e mcpcoding agent,live session reporterslist · json · junitmd · html · github clie2e run · init · mcp confige2e.config.ts collectregister · select schedulerunits → workers workerschild processes fixturesapp · screen · agent executorAI SDK loop · @e2e-dev/acp · decision modelsAPI key · subscription · local server engine contractJSON-RPC 2.0 with LSP framingloopback · spawn · socket @e2e-dev/webPlaywright: Chromium,Firefox, WebKit @e2e-dev/mobileagent-device:iOS, Android BrowserProviderkernel · smolbrowser-runlightpandatestmu · e2b DeviceProvidertestmu · e2bbrowserstacklimrun · eas
e2e architecture: one runner talks to every engine over a JSON-RPC contract, and providers plug into an engine, not into the core.

How agent.act works: replay first, the model second

Flow of one agent.act() call in e2e. The cache is looked up by a key made of test, target, instruction, params and agent. Hit, on the same start route: replay finds each recorded control again by role, name, test id and context, repeats its action, checks the final route and the recorded screen change, and finishes with no model call. Miss (no entry, an unreadable or truncated entry, a retry or another route): the executor loop runs. The model observes the redacted accessibility tree, picks one tool, the dispatcher validates it against a closed schema and policy, the engine performs it, and the loop repeats until complete_step. A replay that stops after the start-route check (rows 2 to 4) hands the step to the executor mid-step. The recorder keeps every committed action; the cache entry is saved only if a later check passed. agent.act(): replay first, the model only when needed src/agent/act.ts · replay.ts · cache/recorder.ts replay no model call executor loop one model call per turn hand-off: replay stops at rows 2–4, the agent continues from that screen a later check: expect(locator) · agent.assert · agent.waitFor no check: unconfirmed, not saved · failure: evicted hit miss agent.act('add one item to the cart') cache lookupkey: test · target · instruction · params · agent 1same start route 2re-find each control 3repeat its action 4check route + screen change 5done · no model call observeredacted tree modelpicks one tool dispatcherschema · policy engineperforms it complete_stepverdict recordercommitted actions, redacted cache entrysaved only if a later check passed
One agent.act() call: a verified recording replays without the model, and anything it cannot match goes to the agent.
Flow of one agent.act() call in e2e. The cache is looked up by a key made of test, target, instruction, params and agent. Hit, on the same start route: replay finds each recorded control again by role, name, test id and context, repeats its action, checks the final route and the recorded screen change, and finishes with no model call. Miss (no entry, an unreadable or truncated entry, a retry or another route): the executor loop runs. The model observes the redacted accessibility tree, picks one tool, the dispatcher validates it against a closed schema and policy, the engine performs it, and the loop repeats until complete_step. A replay that stops after the start-route check (rows 2 to 4) hands the step to the executor mid-step. The recorder keeps every committed action; the cache entry is saved only if a later check passed. agent.act() replay first, the model only when needed replay no model call hand-off: replay stops at rows 2–4, the agent continues from that screen executor loop one model call per turn a later check: expect(locator) · agent.assert · agent.waitFor no check: unconfirmed, not saved failure: evicted hit miss agent.act('add one item…') cache lookuptest · target · instruction · params · agent 1same start route 2re-find each control 3repeat its action 4check route + screen change 5done · no model call observeredacted tree modelpicks one tool dispatcherschema · policy engineperforms it complete_stepverdict recordercommitted actions, redacted cache entrysaved only if a later check passed
One agent.act() call: a verified recording replays without the model, and anything it cannot match goes to the agent.
  1. Lookup. The key: test, target, instruction, params and agent. Also in it: the project, the engine and its SPI version, the platform, the app identity, a digest of the agent’s context, the policy version and the call’s index in the attempt. Not in it: the model, so switching models doesn’t miss. --no-cache forces a live run.
  2. Replay. Same start route required, unless the recording opens by navigating (ids, tokens and timestamps in the path or query read as placeholders); another route is a miss. Each recorded control is re-found by role, name, test id and context; the end check needs everything that appeared in the recording to be there, everything that vanished to be gone, and at least one of those changes (or the route) to have moved during this replay. No model call.
  3. Hand-off. Once past the start-route check, replay never fails a step by itself. A missing control, an ambiguous match or an unexpected alert hands the step to the executor mid-step, with a record of what already ran.
  4. Executor loop. The model gets the accessibility tree with secrets redacted (a masked screenshot only on request), calls one tool, the dispatcher validates it against a closed schema, the engine performs it, until complete_step.
  5. Save. An entry is saved at the end of the attempt only if a later check passed. An act with no passing check after it (expect, agent.assert, agent.waitFor) is unconfirmed and runs live next time.

The cache in CI

Locally the cache is read-write; with CI set it defaults to read-only. --strict-cache fails a stale recording (REPLAY_STALE) instead of handing it to the agent.

How e2e keeps secrets away from the model

How e2e keeps secrets away from the model. The value lives in your process: the config holds credentials and secrets, and tests pass frozen Secret handles that carry a name and a purpose, no value. The model is offered the type_secret tool with names only, for example admin.password, and sees any value on screen as the marker <secret:admin.password>. It calls type_secret with a target and a name. The runner checks that the name is configured, the field enabled and editable, and that a password goes into a password field; otherwise the fill is refused. Only then is the value resolved and typed by the engine. The recording stores the name, never the value. Not covered: a secret the app prints outside a masked field is in the screenshot, and video masks nothing. Secrets: the model gets a name, never the value src/secrets.ts · agent/secrets.ts · internal/redact.ts your process: test + runner the model names and markers only the value stays on this side e2e.config.tsdeclares credentials and secrets Secret handlefrozen: a name and a purpose, no value redactionvalues in screen text become markers runner checks✓ configured ✓ enabled ✓ editable✓ a password only into a password field resolve, then the engine types itread after the checks, handed over once recordingname only; a replay re-runs the checks type_secret, offered:admin.password on screen, in reports:<secret:admin.password> its call: type_secrettarget: 12name: "admin.password" refusedname not configuredfield disabled or not editablepassword, no password field Not covered: a secret the app prints outside a masked field is in the screenshot,and video masks nothing. Use test credentials, never production ones.
A secret crosses the model boundary as a name. The runner checks the field and types the value itself.
How e2e keeps secrets away from the model. The value lives in your process: the config holds credentials and secrets, and tests pass frozen Secret handles that carry a name and a purpose, no value. The model is offered the type_secret tool with names only, for example admin.password, and sees any value on screen as the marker <secret:admin.password>. It calls type_secret with a target and a name. The runner checks that the name is configured, the field enabled and editable, and that a password goes into a password field; otherwise the fill is refused. Only then is the value resolved and typed by the engine. The recording stores the name, never the value. Not covered: a secret the app prints outside a masked field is in the screenshot, and video masks nothing. Secrets the model gets a name, never the value your process the model: names only the value stays on this side e2e.config.tsdeclares credentials and secrets Secret handlefrozen: a name and a purpose, no value type_secret, offered:admin.password redactionvalues in screen text become markers on screen, in reports:<secret:admin.password> its call: type_secrettarget: 12name: "admin.password" runner checks✓ configured ✓ enabled ✓ editable✓ password → password field only refusedname not configuredfield disabled or not editablepassword, no password field resolve, then the engine types itread after the checks, handed over once recordingname only; a replay re-runs the checks Not covered: a secret the app prints outside amasked field is in the screenshot, and videomasks nothing. Use test credentials.
A secret crosses the model boundary as a name. The runner checks the field and types the value itself.

More examples: config, locators, sign-in

The config

From examples/with-vite. It starts the dev server and waits for the URL:

// e2e.config.ts
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { gateway } from 'ai';

export default {
  // The Vercel AI Gateway serves the model and reads AI_GATEWAY_API_KEY.
  // Only tests that use `agent` need it; tests/deterministic.e2e.ts runs without one.
  agents: {
    default: {
      model: gateway('openai/gpt-6-luna-fast'),
      system: 'You are a thorough QA agent. Verify every outcome.',
    },
  },
  targets: [
    {
      engine: web(),
      app: {
        url: 'http://localhost:5173',
        // The runner starts the Vite dev server and waits for the URL to answer.
        // Outside CI, a server you already started on that port is reused.
        command: { executable: 'npm', args: ['run', 'dev'], reuseExisting: true },
      },
    },
  ],
} satisfies E2EConfig;

A deterministic test, no model

// tests/deterministic.e2e.ts
import { test } from '@e2e-dev/web';
import { expect } from 'e2e';

test('greets the name you type', async ({ app, screen }) => {
  await app.open('/');

  await screen.getByLabel('name').fill('Ada');
  await screen.getByRole('button', 'greet').click();

  await expect(screen.getByRole('status')).toHaveText('hello, Ada!');
});

Sign in once, reuse the session

From apps/testbed/tests/auth.setup.e2e.ts:

import { test } from '@e2e-dev/web';
import { expect, credentials } from 'e2e';

test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, screen, session, browser }) => {
  const admin = credentials.user('admin');

  await app.open('/login');
  await screen.getByLabel('Username').fill(admin.username);
  await screen.getByLabel('Password').fill(admin.password);
  await screen.getByRole('button', 'Sign in').tap();

  await expect(browser).toHaveURL('/dashboard');
  await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');

  await session.save('admin');
});

Run it

npx e2e init                              # engine, model provider, config, example test, agent skill
npx e2e run                               # every test
npx e2e run tests/agentic.e2e.ts --headed # watch the browser
npx e2e run --no-cache                    # every agent step live, e.g. to try a new model
npx e2e run tests/agentic.e2e.ts:7 --step # one test, pause between steps

Agent steps or locators: when to use which

UseWhen
agent.actlong flows that are easy to say and churn in the UI: forms, checkout, onboarding
agent.assertmany wordings or layouts satisfy one requirement; { vision: true } for charts
locators + expectshort stable flows, exact values, every check that confirms an act
boththe default: goal in English, outcome pinned in code

A deterministic test never calls a model: no token cost, no model flake. An agentic step costs model calls on its first run and after a UI change; in between it replays.

e2e next to a Rails app

e2e is TypeScript, but the app under test can be anything with a URL. The native, Flutter and .NET MAUI examples keep the suite in an e2e/ folder next to the app, and a Rails repo can do the same.

app.command starts the server before the first test and stops it on every exit path; the executable is looked up on PATH, never run through a shell, and {port} expands to the port of url. A sketch (not from the repo), e2e/e2e.config.ts:

import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';

export default {
  targets: [
    {
      engine: web(),
      app: {
        url: 'http://localhost:3000',
        readyUrl: 'http://localhost:3000/up',
        command: {
          executable: 'bundle',
          args: ['exec', 'rails', 'server', '-p', '{port}'],
          // resolved from the directory of this config
          cwd: '..',
          env: { RAILS_ENV: 'test' },
          inheritEnv: ['DATABASE_URL', 'RAILS_*', 'BUNDLE_*', 'GEM_*'],
          // relative to this config's directory: e2e/log/e2e-server.log; a failed boot quotes its last lines
          log: 'log/e2e-server.log',
          startupTimeout: 120_000,
        },
      },
    },
  ],
} satisfies E2EConfig;

The child gets only PATH, HOME, the temp-dir variables, what inheritEnv matches and env, so name what Rails needs (inheritEnv takes exact names or PREFIX_*; add RBENV_* or ASDF_* if your Ruby comes from one). RSpec keeps the unit and request specs; cd e2e && npx e2e run covers the browser.

I take Rails work as a contractor: Ruby on Rails contractor.

FAQ

Does e2e replace Playwright?

It doesn’t have to. @e2e-dev/web drives Chromium, Firefox and WebKit through its own pinned playwright-core, and e2e can run beside @playwright/test in one project while you move tests one file at a time (migration guide).

Does every test run call a model?

No. Locators and expect never call one. agent.act calls it on the first run and after the UI changes; a verified recording replays without it. agent.assert, agent.waitFor and agent.extract call it on every run.

Does the model see my passwords?

No. It sees the name, such as admin.password, and the marker <secret:admin.password> wherever the value would show in text. The runner types the value itself. A secret the app prints outside a masked field is still in the screenshot.

Does e2e test mobile apps?

Yes. @e2e-dev/mobile runs tests on iOS simulators and Android emulators through agent-device, with devices from testmu, browserstack, limrun, eas or e2b when you add a provider. The repo has Expo, SwiftUI, Jetpack Compose, Kotlin Multiplatform, Flutter and .NET MAUI examples.

Questions? Let's Chat :)

open source