e2e: agentic E2E tests for web and mobile
e2e is an open-source framework for agentic end-to-end testing of web and mobile apps (Apache-2.0, npm e2e). A test states the goal in plain English, an AI agent works out the clicks, and locators with expect check the result without asking a model. A verified agent step is recorded and replays with no model calls.
Everything here is read from the source at e2e@0.23.1. Docs: e2e.com/docs.
What an e2e test looks like
From examples/with-vite, an agent step with a deterministic check next to it:
// tests/agentic.e2e.ts
import { test } from '@e2e-dev/web';
import { expect } from 'e2e';
// Agent steps call the model in e2e.config.ts. Without a key these tests are skipped.
const skip = process.env.AI_GATEWAY_API_KEY ? false : 'set AI_GATEWAY_API_KEY to run agent tests';
test('the agent gets a greeting', { skip }, async ({ app, agent, screen }) => {
await app.open('/');
// The agent works out which field and button to use.
await agent.act('get the app to greet {name}', { params: { name: 'Grace' } });
// Pair each agent step with a check that does not depend on the model.
await expect(screen.getByRole('status')).toHaveText('hello, Grace!');
await agent.assert('the app greets Grace by name');
});
| Call | What it does | Model |
|---|---|---|
agent.act | the agent drives the app to the goal | first run, replayed after |
agent.assert | judges the current screen once | every run |
agent.waitFor | polls until the claim holds | every run |
agent.extract | structured data off the screen, typed by a schema | every run |
screen.getBy* | locators, like Playwright’s | never |
expect | deterministic assertions | never |
The model is your choice: an API key or AI gateway, a ChatGPT / GitHub Copilot / OpenCode Console / SuperGrok login (npx e2e login openai), a local OpenAI-compatible server through @ai-sdk/openai-compatible, or your Claude Code or Codex session over the Agent Client Protocol with @e2e-dev/acp. A test without agent steps needs no model at all.
e2e architecture: one runner, engines behind JSON-RPC
- The core knows a contract, not an engine.
packages/e2e/srcnever imports an engine package. A target names one:engine: web(). - Engines are JSON-RPC servers. The contract is LSP-style JSON-RPC 2.0 (
Content-Lengthframing,initialize…shutdown). An in-process engine still goes throughloopbackTransport, so a contract change that can’t cross the wire fails every suite.spawnTransportandsocketTransportput an engine in another process, machine or language. - Workers are child processes (
fork()), one target each, thrown away after a failing unit. - Providers plug into the engine, never the core:
BrowserProviderfor web,DeviceProviderfor mobile. testmu and e2b ship both. e2e mcpis an MCP server over stdio: Claude Code or Cursor opens a live session, taps through the real app, and each action prints the test line that repeats it.
How agent.act works: replay first, the model second
- Lookup. The key: test, target, instruction, params and agent. Also in it: the project, the engine and its SPI version, the platform, the app identity, a digest of the agent’s context, the policy version and the call’s index in the attempt. Not in it: the model, so switching models doesn’t miss.
--no-cacheforces a live run. - Replay. Same start route required, unless the recording opens by navigating (ids, tokens and timestamps in the path or query read as placeholders); another route is a miss. Each recorded control is re-found by role, name, test id and context; the end check needs everything that appeared in the recording to be there, everything that vanished to be gone, and at least one of those changes (or the route) to have moved during this replay. No model call.
- Hand-off. Once past the start-route check, replay never fails a step by itself. A missing control, an ambiguous match or an unexpected alert hands the step to the executor mid-step, with a record of what already ran.
- Executor loop. The model gets the accessibility tree with secrets redacted (a masked screenshot only on request), calls one tool, the dispatcher validates it against a closed schema, the engine performs it, until
complete_step. - Save. An entry is saved at the end of the attempt only if a later check passed. An
actwith no passing check after it (expect,agent.assert,agent.waitFor) isunconfirmedand runs live next time.
The cache in CI
Locally the cache is read-write; with CI set it defaults to read-only. --strict-cache fails a stale recording (REPLAY_STALE) instead of handing it to the agent.
How e2e keeps secrets away from the model
- A handle, not a value.
credentials.user('admin').passwordis a frozenSecretnamedadmin.password: a name and a purpose. Asecrets.get()reference read in the config throws if you turn it into a string. - Names only for the model. The
type_secrettool is offered only when the step declared secrets, and lists names and purposes. - The runner decides. An unconfigured name fails with
SECRET_UNAVAILABLE(AUTH_CREDENTIAL_UNAVAILABLEfor a password); a disabled or non-editable field, or a password aimed at a non-password field, fails withPOLICY_DENIED. - One hand-over. The value is resolved after those checks, handed to the engine once, and never logged, cached or sent to a model.
- What still leaks, per the docs: a secret the app prints outside a masked field is in the screenshot, and video masks nothing.
More examples: config, locators, sign-in
The config
From examples/with-vite. It starts the dev server and waits for the URL:
// e2e.config.ts
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { gateway } from 'ai';
export default {
// The Vercel AI Gateway serves the model and reads AI_GATEWAY_API_KEY.
// Only tests that use `agent` need it; tests/deterministic.e2e.ts runs without one.
agents: {
default: {
model: gateway('openai/gpt-6-luna-fast'),
system: 'You are a thorough QA agent. Verify every outcome.',
},
},
targets: [
{
engine: web(),
app: {
url: 'http://localhost:5173',
// The runner starts the Vite dev server and waits for the URL to answer.
// Outside CI, a server you already started on that port is reused.
command: { executable: 'npm', args: ['run', 'dev'], reuseExisting: true },
},
},
],
} satisfies E2EConfig;
A deterministic test, no model
// tests/deterministic.e2e.ts
import { test } from '@e2e-dev/web';
import { expect } from 'e2e';
test('greets the name you type', async ({ app, screen }) => {
await app.open('/');
await screen.getByLabel('name').fill('Ada');
await screen.getByRole('button', 'greet').click();
await expect(screen.getByRole('status')).toHaveText('hello, Ada!');
});
Sign in once, reuse the session
From apps/testbed/tests/auth.setup.e2e.ts:
import { test } from '@e2e-dev/web';
import { expect, credentials } from 'e2e';
test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, screen, session, browser }) => {
const admin = credentials.user('admin');
await app.open('/login');
await screen.getByLabel('Username').fill(admin.username);
await screen.getByLabel('Password').fill(admin.password);
await screen.getByRole('button', 'Sign in').tap();
await expect(browser).toHaveURL('/dashboard');
await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');
await session.save('admin');
});
Run it
npx e2e init # engine, model provider, config, example test, agent skill
npx e2e run # every test
npx e2e run tests/agentic.e2e.ts --headed # watch the browser
npx e2e run --no-cache # every agent step live, e.g. to try a new model
npx e2e run tests/agentic.e2e.ts:7 --step # one test, pause between steps
Agent steps or locators: when to use which
| Use | When |
|---|---|
agent.act | long flows that are easy to say and churn in the UI: forms, checkout, onboarding |
agent.assert | many wordings or layouts satisfy one requirement; { vision: true } for charts |
locators + expect | short stable flows, exact values, every check that confirms an act |
| both | the default: goal in English, outcome pinned in code |
A deterministic test never calls a model: no token cost, no model flake. An agentic step costs model calls on its first run and after a UI change; in between it replays.
e2e next to a Rails app
e2e is TypeScript, but the app under test can be anything with a URL. The native, Flutter and .NET MAUI examples keep the suite in an e2e/ folder next to the app, and a Rails repo can do the same.
app.command starts the server before the first test and stops it on every exit path; the executable is looked up on PATH, never run through a shell, and {port} expands to the port of url. A sketch (not from the repo), e2e/e2e.config.ts:
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
export default {
targets: [
{
engine: web(),
app: {
url: 'http://localhost:3000',
readyUrl: 'http://localhost:3000/up',
command: {
executable: 'bundle',
args: ['exec', 'rails', 'server', '-p', '{port}'],
// resolved from the directory of this config
cwd: '..',
env: { RAILS_ENV: 'test' },
inheritEnv: ['DATABASE_URL', 'RAILS_*', 'BUNDLE_*', 'GEM_*'],
// relative to this config's directory: e2e/log/e2e-server.log; a failed boot quotes its last lines
log: 'log/e2e-server.log',
startupTimeout: 120_000,
},
},
},
],
} satisfies E2EConfig;
The child gets only PATH, HOME, the temp-dir variables, what inheritEnv matches and env, so name what Rails needs (inheritEnv takes exact names or PREFIX_*; add RBENV_* or ASDF_* if your Ruby comes from one). RSpec keeps the unit and request specs; cd e2e && npx e2e run covers the browser.
I take Rails work as a contractor: Ruby on Rails contractor.
FAQ
Does e2e replace Playwright?
It doesn’t have to. @e2e-dev/web drives Chromium, Firefox and WebKit through its own pinned playwright-core, and e2e can run beside @playwright/test in one project while you move tests one file at a time (migration guide).
Does every test run call a model?
No. Locators and expect never call one. agent.act calls it on the first run and after the UI changes; a verified recording replays without it. agent.assert, agent.waitFor and agent.extract call it on every run.
Does the model see my passwords?
No. It sees the name, such as admin.password, and the marker <secret:admin.password> wherever the value would show in text. The runner types the value itself. A secret the app prints outside a masked field is still in the screenshot.
Does e2e test mobile apps?
Yes. @e2e-dev/mobile runs tests on iOS simulators and Android emulators through agent-device, with devices from testmu, browserstack, limrun, eas or e2b when you add a provider. The repo has Expo, SwiftUI, Jetpack Compose, Kotlin Multiplatform, Flutter and .NET MAUI examples.
Links
- Source: github.com/e2e-inc/e2e
- Docs: e2e.com/docs
- Examples: Vite, Next.js, Astro, Angular, Expo, SwiftUI, Jetpack Compose, Kotlin Multiplatform, Flutter, .NET MAUI in
examples/