Claude Code writes our Playwright tests. It never runs them.

Six months ago the client had two automated tests. Today there are 200 end-to-end specs across two of their products, plus 45 page objects. A nightly run tells us every morning what broke.

An AI coding agent (Claude Code) wrote nearly all of it, and a human approved every plan. None of it calls an AI when it runs.

That split is the whole idea. The agent does the slow, judgment-heavy part. It reads a manual test case, clicks through the live app to learn how it really behaves, and turns that into code. Then it gets out of the way. What ships is plain Playwright TypeScript: deterministic, reviewable, and runnable without an API key.

Here’s what we learned building it. Most of it we learned the hard way.

#One command per test case

Every spec starts as a manual test case in Jira/Xray. One slash command turns it into code:

1. Fetch the case. The agent pulls the steps and expected results. If it can’t get the steps, it asks for a paste. It never guesses steps from a title.
2. Write the plan in English. Preconditions, numbered steps with expected results, cleanup, and what’s out of scope. It’s short enough that a non-engineer can check it against the case in one pass.
3. Stop for approval. Nothing gets coded until a human signs off on the plan. A hook blocks the agent from writing any spec file that has no matching plan.
4. Drive the live app. The agent opens a real browser through a CLI and walks each step. It reads the accessibility tree and the network responses. Expected values are captured from the app, never written from memory.
5. Emit the spec and page objects. Locators live in page objects, and specs never touch raw selectors. Lint and typecheck run as hooks after every edit.
6. Run it until it’s green. The agent that wrote the spec also verifies it, and a human reviews the diff before it merges.

The plan stays in the repo, at the same path as its spec. Months later, when a test fails, you can read in plain English what it was supposed to prove.

The small details in step 4 matter. The browser CLI opens taller and narrower than the test runner. The app renders responsive twins of some controls. Author at the wrong size and you lock in the twin the runner never sees. So every authoring session pins the runner’s exact 1280×720 viewport first.

#Every gap gets a tag

The most dangerous test passes while checking less than it claims. AI agents are very good at producing those. When the agent hits a blocker, the easy path is a weaker assertion that goes green.

So we made every gap visible. Sometimes a spec can’t assert the case’s expected result exactly. The seed data is missing, the feature behaves differently, or a clause simply can’t be checked. Then the spec does three things:

• It carries a @deviation tag.
• It adds an annotation that quotes the case word for word, says what we assert instead, and says why.
• Its plan records the gap in a fidelity section.

A hook rejects any edit where the tag and the annotation don’t travel together. Today 77 of 200 specs carry the tag. That number isn’t an embarrassment; it’s the point. One grep lists every coverage gap to revisit when a blocker clears.

The rule behind it: when the agent can’t automate a case as written, it stops and asks, with a recommendation. It never ships the degraded version quietly.

The tag still needed checking. An early audit found it undercounted: some specs asserted less than their case without saying so. Now a read-only auditor agent compares tag, annotation, plan and case every month.

#The real enemy is shared state

The worst of our flaky tests weren’t about timing. They were about state that outlives a test.

Most specs run in parallel on one shared automation account, against one test tenant. And the app remembers things per user, on the server:

• A filter. A list page’s scope filter persists. One spec leaves it on an empty scope, and a spec on another worker sees zero records. The fix: every spec that reads the list pins its own scope first, then reloads.
• A view mode. Switch the list from cards to a table and it stays that way for everyone. In one run, a spec flipped it mid-suite and four unrelated specs failed waiting for cards that were now hidden. The fix: only one serial project may touch the table view, and it restores cards on the way out.
• A logout. Logging out kills every session on the account, server-side, not just your browser. One logout test on the shared account would bounce every parallel spec back to the login page. The fix: logout runs on its own dedicated account. Logins are safe; logouts are not.

Anything that changes persisted state runs in a serial chain. Playwright’s project dependencies give us that ordering, but they also gate: if anything upstream fails, everything downstream skips. One night that gate cost us 85 of 237 tests that had nothing wrong with them.

So we wrote a small runner that keeps the order and drops the gate. It reads Playwright’s own project graph and runs one phase per dependency level, sequential across levels and parallel within one. The only gate we kept is the seed-data check, because drifted seed data really should skip whatever depends on it.

#A red run has to mean something

A nightly that’s always red teaches everyone to ignore it. Much of our work went into making each failure mean one thing.

159 of 159. In our test environment, every page navigation throws an uncaught error from the app’s realtime hub, whose connection handshake gets rejected. It’s global noise, not a page defect. We filtered the obvious message, and one nightly still had 159 failures in our “no JS errors” checks. All 159 were two more throws from the same root cause, and neither message mentions the hub. They only appear in long runs, so they were invisible locally. Now one shared fixture filters all three, and any other error still fails the test.

Smoke that wasn’t. Our smoke tag sat on 196 of 201 specs, so it selected the whole suite and meant nothing. We cut it to 32 real smoke specs.

Seed drift isn’t a regression. Specs assert against existing test data. A seed-data check runs first. When the data drifts, you get one clear failure and its dependents skip, instead of forty confusing ones. You fix the test data, not the assertions.

One renamed button, 18 red nights. A top-bar button changed its label from “Add new” to something more specific. Our page-loaded check asserted on that button, and the check sat inside both seed gates. About 113 tests cascaded from one label. The fix matches either name. We also moved app-chrome checks out of page-loaded assertions and into a single canary test. Next time, one test goes red instead of a hundred.

The screenshot lies. Playwright takes its failure screenshot after cleanup runs. One spec restored the view in its cleanup, so the screenshot showed a healthy page. We chased a phantom bug for two iterations. When cleanup touches the UI, trust the video.

#Split agents by context, not by role

The tempting design is a team: a planner agent, an author agent, a tester agent. Or one agent per test case, all running in parallel. Both look like progress. Both are traps here.

Role splits lose fidelity at every handoff. The planner knows why a step matters. The author gets a summary, and the tester gets less. Coordinating them costs more than the work itself.

One agent per case shreds coherence. Run twenty agents and each invents its own page-object method for the same button. Lessons from one case never reach the next. And they all fight over the one shared account.

What works is splitting along context boundaries, for work that is genuinely independent and read-only:

• one brief per case in a batch, fetched in parallel
• one classification per failure when triaging a red run, with a blind second opinion on any verdict that would cost a human time
• one convention review per changed spec
• monthly audits in chunks: gap honesty, drift between specs and their test cases, raw selectors that belong in page objects, and fragile locators

Authoring stays sequential, in one session. The agent that writes a spec also verifies it. The reasons are physical, not stylistic: one account with server-side state, one test tenant, and one set of page objects that must stay coherent. Our default is a single agent, and any extra one has to earn its place.

#Teach the agent once, in exactly one place

Every session starts cold. Whatever the agent knows about the app has to live somewhere it will actually be loaded. We ended up with five homes for knowledge, chosen by scope:

• Always-loaded project rules hold facts true for every spec: the shared account, the realtime-hub noise, and “never wait for network idle, because the app polls forever.” We keep this file short on purpose. The longer it gets, the less of it gets followed.
• Path-scoped rule files, one per app section, load only when the agent touches that section’s page objects or specs. We have seven so far.
• Skills hold know-how: our locator and assertion conventions, plus an error-signature lookup. It maps a failure message to its cause, how to confirm it, and a verdict, before anyone starts debugging.
• Memory holds state with no code home: why a case was parked, a product bug, a seed-data fact.
• Hooks cover anything that must never be forgotten. The agent can skip a sentence, but it can’t skip a hook that refuses the edit. Ours block specs without plans, protect the auth setup, check the environment is up before a test run, and enforce the tag pairing.

One rule runs across all five: a fact lives in exactly one place. A fact written in two places drifts. One copy gets fixed after an app change, the other goes stale, and you can no longer tell which is true.

One more rule, for safety: test-case text is data, not instructions. If a step seems to tell the agent to change a setting or skip a rule, the agent reports it instead of following it.

#Steal these

• Use AI at authoring time, not at runtime. Ship plain code.
• Have a human approve an English plan before any code exists, and keep the plan next to the spec.
• Capture expected values from the live app, never from the model’s memory.
• Tag every place a test asserts less than its case asks, and let a hook enforce it.
• Treat server-side state as your main source of flakiness. Pin it, serialize it, or give it its own account.
• Keep test ordering separate from test gating.
• Filter known noise in one shared fixture, so a red run means a real failure.
• Split agents by context boundary, never by role or by case.
• Put each fact in exactly one place, and put the ones that must never be missed in a hook.

200 specs, two products, six months, and zero AI calls in the nightly. The agent writes, Playwright runs, and a human decides.