A Prompt Is Not a Contract: I Measured What AI Really Does With Requirements

My prompt had three explicit requirements: modern locators (getByRole, getByLabel) instead of CSS selectors, flakiness-free tests, fixtures instead of beforeEach. The agent confirmed, generated 89 TypeScript tests — and broke all three requirements. That part didn’t surprise me. What surprised me was where the code actually broke. And that two fixes, sixteen lines in total, took the pass rate from 27% to 82%.

The experiment: one command, zero corrections

The starting point: my own E2E test suite for OWASP Juice Shop, hand-written in Cypress (JavaScript) — page objects, functional tests, and security challenge tests. The goal: move all of it to Playwright with TypeScript.

The setup, so the numbers are reproducible: opencode wired up to GitHub Copilot, with Claude Opus 4.6 underneath. The generated project is Playwright 1.58 + TypeScript, the app under test is Juice Shop v13.0.0, and the experiment I ran in July 2026. The process: a short dialog, in practice a single prompt. I deliberately didn’t split the migration into stages — no “config first, then page objects, then tests file by file”. I wanted to see what I’d get with minimal effort at maximum speed. I expected the output to drift from the one prompt. That was the point.

The measurement rules were simple. I don’t fix a single line of the generated code by hand — I measure first. The suite runs against a local Juice Shop v13 on Chromium. The raw output is frozen in my public repo under the ai-raw-output tag, so you can verify every number in this article yourself.

The contract I signed with the agent

The prompt wasn’t sloppy. It wasn’t even mine, it was generated by a model, tailored strictly to the model that would perform the migration. AI wrote the contract for AI — in theory a perfect match, no human misunderstandings about format or expectations. It included a role (senior QA automation engineer), context, a step-by-step migration plan, and three hard technical requirements:

1. Locators: “Use modern Playwright locators (prefer getByRole, getByText, getByLabel over CSS/XPath selectors).”
2. Stability: “Ensure test readability and resilience (flakiness-free)” — with the rules spelled out: web-first assertions, test isolation.
3. Architecture: “Fixtures instead of BeforeEach: instead of creating page instances in beforeEach, use custom fixtures.”

The prompt even included a model Page Object class — with page.getByLabel(...) and page.getByRole('button', ...) in the constructor — as the standard the generated code should follow. It’s hard to write a clearer contract without writing the code yourself.

The agent replied with a plan that repeated these requirements. Then it generated the code.

The violation inventory

I counted; I didn’t eyeball. In the generated code (page objects + tests):

• Locators: CSS and attribute selectors dominate the entire suite — #email, [aria-label="Show/hide account menu"], [class*="success"]. User-facing locators, the ones the prompt explicitly demanded? One. A single getByLabel('Add to Basket') across all page objects and tests combined.
• Flakiness-free: 39 occurrences of page.waitForTimeout(...) — hard sleeps of 500–1000 ms, the number one anti-pattern in Playwright and something “flakiness-free” rules out by definition.
• Fixtures: zero. Not a single test.extend. Instead, test.beforeEach with page objects instantiated by hand in all seven test files — the exact pattern the prompt said to replace.

For dessert: the agent also generated its own migration documentation. A pretty one, with tables and emoji. It claims “4 suites / 34 scenarios”. The repo contains 7 spec files with 89 tests. The documentation drifted away from the code before anyone had a chance to read it.

If I had stopped the experiment here, the takeaway would be trivial: “AI ignores prompts, we knew that.” But code review is one, let we try run the code!

Fine, but does it work?

I ran the suite against a local Juice Shop v13. The raw output’s score:

24 of 89 tests pass (~27%). Login: 0/11. Registration: 0/15. The best file was homepage (10/16) — not because it’s the best written, but because it’s the most defensively written: a good share of its assertions sit inside if (await element.isVisible()), so when the element isn’t there, the test “passes” having checked nothing.

Zero on both login and registration suggests a shared cause. And this is where it gets interesting, because the cause is none of the prompt violations from the previous section.

On first visit, Juice Shop shows a welcome dialog and a cookie banner. The generated dismissAllPopups() helper was supposed to close them. The problem: neither of its locators matched anything in the application’s DOM. The agent looked for a [mat-dialog-close] button — the real button is button[aria-label="Close Welcome Banner"]. It looked for a [mat-button] with the text “Accept” — the real element is an a[aria-label="dismiss cookie message"] link with the text “Me want it!”. On top of that, the helper checked isVisible() immediately after navigation, before the asynchronous dialog had time to render.

In code, it looked like this:

// Raw AI output: this attribute doesn't exist in Juice Shop's DOM
this.welcomeDismissButton = page.locator('[mat-dialog-close]');

// ...and every call in the tests wrapped in a silencer:
try {
  await firstVisitPopups.dismissAllPopups();
} catch (e) {
  // Popups might not be present
}

The result: dismissing the popups never worked. Not once. And because every call was wrapped in a try/catch with an empty catch block, the logs never told me. The overlay backdrop stayed in the DOM and intercepted every subsequent click. In a single run of the suite, the same error — <div class="cdk-overlay-backdrop"> intercepts pointer events — occurred 442 times. Tests failed far from the cause, with timeout messages pointing at buttons that were visible, clickable, and completely innocent.

One helper. Two invented selectors. One empty catch. Two thirds of all failures in the suite.

Sixteen lines later

Since the diagnosis pointed at localized defects, I ran a controlled test: fix only the diagnosed spot, don’t touch any test logic, measure again.

Fix #1: 15 lines in one file. I corrected both locators in FirstVisitPopups to ones that exist in the DOM, and added waiting for the dialog to appear and for the backdrop to detach:

this.welcomeDismissButton = page.locator('button[aria-label="Close Welcome Banner"]');
// the dialog renders asynchronously — wait for it instead of checking instantly
await this.welcomeDismissButton.waitFor({ state: 'visible', timeout: 3000 });
await this.clickWelcomeDismissButton();
await this.page.locator('.cdk-overlay-backdrop').waitFor({ state: 'detached' });

The result: from 24 to 57 passing tests. The 2-star challenge file jumped from 3/11 to 11/11.

Fix #2: one line. Registration was still stuck at 0/15, but for a different reason. Every registration test navigates to the login page and clicks the link to the registration form — and the generated page object looked for #alreadyACustomerLink there. That id exists only on the registration page (it leads back to login). The login page has #newCustomerLink. The agent wired the navigation backwards — and, best of all, it had already written the correct selector itself, a few dozen lines earlier, in LoginPage. One changed line: registration went from 0/15 to 14/15.

The full balance sheet:

Sixteen lines across two page objects. Pass rate times three, runtime cut in half — and the time drop is no “speedup”, the suite simply stopped burning 10–30 seconds of timeout on every dead click. This is the most important observation of the whole experiment: the generated code was not uniformly bad. It was mostly correct, with a few sharp defects with an enormous blast radius — hidden exactly where no log would show them.

The remaining 16 failures are the same defect families at a smaller scale: a call to a method the agent never defined (typeInSearchInput — the code calls something that doesn’t exist), a wrong assumption about the homepage URL, fragile timing in the basket flows. No new class of problem — smaller shrapnel from the same shells.

Why the prompt lost to the source code

I had a hypothesis: the agent translated my Cypress suite mechanically, one to one, and that’s where the CSS selectors came from instead of getByRole. So I compared both repositories file by file. The truth turned out to be more interesting — and less flattering for the agent.

What should have been rewritten, it copied. Cypress’s [id="email"] became Playwright’s #email. The entire selector layer came through untouched, against the prompt’s requirement. Here the hypothesis holds: existing code is a stronger instruction for the model than any “prefer getByRole” in the prompt.

What should have been copied, it rewrote — and broke. This is the heart of it. My Cypress suite had working popup selectors: [aria-label="Close Welcome Banner"], .close-dialog, .cc-btn. The agent didn’t translate them. It replaced them with invented [mat-dialog-close] and [mat-button] with “Accept” — selectors that look like Angular Material but don’t exist in this application. The most expensive bug in the suite was not inherited from the source. The agent degraded working code, showing initiative exactly where it shouldn’t have.

What it copied correctly, it used in the wrong direction. This is the bug that fix #2 repaired. #alreadyACustomerLink in my Cypress was semantically fine — a link to the login page, used on the registration page. The agent carried the locator over faithfully, but composed the test flow backwards, so it clicked an element that was never on that page.

And to complete the set, a third defect class, purely Playwright-specific: the one registration test that still fails after the fixes fails on correct application behaviour. Selecting the security question works, but the assertion queries [class*="mat-select-value"], which matches two elements — and Playwright’s strict mode rightly rejects that. Healthy app, sick test.

The pattern is consistent: the agent doesn’t break the prompt randomly. It copies what it sees (even when the prompt says rewrite), improvises where it can’t see something (even when it could copy), and verifies none of its decisions against the running application.

What to do with this on Monday

If you’re planning an AI test migration — or you’ve already received a pull request with one from an agent — here are five things that would have paid for themselves immediately in this experiment:

1. Verify prompt requirements with grep, not with trust. Thirty seconds and you know whether the contract was honoured:

   grep -rEo "getByRole|getByLabel|getByText" src tests | wc -l   # mine: 1
   grep -rn "waitForTimeout" src tests | wc -l                    # mine: 39
   grep -rn "test\.extend" src tests | wc -l                      # mine: 0
   

2. Hunt for empty catches before hunting for bugs. A try/catch that swallows the exception is a machine for turning one defect into sixty failures with misleading messages. It’s the same thinking as in a security review: a mechanism that suppresses the error signal is more dangerous than the error itself — and in a suite that tests Juice Shop’s security challenges, an empty catch can mask a security regression just as effectively as it masked dead selectors. Treat every empty catch in generated code as suspect number one.
3. Run it before you judge it. A code review of the raw output would have told me about CSS selectors and sleeps — and nothing about the two bugs that took down two thirds of the suite. Static assessment of AI code measures style. Running it measures truth.
4. Start the review with page objects, not tests. The blast radius lives in the shared layer. One bad helper is sixty red tests; one bad test is one red test. The review order should reflect the blast radius.
5. Grep for conditional assertions too. if (await element.isVisible()) { expect(...) } is a test that passes when the element is missing — a false pass, more dangerous than a failure because no report will ever show it. In this experiment, the “best” file in the suite owed its score to exactly this pattern.

And the overriding rule this experiment taught me: hold the agent accountable like a vendor, not like a teammate. A teammate who says “done as agreed” usually did it. An agent that says so generated documentation claiming it did.

Was it worth it?

Yes — and this is the second, less obvious punchline. Within minutes I had a working repo to build on: project structure, types, configuration, 89 translated tests. The whole migration, including my debugging and the sixteen lines of fixes, cost a fraction of rewriting by hand — in that time I wouldn’t have finished the first two files myself.

But the word “migration” suggests the work is moving the code over. It isn’t. In an AI migration, the work is the audit: holding the output against the contract, finding three sharp defects among a thousand correct lines, and distrusting every empty catch. It’s different work than writing tests. But it is still work — and it still needs someone who knows what the application’s DOM looks like 442 errors before the logs do.

What would I do differently? I’d split the migration into stages and run it systematically: a well-described plan, steps — config, page objects, then tests file by file — and output verification after every stage, instead of one prompt for everything. This experiment deliberately measured the “quick and dirty” variant.

The whole experiment is public: the repo, the raw AI output frozen under the ai-raw-output tag, the prompt in the README, every number verifiable with a single npx playwright test. If you measure your own AI migration — I’d love to compare notes.

This article was written with AI assistance. The experiment, measurements, and conclusions are my own; an AI model helped with drafting and editing the text.