# PromptQA: test once with AI, replay without it

An AI agent walks through each test case once and turns it into a plain script: 35 of 35 recorded flows then replayed green with no AI.

- Author: Laszlo Csiki (https://www.linkedin.com/in/laszlo-csiki-7ba454108/)
- Published: 2026-11-09
- Kind: Video
- Page: https://x2labs.ai/notes/promptqa

**TL;DR**

- Testing an app by hand before every release is slow, and automated test scripts break whenever a button moves. Letting an AI click through every test every day works... but it’s too slow and too expensive.
- PromptQA lets an AI agent walk through a written test case once, like a person would, and turns what it did into an ordinary test script. That script then runs every day without AI. When the app changes and a step breaks, the AI comes back for that one step only.
- The same engine also tests tickets from our team’s board. It decides how each ticket can be tested, posts a verdict with screenshots, and asks the team when a ticket isn’t clear. People keep the final call.
- So far: 35 of 35 recorded test flows replayed green with no AI involved. A sharper visual judge cut wrong passes from 7 of 36 to 3 of 36. A backend check costs about 11 to 14 cents.

Every app has a list of things someone checks before a release... log in, change a password, make sure the little red badge shows the right number. We wanted a machine to go through that list, without paying a machine to think about it every day. So we built PromptQA, and the first numbers are in.

**Video:** A narrated explainer, animated · the screens use a sample shop app; the numbers come from real runs. <https://x2labs.ai/media/promptqa/recording.mp4>

**Transcript:** Meet PromptQA: an AI that tests apps the way a person does. Test cases are written in plain words. Turning them into test code is slow, and scripts break when a screen changes. PromptQA reads a test case and uses the app like a person: it sees the screen, taps, types and scrolls. Every step is checked by a visual judge with a confidence score. Too unsure, and a person takes a look. The recorded steps become a real test script, ready to run again and again. Daily runs replay the script without AI: fast, with no AI cost. When a screen changes and one step breaks, the AI repairs only that step, and the script is green again. Ticket mode: it picks up tickets, tests the app or backend, posts a verdict with screenshots, and asks when unsure. Our Slack assistant Xylo starts runs, posts verdicts, relays questions, and lets people confirm or overturn. From real runs: thirty-five of thirty-five flows replay with no AI, and a sharper judge cut the mistakes it let through from seven to three in thirty-six. Test once with AI. Replay without it. Repair only what breaks.

## The problem

Regression testing is checking that what worked yesterday still works today. Most teams write these checks down as test cases in a test-management tool: a title, a few steps, and an expected result for each step. Plain sentences.

That’s usually where it stops. The test cases exist as text, the test code doesn’t, because writing it takes engineering time nobody has. And when the code does exist, it breaks the moment the screen changes. A button moves, a label gets renamed, and a script that was fine on Monday fails on Tuesday.

The obvious AI answer is to let a vision model, an AI that can read screenshots, click through every test every day. It works, and it’s slow and expensive: one AI-driven check of an app takes minutes and can cost a couple of dollars. Multiply that by a few hundred test cases, every day, and the bill outgrows the problem.

## What we built

PromptQA is two tools on one engine.

The engine is an AI agent that can see a screen and act on it. It runs in a browser or on an Android emulator, which is a phone simulated on a computer. It looks at a screenshot, decides what to do, and taps, types or scrolls, like a person would. Next to it sits a judge: a second model that looks at the screenshot after each step and says if the step’s expected result is really on screen, with a confidence score from 0 to 1.

### Mode 1: a regression suite that writes itself

1. **Read.** It reads the written test cases from the test-management tool.
2. **Drive once.** The agent goes through each case on the app, step by step, and the judge checks every step.
3. **Compile.** Every tap and every typed word is recorded, and the recording becomes a real test script, the same kind an engineer would write by hand.
4. **Run without AI.** From then on the script runs every day on its own. No model is called, so it’s fast and costs nothing beyond the machine it runs on.
5. **Heal on break.** When the app changes and a step fails, the AI comes back for that step only, finds the new way through and repairs the script. After a set number of repairs it asks for a fresh recording instead of patching forever.

Results go back to the test-management tool, with screenshots. Short version: compile once, run forever, self-heal on break.

### Mode 2: a QA agent for the team’s board

The second mode works on tickets instead of test cases. It watches one column on our team’s board in an issue tracker like Jira, the one where finished work waits for testing. For each ticket it does five things.

**It decides how to test it.** A ticket about the screen gets tested on the app, in an emulator or a browser. A ticket about the server gets tested on the backend with small read-only requests, we call them probes. A ticket a machine can’t test goes to an engineer, with a one-line reason.

**It tests it.** The probes are fenced in: approved addresses only, read-only on production, a list of actions it can never call, and a request limit per step. Before testing, it checks that the fix is actually deployed.

**It reports.** It posts a structured verdict on the ticket with screenshots or the evidence it collected, and adds a label: passed, failed, needs review, needs engineering, or skipped.

**It remembers.** It keeps notes on what it learned, so the next run starts smarter.

**It asks.** When a ticket is unclear, it asks the team one question instead of guessing. If nobody answers in three days, the ticket goes to a person.

Two rules protect the team. The agent labels tickets but never moves them, so a person decides what happens next. And when the testing machine itself breaks, the ticket gets no label: a broken emulator is reported as a broken emulator, never as a failed feature.

## How we use it

The board decides when work happens. When a ticket lands in the QA column, a run starts within seconds.

Everything runs in the cloud, on demand. A backend check is a short cloud job that starts, runs and disappears. An app check needs an emulator, so a machine starts up for it and shuts itself down afterwards. Nothing sits idle waiting for work.

Spending is capped per ticket: at most two runs per pass, plus a dollar ceiling. Hitting a cap sends the ticket to a person, never to “failed”. Once a week the team gets a digest with what was tested and what it cost.

## Xylo, the voice in Slack

Xylo is our AI team assistant, and it lives in Slack. For QA it’s the dispatcher and the voice.

**It starts runs** when a ticket lands in the QA column, with a scheduled check as a safety net.

**It posts every verdict** in the team’s channel, where anyone can confirm it with ✅ or overturn it with ❌.

**It carries questions both ways.** The agent’s question goes to the team, the answer comes back, and the ticket gets tested again.

**It tracks progress,** so anyone can ask “what is QA doing right now?” and get a real answer.

**It sends the weekly digest:** runs and cost.

PromptQA does the testing. Xylo decides when it starts and tells the team what happened.

## What we measured

### The judge that looks at every step

The judge matters most, because a wrong pass lets a bug through. So we built a labelled test for it: 49 real step screenshots, where a person decided if each one should pass or fail. Each screenshot was judged three times, and every setup ran twice.

It all started with one misreading. On a sports app, a tiny “2” on a badge was read as a “3”. To save cost, we gave the judge a copy of the screenshot shrunk to 1024 pixels on the long side, and the digit ended up about 8 to 10 pixels tall. So that one was on us.

**The original judge** (a small model with the shrunk screenshot) wrongly passed 7 of 36 checks that should have failed.

**The new judge** (a stronger model with the full-resolution screenshot): 3 of 36. All three were the same outdated test screenshot, overwritten by a later recording, so its label didn’t match the image anymore. Without it, 0 of 33.

**The case that started it:** the checks that should have passed went from 1 of 6 to 6 of 6.

**Cost:** about 2 cents per judged step with the stronger model, up from about 0.6 cents.

About three times the price per step, for a judge that stopped passing things it should fail. We switched.

### Replay without AI

On a real mobile app we replayed all 35 recorded test flows with no AI involved. 35 of 35 passed, and no test account was left changed. That last part matters more than it sounds... a test that changes its own starting point makes the next run lie.

### One week on a real board

In one week on a production board, the ticket agent handled 22 tickets. 10 passed and 8 went to an engineer, each one with a reason. 2 turned into questions for the team, 1 was parked for human review and 1 was skipped. And zero tickets were marked “failed” because of an infrastructure problem.

### What a check costs

**A backend ticket** takes about 25 to 30 seconds and about 11 to 14 cents of model time.

**An app ticket** takes a few minutes on an emulator and about $1 to $2.50 in total. The testing itself is $0.30 to $1.50, and the agent’s own reasoning around it is $0.10 to $0.95.

**An empty board** costs nothing. A free check of the board runs before any model is called.

### Reading the team’s replies

When someone replies in Slack, Xylo has to know what kind of reply it is: an answer to the agent’s question, a hand-over to a colleague (“no idea, ask them”), or a question for the assistant itself. A small model decides. On a labelled set of 36 real-style replies, three runs each, it got 108 of 108 right, and no answer was lost.

### Two lessons with numbers

**A cost nobody reports is unknown, not free.** For a while the weekly digest said QA cost $0.00. Cheapest QA ever... except we simply never sent the cost. Now a missing cost shows up as “not reported”, not as zero.

**Don’t poll for work, get told about it.** At first the agent woke up every 15 minutes to ask the board if anything needed testing. Over one day that was 81 wake-ups that found nothing. Now the board itself triggers the run, and the schedule is only a safety net.

## What doesn’t work yet

A few things are missing, and some numbers are softer than they look.

**iOS.** The iPhone target is only partly built and has never run live. Today it’s Android and the web.

**“Needs review.”** The stronger judge is more careful. In the labelled test it said “needs review” on 8 or 9 of the 111 steps that should have passed, instead of deciding. That’s safer than a wrong pass... and it still costs a person a look.

**Costs are model time only.** The numbers above leave out the cloud machines, and the counter for the testing part prices every model at one flat rate, so it overstates the cheaper model.

## My take here

The part I like most is where the AI shows up. It walks through a test once, writes it down, and gets out of the way. After that a plain script does the daily work, and the AI only comes back when something changed.

The other part is that the agent knows when to stop. It hands tickets to engineers, asks questions, and never moves a ticket by itself. Eight of 22 tickets went to an engineer that week, and I count that as the agent working, not failing.
