Evaluation method
How to Evaluate an AI QA Agent for a B2B SaaS Team
Evaluate an AI QA agent on a small, controlled set of web application scenarios. Judge the findings against independent evidence, retain blocked runs and retries, and decide whether the workflow helps your team make a release decision. This guide provides a test protocol, not results from a completed Braio benchmark.
Download the pilot checklist (CSV template, not run results) →
Start with the decision your team needs to make
Write down one practical question: can this workflow check a staging release, turn a reported problem into a reproducible finding, or verify a change after review? A long list of agent features will not answer that question. Choose the person who will accept or reject the result before running the evaluation.
Set the application build, browser context, test account, permissions, data reset procedure and allowed actions. Use test data that you are permitted to share. State which areas are excluded, including payments, messages to real users or destructive actions. A blocked sign-in is a limitation of the attempted check; it is not evidence that the product passed or failed.
- Choose a single workflow with an observable outcome, such as creating a draft record and finding it in a list.
- Define the expected outcome independently of the agent's wording.
- Set a maximum number of attempts, an elapsed-time budget and a spending limit.
- Name a human reviewer who can verify findings and resolve disagreements.
Use a balanced sample with known outcomes
Include a working journey, a deliberately introduced defect in an authorized test project, an access-blocked case and an ambiguous case that requires clarification. Keep the correct outcomes in a separate answer sheet. This tests whether the workflow can recognize uncertainty as well as locate a defect.
Freeze the scenarios before the first run. Give each candidate the same acceptance criteria and comparable access. Record every attempt, including setup failures and retries; changing the task after a disappointing result creates a different experiment. A small sample can expose failure modes, but it cannot establish a general reliability rate for your whole application.
Score outcomes and the work needed to use them
Use the following rubric as a worksheet. Agree on thresholds for your application before the run rather than assigning a universal score. For a payment or access-control journey, a missed defect may matter more than review time. For a low-risk content workflow, the balance may differ.
| Dimension | What to record | Acceptance question |
|---|---|---|
| Confirmed findings | Reviewer-confirmed defects / all reported defects | Can another person reproduce the reported problem? |
| Missed known defects | Known seeded defects missed / known seeded defects attempted | Did the workflow miss an important failure? |
| False positives | Rejected defect claims / all reported defect claims | How much investigation did incorrect claims create? |
| Blocked work | Blocked scenarios / all assigned scenarios, with reasons | Can the team resolve the access or environment blocker? |
| Evidence | Build, steps, expected/actual outcome and redacted artifacts | Is there enough context for an independent decision? |
| Maintenance | Minutes changing fixtures, instructions and checks | Who owns keeping this workflow useful? |
| Total effort and cost | All attempts, tools, setup and human review | Is the accepted result worth the complete cost? |
Verify the finding before counting it as a success
Ask the reviewer to reproduce the finding on the recorded build, or explain why reproduction is impossible. A screenshot is an observation at one moment; a log records events; an assertion checks a defined condition. Link these artifacts to the scenario rather than treating their presence as proof that the intended behavior was checked.
Keep separate labels for confirmed defect, rejected finding, inconclusive and blocked. If you report a percentage, show its numerator and denominator. A zero denominator means that metric is unavailable, not a perfect result. Show the counts and reasons even when the sample is too small to support a reliable percentage.
For a fix workflow, record the reviewed change and run the original scenario again on the changed build. Tests passing in the coding task do not replace that retest. An unfinished or blocked retest leaves the fix unverified.
Repeat selected checks and make a bounded decision
Repeat a small selection without changing the success criteria. Investigate inconsistent outcomes before adding retries to hide them. Compare the time spent confirming findings with the time your team normally spends on the same scoped work; label any baseline that was estimated rather than measured.
End the pilot with one of three decisions: adopt for the defined workflow, revise the setup and repeat the evaluation, or stop. Preserve the failed and blocked attempts in the decision record. Expansion to a new role, environment or critical journey needs its own acceptance check.
- Decision record: scope, build, sample, all outcomes, reviewer, cost boundary and unresolved risks.
- Operational owner: who maintains access, test data, scenarios and result review.
- Recheck trigger: a changed application flow, provider, integration or permission model.
Apply the rubric to a Braio evaluation
Braio's documented QA implementation targets web application scenarios and records findings with evidence, including checks that could not be completed. Confirm the required deployment, plan, test account and integration setup for your pilot. This scope does not establish native iOS or Android testing, complete coverage or production reliability.
The documented defect-to-coding workflow depends on repository and executor configuration. A human reviews and merges the PR. The fix loop permits at most two attempts, and a completed retest is needed to confirm the scenario after the change. Use the same independent rubric when evaluating that workflow.
Questions and answers
- How many scenarios should an AI QA pilot contain?
- Start with a small set that your reviewer can independently verify, including working, defective and blocked cases. There is no universal sample size that establishes reliability for every application. Report the actual counts and expand the sample when the decision requires stronger evidence.
- Should a blocked test count as a product bug?
- No. Keep the assigned scenario in the total and record the blocker separately. Only classify a product defect when the evidence supports a failure of the expected product behavior.
- Does this guide report a completed Braio benchmark?
- No. It is an evaluation method and worksheet. A public walkthrough with measured outcomes requires its own run artifacts, review and publication approval.
Sources
Continue with your workflow
- Failed Test, Product Bug or Blocked Environment? A Triage Guide for AI QA
- A Seven-Day Pilot Plan for an AI Coworker in a Software Team
- AI Testing Agents, Playwright and Managed QA: Who Owns Maintenance?
- What Counts as Evidence in AI Testing?
- AI web application testing - from scenario to retest | Braio
- Braio help - first workflow, QA and SEO/GEO reports

