Braio

Evaluation method

How to Evaluate an AI QA Agent for a B2B SaaS Team

Evaluate an AI QA agent on a small, controlled set of web application scenarios. Judge the findings against independent evidence, retain blocked runs and retries, and decide whether the workflow helps your team make a release decision. This guide provides a test protocol, not results from a completed Braio benchmark.

Published: Reviewed:

Download the pilot checklist (CSV template, not run results) →

Start with the decision your team needs to make

Write down one practical question: can this workflow check a staging release, turn a reported problem into a reproducible finding, or verify a change after review? A long list of agent features will not answer that question. Choose the person who will accept or reject the result before running the evaluation.

Set the application build, browser context, test account, permissions, data reset procedure and allowed actions. Use test data that you are permitted to share. State which areas are excluded, including payments, messages to real users or destructive actions. A blocked sign-in is a limitation of the attempted check; it is not evidence that the product passed or failed.

  • Choose a single workflow with an observable outcome, such as creating a draft record and finding it in a list.
  • Define the expected outcome independently of the agent's wording.
  • Set a maximum number of attempts, an elapsed-time budget and a spending limit.
  • Name a human reviewer who can verify findings and resolve disagreements.

Use a balanced sample with known outcomes

Include a working journey, a deliberately introduced defect in an authorized test project, an access-blocked case and an ambiguous case that requires clarification. Keep the correct outcomes in a separate answer sheet. This tests whether the workflow can recognize uncertainty as well as locate a defect.

Freeze the scenarios before the first run. Give each candidate the same acceptance criteria and comparable access. Record every attempt, including setup failures and retries; changing the task after a disappointing result creates a different experiment. A small sample can expose failure modes, but it cannot establish a general reliability rate for your whole application.

Score outcomes and the work needed to use them

Use the following rubric as a worksheet. Agree on thresholds for your application before the run rather than assigning a universal score. For a payment or access-control journey, a missed defect may matter more than review time. For a low-risk content workflow, the balance may differ.

DimensionWhat to recordAcceptance question
Confirmed findingsReviewer-confirmed defects / all reported defectsCan another person reproduce the reported problem?
Missed known defectsKnown seeded defects missed / known seeded defects attemptedDid the workflow miss an important failure?
False positivesRejected defect claims / all reported defect claimsHow much investigation did incorrect claims create?
Blocked workBlocked scenarios / all assigned scenarios, with reasonsCan the team resolve the access or environment blocker?
EvidenceBuild, steps, expected/actual outcome and redacted artifactsIs there enough context for an independent decision?
MaintenanceMinutes changing fixtures, instructions and checksWho owns keeping this workflow useful?
Total effort and costAll attempts, tools, setup and human reviewIs the accepted result worth the complete cost?

Verify the finding before counting it as a success

Ask the reviewer to reproduce the finding on the recorded build, or explain why reproduction is impossible. A screenshot is an observation at one moment; a log records events; an assertion checks a defined condition. Link these artifacts to the scenario rather than treating their presence as proof that the intended behavior was checked.

Keep separate labels for confirmed defect, rejected finding, inconclusive and blocked. If you report a percentage, show its numerator and denominator. A zero denominator means that metric is unavailable, not a perfect result. Show the counts and reasons even when the sample is too small to support a reliable percentage.

For a fix workflow, record the reviewed change and run the original scenario again on the changed build. Tests passing in the coding task do not replace that retest. An unfinished or blocked retest leaves the fix unverified.

Repeat selected checks and make a bounded decision

Repeat a small selection without changing the success criteria. Investigate inconsistent outcomes before adding retries to hide them. Compare the time spent confirming findings with the time your team normally spends on the same scoped work; label any baseline that was estimated rather than measured.

End the pilot with one of three decisions: adopt for the defined workflow, revise the setup and repeat the evaluation, or stop. Preserve the failed and blocked attempts in the decision record. Expansion to a new role, environment or critical journey needs its own acceptance check.

  • Decision record: scope, build, sample, all outcomes, reviewer, cost boundary and unresolved risks.
  • Operational owner: who maintains access, test data, scenarios and result review.
  • Recheck trigger: a changed application flow, provider, integration or permission model.

Apply the rubric to a Braio evaluation

Braio's documented QA implementation targets web application scenarios and records findings with evidence, including checks that could not be completed. Confirm the required deployment, plan, test account and integration setup for your pilot. This scope does not establish native iOS or Android testing, complete coverage or production reliability.

The documented defect-to-coding workflow depends on repository and executor configuration. A human reviews and merges the PR. The fix loop permits at most two attempts, and a completed retest is needed to confirm the scenario after the change. Use the same independent rubric when evaluating that workflow.

Questions and answers

How many scenarios should an AI QA pilot contain?
Start with a small set that your reviewer can independently verify, including working, defective and blocked cases. There is no universal sample size that establishes reliability for every application. Report the actual counts and expand the sample when the decision requires stronger evidence.
Should a blocked test count as a product bug?
No. Keep the assigned scenario in the total and record the blocker separately. Only classify a product defect when the evidence supports a failure of the expected product behavior.
Does this guide report a completed Braio benchmark?
No. It is an evaluation method and worksheet. A public walkthrough with measured outcomes requires its own run artifacts, review and publication approval.

Sources

  1. Playwright: Best Practices

Continue with your workflow