Braio

Evaluation method

What Counts as Evidence in AI Testing?

Evidence connects a claim about behavior to an observable event and a defined expectation. A confident summary or a folder of screenshots is insufficient by itself. This guide proposes an evidence checklist; it does not present artifacts from an executed Braio test.

Published: Reviewed:

Connect the requirement, observation and conclusion

A useful finding states what should happen, what was observed, where it happened and why the difference matters. Keep the build, environment, account role, preconditions and steps with the artifacts. Without that context, a reviewer may be unable to distinguish a product defect from a setup problem.

Keep observation and inference separate. 'The save request returned an error on build X' describes an observation. 'All customers are unable to save' extends beyond that evidence unless a broader investigation supports it.

Know the limits of each artifact

Use complementary artifacts when the decision needs them. More files do not automatically make the conclusion stronger.

ArtifactWhat it can establishWhat it does not establish alone
ScreenshotVisible state at a particular momentThe complete interaction or persisted data
Event or request logRecorded events and responsesThat the intended user requirement was satisfied
AssertionA specified condition was checkedThat the condition was meaningful or sufficiently broad
Recorded traceA sequence available for inspectionThat every important journey was attempted
RetestThe scenario's later result on an identified buildAbsence of unrelated regressions
Agent summaryThe agent's interpretationIndependent confirmation of its claims

Inspect the check as well as its green result

Read what the assertion actually tests. Does it distinguish the required outcome from a plausible wrong one? Does it check the right user, resource and build? An assertion that only confirms a button exists may leave the action's effect untested.

Playwright documents asynchronous web assertions that retry until the condition succeeds or times out and should be awaited. That behavior can help check changing UI state. It still depends on choosing the right condition; repeated checking of a weak condition does not strengthen the requirement.

Make the result independently reviewable

Give the reviewer the expected behavior, steps, artifact references and unresolved limitations. Record their classification: confirmed, rejected, inconclusive or blocked. After a change, retain a new result on the changed build and link it to the original finding.

Before sharing, remove secrets and data you do not have permission to publish. Braio's documented web QA implementation can retain findings with evidence and blocked outcomes; confirm the deployment and permissions for a pilot. A public worked example requires actual artifacts and publication approval, rather than an illustrative screenshot being labeled as a real run.

Sources

  1. Playwright: Assertions

Continue with your workflow