Braio

Evaluation method

When AI Tests Pass for the Wrong Reason

A passing test establishes only the condition it actually checked. Inspect whether that condition would fail when the required behavior is broken. The examples here are illustrative review exercises, not a record of executed tests or measured Braio performance.

Published: Reviewed:

A visible control may not prove the action worked

Imagine a test that clicks Save and only checks that the Save button remains visible. It can pass while no data is stored. A more discriminating check might verify the saved record after a reload using the intended user and data. The exact check depends on the requirement; adding assertions without a behavioral reason is insufficient.

Playwright's asynchronous web assertions retry conditions and should be awaited. Use that behavior for a meaningful expected state. Replacing an awaited assertion with a truthiness check on an object can leave the intended state untested.

Review the ways a test can avoid the real requirement

Use this table to design review checks. It suggests controlled counterexamples; no test execution or detection rate is claimed.

Failure modeIllustrative passing conditionControl to inspect or run
Weak assertionThe page or button existsBreak the required effect in a test project and verify the check detects it
Excessive mockingA mocked save always succeedsAdd an appropriate integration check against the persistence boundary
Wrong environmentThe test checks an old or unrelated deploymentRecord and verify the target build and environment
Wrong roleAn administrator can perform the actionCheck the required role and an unauthorized counterpart
Skipped scenarioThe suite is green but the relevant case did not executeInspect executed cases, skips and early returns
Changed requirementThe assertion was relaxed to match a defectCompare the new check with the agreed acceptance criterion

Ask what would make the test fail

For important behavior, construct a permitted negative case in a test project: omit a required write, use a wrong account role or return invalid data. Verify whether the test rejects it. Keep this separate from any production environment, and record which counterexamples were actually attempted.

Do not claim that one negative case proves complete coverage. It tests a specific weakness. Combine the check with code review, requirements and relevant integration evidence. If a counterexample cannot be executed, label that limitation rather than reporting it as passed.

Keep the requirement and result together

A useful review record includes the requirement, changed test, target build, executed cases, artifacts and unresolved gaps. Review changes to test expectations with the same care as application changes. A coding agent's own green summary remains an interpretation until the evidence is inspected.

In a Braio evaluation, use an independently agreed sample and reviewer. Documented coding and review workflows need repository access, configuration, provider and budget. This article supplies a procedure for judging tests; a product-specific worked example requires its own completed run and publication rights.

Sources

  1. Playwright: Assertions

Continue with your workflow