Evaluation method
Failed Test, Product Bug or Blocked Environment? A Triage Guide for AI QA
A failed check describes an observation. It does not identify the cause. Use the environment, expected behavior and reproducible evidence to decide whether to fix the application, repair the test setup or investigate further. The examples below are illustrative triage cases, not logs from completed Braio runs.
Check the preconditions before naming the cause
Begin with the exact build and environment used by the runner. Confirm that the page is reachable, the test account is valid, its role permits the action and the expected data exists. Do not substitute an administrator account to make a restricted workflow pass without changing the scenario definition.
Compare the observation with the written acceptance criterion. If the criterion is unclear, mark the result inconclusive and ask the product owner. Preserve the original result when retrying so that an eventual pass does not erase a meaningful failure.
- Record the scenario, build, time, role and allowed action.
- Identify the earliest failed precondition.
- Inspect the relevant artifact and reproduce the observation when permitted.
- Assign an outcome and an owner for the next action.
Use outcomes with different next actions
These examples describe how to reason about a result. They are not claims about failures observed in a customer application.
| Observation | Provisional classification | Next action |
|---|---|---|
| A permitted user saves valid data; it disappears after reload on the recorded build | Suspected product defect | Reproduce, capture expected/actual behavior and review the application evidence |
| The login flow stops at a CAPTCHA or unavailable second factor | Blocked | Agree an authorized test-access method; do not infer the protected journey's outcome |
| The assertion expects obsolete wording after an approved copy change | Test specification problem | Review the acceptance criterion and update the check |
| The environment returns a gateway error before the scenario starts | Environment blocker | Restore the environment, retain the blocked attempt and repeat |
| Two reviewers disagree about what the feature should do | Inconclusive | Clarify the requirement before classifying a defect |
Keep enough context to challenge the diagnosis
Keep the last successful step, the failed step, the relevant page or request, and the precise expected outcome. Redact secrets and personal data before sharing. A second run should have its own identifier and should state what changed between attempts.
A recorded browser trace can help inspect what happened during a test. Playwright provides a trace viewer for that purpose. A trace is still evidence to interpret against the requirement; it cannot decide whether the requirement itself is correct.
Close the issue only after the relevant check completes
Assign product defects to the application owner, test problems to the scenario owner, and access or availability blockers to the environment owner. A timeout increase is justified only when the observed wait is legitimate and the acceptance condition remains intact.
After a fix, rerun the original behavior on the changed build. Keep blocked and incomplete retests open. For a Braio pilot, confirm the supported account and integration configuration: documented web QA can report blocked checks, while OAuth, CAPTCHA or a second factor may prevent a scenario from completing.
Questions and answers
- Can a retry turn a failed finding into a pass?
- A retry can provide another observation. Retain both results, identify what changed and investigate instability. An eventual pass alone does not explain the original failure.

