Braio

Evaluation method

Failed Test, Product Bug or Blocked Environment? A Triage Guide for AI QA

A failed check describes an observation. It does not identify the cause. Use the environment, expected behavior and reproducible evidence to decide whether to fix the application, repair the test setup or investigate further. The examples below are illustrative triage cases, not logs from completed Braio runs.

Published: Reviewed:

Check the preconditions before naming the cause

Begin with the exact build and environment used by the runner. Confirm that the page is reachable, the test account is valid, its role permits the action and the expected data exists. Do not substitute an administrator account to make a restricted workflow pass without changing the scenario definition.

Compare the observation with the written acceptance criterion. If the criterion is unclear, mark the result inconclusive and ask the product owner. Preserve the original result when retrying so that an eventual pass does not erase a meaningful failure.

  1. Record the scenario, build, time, role and allowed action.
  2. Identify the earliest failed precondition.
  3. Inspect the relevant artifact and reproduce the observation when permitted.
  4. Assign an outcome and an owner for the next action.

Use outcomes with different next actions

These examples describe how to reason about a result. They are not claims about failures observed in a customer application.

ObservationProvisional classificationNext action
A permitted user saves valid data; it disappears after reload on the recorded buildSuspected product defectReproduce, capture expected/actual behavior and review the application evidence
The login flow stops at a CAPTCHA or unavailable second factorBlockedAgree an authorized test-access method; do not infer the protected journey's outcome
The assertion expects obsolete wording after an approved copy changeTest specification problemReview the acceptance criterion and update the check
The environment returns a gateway error before the scenario startsEnvironment blockerRestore the environment, retain the blocked attempt and repeat
Two reviewers disagree about what the feature should doInconclusiveClarify the requirement before classifying a defect

Keep enough context to challenge the diagnosis

Keep the last successful step, the failed step, the relevant page or request, and the precise expected outcome. Redact secrets and personal data before sharing. A second run should have its own identifier and should state what changed between attempts.

A recorded browser trace can help inspect what happened during a test. Playwright provides a trace viewer for that purpose. A trace is still evidence to interpret against the requirement; it cannot decide whether the requirement itself is correct.

Close the issue only after the relevant check completes

Assign product defects to the application owner, test problems to the scenario owner, and access or availability blockers to the environment owner. A timeout increase is justified only when the observed wait is legitimate and the acceptance condition remains intact.

After a fix, rerun the original behavior on the changed build. Keep blocked and incomplete retests open. For a Braio pilot, confirm the supported account and integration configuration: documented web QA can report blocked checks, while OAuth, CAPTCHA or a second factor may prevent a scenario from completing.

Questions and answers

Can a retry turn a failed finding into a pass?
A retry can provide another observation. Retain both results, identify what changed and investigate instability. An eventual pass alone does not explain the original failure.

Sources

  1. Playwright: Trace viewer

Continue with your workflow