Evaluation method
Independent Review for AI-Generated Code: An Evaluation Checklist
Evaluate a review by whether its findings help a person make a correct decision about a specific change. This protocol describes how to assemble and judge a review sample. It does not claim a completed Braio review benchmark or publish fabricated PR results.
Review exercise: a correct label with the wrong destination
Use the public seeded invoice fixture as a small review exercise. In before.html, a link labelled Open INV-102 points to the INV-101 detail. The changed fixture points to INV-102. Actual local browser observations record a failed original scenario and a successful retest of the changed link. This is a controlled repository/browser fixture, not a Braio agent benchmark, customer outcome or independent human-review claim.
A label-only assertion would miss the wrong destination. A useful review asks which business record the action opens and which visible facts distinguish it. Here the heading and amount distinguish the records. Check the source change against both facts, then repeat the same interaction.
The exercise covers one static link. It does not establish server-side authorization, ownership boundaries or the correctness of an invoice API. Those need their own acceptance tests. Preserve this narrow boundary when deciding whether a source diff is sufficient for your real application.
Prepare a reviewable sample and an independent answer sheet
Choose changes you are allowed to inspect and publish. Include a correct change, a known behavioral regression, a permission-sensitive case and a change with incomplete requirements. Record the base and head revisions, affected files, relevant requirements and known issues before asking for review.
Have a qualified person assess the sample independently. If the author or the same agent provides the only judgment of its own work, record that limitation. Resolve disagreements using the requirements, code and behavior, rather than counting agreement with a model as ground truth.
Require findings that can be checked
For each finding, ask whether it identifies a concrete failure and gives enough context to verify it. Style suggestions can be useful, but keep them separate from defect counts.
- Does it refer to the current revision and relevant code?
- Does it describe a trigger, affected behavior and practical consequence?
- Can the reviewer reproduce or reason through the failure from available evidence?
- Does the proposed fix preserve the intended requirement and permission boundary?
- Does it distinguish a demonstrated defect from a question or an assumption?
Count useful findings and missed problems separately
Record confirmed defects out of all defect claims, rejected claims out of all defect claims, and missed known defects out of the known defects in scope. Keep the counts visible. An empty set of reported findings does not establish perfect precision, and zero known seeded defects cannot establish recall.
Also record human investigation time, files or paths not examined, unavailable checks and the cost of all attempts. Review coverage is an explicit scope statement. It is not a guarantee that the unflagged code has no problems.
Review the resulting change, then verify behavior
Keep the review decision attached to the revision it examined. GitHub reviews distinguish comments, approvals and requests for changes; repository administrators can configure required approvals. Select the policy appropriate to your team and verify its actual configuration.
After a finding is fixed, inspect the updated diff and run the relevant checks. A person remains responsible for the merge decision in Braio's documented coding workflow. Confirm repository access, provider availability, budget and deployment for a pilot. A public worked example requires a separate, redacted artifact set and independent assessment.
Sources
Continue with your workflow
- How to Evaluate an AI QA Agent for a B2B SaaS Team
- Human Approval for AI Agents: What to Approve Before a Tool Changes Data
- How to Write a Bug Brief an AI Coding Agent Can Act On
- When AI Tests Pass for the Wrong Reason
- AI web application testing - from scenario to retest | Braio
- AI agent security - access, approvals and costs | Braio

