Benchmarks

Benchmarks replay a captured task against baseline and variant overlays. A replay is correct when the clean-checkout regrade is pass (grade_clean). The in-place grade (grade_in_place) is recorded next to it. kusaw bench run queues those replays (default --repeat 3). kusaw bench report --tasks a,b --variants baseline,cheap reads reliability from the latest batches. kusaw bench report <batch> --markdown writes the comparison table for you to keep.

The factory does not switch a default because a variant won.

Reading reliabilityCopy Anchor Link

Agent runs vary from try to try. One try per task and variant measures noise, not capability.

• pass@k is the chance that at least one of k tries is correct. The report uses the unbiased estimator 1 − C(n − c, k) / C(n, k). pass@1 is c / n.
• pass^k is the chance that all k tries are correct — the number that matters when every unattended run must succeed. It is C(c, k) / C(n, k). pass^n is 1 only when every try passed.

n is finished replays that were not stopped or cancelled. c is how many of those were correct. Values are shown with their counts, never alone: pass^3 0.42 (n=5, c=4).

Use 3 to 5 tries per task and variant. Three is the default for kusaw bench run. Use 5 when you will act on the comparison. A single try is noise.

Change one axis at a time (model, harness, or effort) so the report can say which change made the difference. The report warns when a variant has fewer than 3 tries (W1), when two variants differ in more than one axis (W2), or when the report has fewer than 5 tasks (W3).

Designing an experimentCopy Anchor Link

A measured experiment records what changed and how often it worked. Prefer tasks written for the experiment over replays of public history.

1. Write tasks fresh for the experiment, in a private repo or a throwaway branch you never push, and keep them private. Replays of public commits measure memory as well as skill.
2. Check the task's tests first (GF-123).
3. Run 3 to 5 tries per task and variant, and read pass^k with its counts (GF-121).
4. Change one axis at a time: model, harness or effort (GF-121 W2).
5. Grade on a clean checkout (GF-120).
6. Read the transcripts of a sample of tries (runs/<run-id>/ logs), not only the numbers; don't time people against agents.
7. Record the experiment: tasks and their origin, variants, kusaw version, the report (kusaw bench report --json).

Each task has an origin you set: fresh (written for this experiment, never merged anywhere public), private (from a private repo's history), public (from history that is public), or unknown (not said). Set it on capture with kusaw bench add <run> --origin fresh|private|public|unknown (default unknown), or later with kusaw bench set-origin <name> <origin>. The factory does not guess origin from remotes or hosts.

kusaw bench show, kusaw bench report, and the API show origin per task. A report that includes public tasks prints <n> of <m> tasks come from public history; models may have seen them, so treat their results as an upper bound. JSON reports include origins: {fresh, private, public, unknown} counts.

Example: two fresh demo tasksCopy Anchor Link

Two tasks written for the experiment, two variants, five tries each:

kusaw bench add <run-a> --name demo-a --origin fresh
kusaw bench add <run-b> --name demo-b --origin fresh
kusaw bench run demo-a --variants cheap --repeat 5
kusaw bench run demo-b --variants cheap --repeat 5
kusaw bench report --tasks demo-a,demo-b --variants baseline,cheap --json

cheap is a variant overlay in benchmarks/variants/cheap.yaml that changes one axis.

Review-step benchmarksCopy Anchor Link

A review task measures whether a review step finds planted bugs in a change. The factory never puts the planted-bug list in the request or the worktree.

kusaw bench add-review demo --repo demo --base HEAD --patch change.patch --bugs bugs.yaml --origin fresh
kusaw bench run-review demo --step review-standards --repeat 3 --budget 5.00
kusaw bench show demo
kusaw bench report --tasks demo
kusaw bench label-finding <batch> 1 1 real

add-review checks that the patch applies to the base (E1) and that each bug's lines sit on lines the patch changes or adds (E2), then stores kind = review. run-review replays only that step on a detached worktree of the base with the patch applied: no branch, no delivery, priority = -1, budget cap, worktree removed afterwards.

Findings come from built-in review (artifacts/review-<focus>.json) or a custom step's outputs.findings (review-findings.schema.json). No valid findings is E3. A step that is not a review step and has no findings schema is E4. severity: info is a comment unless --count-info.

The default matcher is deterministic: same file, and the finding's line within match.line_tolerance (default 3) of the bug. Each bug and each finding is matched at most once, closest line first. A finding that matches known_ok is neither a hit nor a false alarm. There is no built-in model judge. label-finding … real adds that finding to known_ok for later batches.

--match or match.command in bugs.yaml replaces the location matcher. The factory sends a protocol v2 mode: "match" request with the bugs and one try's findings; the command returns outputs.pairs (review-match.schema.json). It runs with the GF-28 environment and its declared secrets. If the command calls a model, that is the user's command; the factory does not.

Per try the report shows TP, FP, FN, precision, recall, F1, and findings per try. Per variant it shows mean/min/max of each plus per-bug find rate, and mean pairwise Jaccard of bugs found and of findings (bug agreement and finding agreement). Existing benchmark API endpoints return those metrics for kind = review tasks. Event bench.review_try carries {batch, variant, repeat_no, tp, fp, fn}.

A shipped example lives at examples/bench/review/demo/: a small Go cache written for this task, a change with six planted bugs across four categories (off-by-one, ignored error, missing check, wrong lock order), and bugs.yaml. Origin is fresh.

Clean regradeCopy Anchor Link

After a replay records its per-repo diff, the factory applies that diff to a clean checkout of each repo's base_ref and re-runs Verification plus held-out checks. Correctness is grade_clean == pass. kusaw bench show and the benchmark API list both grade_in_place and grade_clean. When they differ, the note is in place: pass, clean: fail.

Changed checksum/lock files in the agent's diff are notes, never failures: go.sum, package-lock.json, pnpm-lock.yaml, yarn.lock, Cargo.lock, poetry.lock, uv.lock.

regrade.delivery is off (default), record, or block. block stops delivery with needs_attention / regrade_failed when the clean grade is fail or error. kusaw regrade <run-id> records a manual regrade without changing the run's result.