CLI reference
kusaw scorers
List scorers and their calibration
kusaw scorers
List scorers and their calibration
Usage
kusaw scorers [flags]Flags
| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for scorers |
Subcommands
kusaw scorers agreement
Print agreement, kappa, rates, and disagreements
kusaw scorers agreement <scorer> [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for agreement |
Example:
kusaw scorers agreement redundant-testskusaw scorers calibrate
Re-run a scorer on labelled runs and report self-consistency
kusaw scorers calibrate <scorer> [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--budget | float64 | 2 | Dollar budget for this calibrate pass | |
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for calibrate |
--repeat | int | 1 | How many times to re-score each labelled run |
Example:
kusaw scorers calibrate redundant-tests --repeat 3kusaw scorers label
Store a hand label for a scorer and run
kusaw scorers label <scorer> [<run-id> pass|fail|na] [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for label |
--next | bool | false | Show the next run to label; hides the scorer's answer | |
--note | string | Optional note stored with the label | ||
--score-missing | bool | false | Score eligible runs that have no result before picking --next |
Example:
kusaw scorers label redundant-tests run-1 passkusaw scorers labels
Export or import scorer labels
kusaw scorers labels [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for labels |
Example:
kusaw scorers labels export labels.jsonlkusaw scorers labels export
Write current labels to a JSONL file
kusaw scorers labels export <file.jsonl> [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for export |
Example:
kusaw scorers labels export labels.jsonlkusaw scorers labels import
Read labels from a JSONL file; runs must exist
kusaw scorers labels import <file.jsonl> [flags]| Name | Short | Type | Default | Description |
|---|---|---|---|---|
--config | string | Path to kusaw.yaml (overrides KUSAW_CONFIG and parent search) | ||
--factory | string | factory name from kusaw list | ||
--help | -h | bool | false | help for import |
Example:
kusaw scorers labels import labels.jsonlExit codes
| Code | Name | Description |
|---|---|---|
| 0 | ok | Success. |
| 1 | error | Runtime error. |
| 2 | usage | Usage or validation error. |
| 3 | rejected | Control request rejected; the error string is printed verbatim. |
| 4 | not picked up | Control request not picked up within the wait time while serve runs. |
Example
kusaw scorers