Skip to main content
Every candidate a build produces is measured, and the measurement is what the judge scores and the campaign optimises. The evaluator is therefore the one thing a campaign must never be allowed to improve. This page is the contract an evaluator meets and the three states one can be in.

​
The contract

An evaluator is a command the campaign runs from the workspace root. A good one:
  • runs in one command: python kapso_evaluation/evaluate.py, with the candidate’s own entry point called inside it;
  • prints the metric as its last line, name: value, such as auc: 0.8412, and writes kapso_evaluation/result.json with evaluation_script_path, evaluation_output and score;
  • exits non-zero with a printed reason on a malformed submission, so a broken candidate is a clear failure, not a low score;
  • is deterministic given the candidate, so repeats agree and the noise you care about is the data’s, which you can state (a bootstrap standard error beside the baseline);
  • has a --fast mode for smoke checks, marked not comparable with a full score;
  • finishes well under a minute per cycle where the task allows, so a build’s budget goes to experiments, not to measuring them.
The churn example ships one that meets every line.

​
Provided: yours, protected

Pass the directory with --evaluation (Python: evaluation=). It is copied to kapso_evaluation/ in the workspace, committed as the baseline and fingerprinted, file by file. A candidate that modifies, deletes, replaces or extends it is rejected: its score is cleared, the judge never sees it, and it is never chosen as a parent or delivered. The count of such candidates is solution.rejected_candidates. The rule, file by file: every file of the suite is protected by path and content; a new file under it is rejected when it is source by suffix (.py, .sh, .js, .yaml and the like) or executable, and allowed otherwise, so runtime artifacts (result.json, logs, predictions, a profile) pass. An evaluator the campaign writes itself is the candidate’s to edit; one the maintainer registered is the maintainer’s, under the registry’s hash. A result.json an earlier run left in your evaluation directory is refused at launch, since it would be protected with the suite. The mechanics are on Evaluation integrity. Data goes the same way: --data (Python: data=) is copied to kapso_datasets/. Keep hold-out labels out of both directories; the evaluator reaches them where you put them, and the rules in the brief say no candidate may.

​
Proposed: built from the brief

With no --evaluation, the campaign’s first node builds the evaluator in kapso_evaluation/ from the brief’s Measured by line: the data, the split, the seed, the metric and how it is printed. The judge then reads every candidate’s evaluation against the brief, and a candidate that weakens the protocol is scored as what it is. Write the protocol precisely; the fraud example shows one. What the shipped config does not do yet is register that evaluator and protect it file by file the way a provided one is protected. That is the evaluation maintainer’s job, and the maintainer runs only when the config’s evaluation_maintainer block turns it on (the RelBench benchmark’s override file does; the packaged config ships it off).

​
Versioned: the registry

When the evaluation maintainer runs, every evaluator the campaign uses is a version in .kapso/evaluation_registry.json, with its provenance: provided, maintainer_built (proposed from the brief) or derived_fast (a fast mode derived from a full one). A score never crosses a version boundary. The version the delivered candidate was measured under is solution.evaluator:
Without the maintainer, solution.evaluator is None for a provided evaluator too; the provided evaluator is still protected by the integrity check above.

​
Your own evaluator in Python

A caller-owned evaluator can also run once per finalized candidate, in an isolated worktree, with its metrics kept out of the search loop: build(..., iteration_evaluator=callback). The metrics of the delivered candidate are solution.callback_metrics. See External evaluation for the callback’s contract and failure policy.

The brief

Where the measurement is stated

Evaluation integrity

How a provided evaluator is protected

External evaluation

The Python callback

Review the result

Where the score and the version show up
Related pages: The brief · Evaluation integrity · External evaluation · Review the result Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.