Score Kapso candidates with your own external evaluator

Kapso can run a caller-owned evaluator once for every candidate finalized by an evolution iteration, except candidates refused by cost-aware selection. This is useful for integration tests, benchmark harnesses, validation suites, performance measurements, and observational holdout reporting. External metrics are deliberately separate from Kapso’s internal node.score. They do not affect idea generation, stopping, parent selection, or the final branch selected by default.

Basic usage

import json
import subprocess

from kapso import IterationEvaluationResult


def evaluate_candidate(context):
    completed = subprocess.run(
        ["python", "evaluation/run.py", "--json"],
        cwd=context.workspace_dir,
        check=True,
        capture_output=True,
        text=True,
    )
    report = json.loads(completed.stdout)
    return IterationEvaluationResult(
        metrics={
            "validation_accuracy": report["accuracy"],
            "p95_latency_ms": report["p95_latency_ms"],
        },
        primary_metric="validation_accuracy",
        metadata={"suite_version": report["suite_version"]},
    )


solution = kapso.evolve(
    goal="Improve the support agent",
    output_path="./campaign",
    max_iterations=3,
    iteration_evaluator=evaluate_candidate,
)

solution.metadata["external_metrics"] contains the metrics attached to the internally selected best candidate. All candidate metrics are retained in .kapso/experiment_history.json and, for resumable runs, in .kapso/run_state.json.

Callback context

The callback receives an IterationEvaluationContext:

@dataclass(frozen=True)
class IterationEvaluationContext:
    iteration: int
    goal: str
    workspace_dir: Path
    git_ref: str
    parent_ref: str
    node: SearchNode

iteration is the one-based campaign iteration, including resumed calls.
workspace_dir is a temporary detached Git worktree at exactly git_ref.
git_ref is the finalized candidate branch.
parent_ref is the branch from which the candidate was created.
node is a snapshot. Mutating it cannot change Kapso’s live search state.

The worktree exists only during the callback. Read outputs and return metrics before the callback exits; do not retain its path. Changes made inside it are discarded and the root experiment workspace’s current checkout is unchanged.

Result contract

The callback must return IterationEvaluationResult:

@dataclass(frozen=True)
class IterationEvaluationResult:
    metrics: Mapping[str, float]
    primary_metric: str | None = None
    metadata: Mapping[str, Any] = field(default_factory=dict)
    baseline_metrics: Mapping[str, float] = field(default_factory=dict)
    metric_directions: Mapping[str, str] = field(default_factory=dict)

Metric names must be non-empty strings. Values must be finite numbers; booleans, strings, NaN, and infinity are rejected. primary_metric, when supplied, must name one of the returned metrics. Metadata must be JSON-compatible and have string keys. primary_metric describes the report. It does not make that metric a search or selection signal.

Baseline comparisons in watch

Optionally return baseline_metrics and metric_directions with your result:

return IterationEvaluationResult(
    metrics={"validation_accuracy": 0.9, "p95_latency_ms": 8.0},
    baseline_metrics={"validation_accuracy": 0.8, "p95_latency_ms": 10.0},
    metric_directions={
        "validation_accuracy": "maximize",
        "p95_latency_ms": "minimize",
    },
    metadata={"suite_version": "v1", "baseline_ref": "reference-commit"},
)

The caller measures the baseline using the same evaluation suite, dataset, configuration, and units as the candidate. Kapso does not run an extra baseline evaluation or assume that the previous search node is a valid baseline. Keep the baseline fixed across resume, and identify its Git ref and suite version in metadata. Baseline values must be finite numbers, just like candidate values. Every baseline key must name a returned candidate metric; misspelled or extra baseline keys raise IterationEvaluationValidationError. Kapso stores the derived report under external_evaluation_metadata["baseline_comparison"] alongside the existing node, iteration and branch lineage in experiment history and checkpoints. This metadata key is reserved: an evaluator that sets it in its own metadata is rejected. The report records baseline, candidate, signed delta (candidate - baseline), direction, and a verdict for each shared metric. A missing direction yields unclassified; a missing baseline is reported explicitly. improved, regressed, and unchanged are exact numeric comparisons, not statistical significance claims. kapso watch ./campaign shows the last candidate’s comparison; kapso watch ./campaign --tree shows comparisons for checkpointed nodes, including after a restart. These are observational reports, not promotion decisions. Search ranking, stopping, and final selection still use internal scores. This keeps holdout measurements out of the optimization loop.

Lifecycle and persistence

Generic search normally finalizes one candidate per iteration; expansion lanes can finalize several. The evaluator runs exactly once for each new candidate that was not refused by the cost policy. A resumed run evaluates only candidates created after the resume, not candidates already stored in its checkpoint. Make callbacks idempotent. If the process exits after the callback finishes but before the atomic checkpoint replacement, recovery may repeat that candidate’s measurement because the completed result was never made durable. Metrics are attached before experiment history and checkpoint persistence, so durable representations agree. Internal score, feedback, and stop state have already been produced by the strategy and are never replaced by the callback. Kapso does not include external metrics or evaluator errors in the agent-facing experiment-history formatter. They remain available to the caller and in durable run records without becoming feedback for later ideas.

Failure policy

The default record policy keeps the experiment loop running:

solution = kapso.evolve(
    ...,
    iteration_evaluator=evaluate_candidate,
    iteration_evaluator_failure_policy="record",
)

The candidate receives empty external metrics and an external_evaluation_error containing the exception type and message. The error is persisted in experiment history and the checkpoint. Use raise when every external measurement is mandatory:

solution = kapso.evolve(
    ...,
    iteration_evaluator=evaluate_candidate,
    iteration_evaluator_failure_policy="raise",
)

Kapso raises IterationEvaluationError and stops before writing that candidate to experiment history or the run checkpoint. The candidate Git branch remains available for inspection.

Holdout guidance

A true holdout metric should remain observational. Returning it from this hook is safe with the default selection behavior because Kapso continues to rank by internal node.score. Do not copy holdout values into the agent prompt, feedback, internal score, or a custom parent-selection policy. When you want a metric to guide optimization, use a validation or development split and label it accordingly. Reserve the holdout split for final reporting after model and configuration choices are complete. The callback is trusted application code, not a security sandbox. Kapso isolates its normal filesystem view and node object to prevent accidental mutation; the callback process still has the operating-system permissions of the caller.

Resume compatibility

When an evaluator is configured, Kapso includes its module-qualified callable name and failure policy in the run configuration fingerprint. Resume therefore requires the same evaluator entry point and policy. Closures with the same callable name but different captured values cannot be distinguished; keep such parameters in stable application configuration and do not change them during a campaign.

Evaluation integrity

Protecting a provided evaluator

Python API

The iteration_evaluator argument

Related pages: Evaluation integrity · Python API · Overview Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.

Getting started

Evolve system

Knowledge graph

Trajectory learning

Research

Deployment

Benchmarks

Score Kapso candidates with your own external evaluator

Basic usage

Callback context

Result contract

Baseline comparisons in watch

Lifecycle and persistence

Failure policy

Holdout guidance

Resume compatibility

Evaluation integrity

Python API

​Basic usage

​Callback context

​Result contract

​Baseline comparisons in watch

​Lifecycle and persistence

​Failure policy

​Holdout guidance

​Resume compatibility

​Related

Evaluation integrity

Python API

Basic usage

Callback context

Result contract

Baseline comparisons in watch

Lifecycle and persistence

Failure policy

Holdout guidance

Resume compatibility

Related