Estimate platform cost before evaluating a candidate

Generic search can estimate the platform cost of each candidate before running its evaluation. An estimate above the per-candidate cap refuses that candidate; an estimate above the remaining campaign budget stops the campaign. Accepted candidates are ranked using their internal score and platform cost together. This policy is off by default. The existing cost_budget argument still meters model spend in USD on a best-effort basis. Platform cost has its own unit, estimator, caps, and ranking weight.

Configuration

modes:
  GENERIC:
    search_strategy:
      type: generic
      params:
        cost:
          enabled: true
          estimator: my_campaign.costs.estimate_candidate
          cap_per_candidate: 10.0
          campaign_budget: 50.0
          unit: credits
          weight: 0.1

The packaged defaults are enabled: false, estimator: null, cap_per_candidate: null, campaign_budget: null, unit: "credits", and weight: 0.0. A null cap or budget has no limit. Enabling the policy without an estimator fails during initialization. Amounts, caps, budgets, and weights must be finite and non-negative. The packaged source is defaults.search_strategy.params.cost, inherited by every mode. A mode can override individual cost keys. Cost-aware campaigns require the generic strategy and budget.fidelity.mode: off. They also reject evaluator transitions before running a bridge evaluation. Those features can remeasure a candidate, while this policy accounts for one evaluation per node. unit is a label, with no currency conversion or provider pricing lookup. Estimates and measurements must use that same label. The history tools show both the estimate and the measurement when present, and measured cost always takes precedence in ranking and budget accounting.

Estimator contract

The estimator is caller-owned Python code, imported by its module-qualified name. It receives an IterationEvaluationContext with iteration, goal, workspace_dir, git_ref, parent_ref, and node. The workspace is a temporary detached Git worktree at the finalized candidate ref. iteration is one-based, including resumed iterations. The context is frozen and node is a detached snapshot, so mutating it cannot change live search state. The worktree exists only during the callback; changes inside it are discarded. See External evaluation for the shared context contract.

import json

from kapso.execution.cost_estimator import CostEstimate
from kapso.execution.iteration_evaluator import IterationEvaluationContext


def estimate_candidate(context: IterationEvaluationContext) -> CostEstimate:
    plan = json.loads((context.workspace_dir / "platform_plan.json").read_text())
    return CostEstimate(
        amount=plan["estimated_credits"],
        unit="credits",
        basis="Candidate warehouse plan",
        metadata={"plan_version": plan["version"], "ref": context.git_ref},
    )

Return a CostEstimate with an amount, unit, non-empty basis, and JSON-compatible metadata with string keys. Booleans are not valid amounts. Estimator exceptions and malformed results fail loudly. Keep the estimator inexpensive: inspect a plan or ask a platform for its estimate without running the candidate workload.

Admission and refusal

When cost is enabled, the implementation session is instructed to build the candidate and evaluator without running either. Kapso finalizes the candidate, checks evaluator integrity, calls the estimator, and admits evaluation only after checking the caps. The admitted run uses the registered evaluator when available, otherwise kapso_evaluation/evaluate.py. The judge then reads what the run captured, stdout and stderr, together with how the run ended; the session’s own evaluation claims never count, and in cost mode the manifest of record is the line the frame run printed. A run that fails or overruns its deadline, and a build that errored before any run, are judged on that output too and stay unscored and invalid, with feedback that explains why. Only a candidate refused by the cost check skips the judge. The evaluator retains the session’s configured env_strip, session_env_defaults, and expansion_lane_env: stripped names are removed, lane values override inherited values, and defaults apply only to absent names. Other inherited credentials and runtime settings remain available to the evaluator. An estimate strictly greater than cap_per_candidate sets score to None, marks evaluation_valid false, and records cost_refusal with the estimate and cap. The candidate remains in recent and similar history so the next idea can learn why it was refused. It cannot become a parent or top experiment. Refused candidates also skip the external iteration evaluator. The campaign adds each admitted candidate’s measured cost, or its estimate when no measurement is available. Refused candidates add no platform cost. If the next estimate would exceed campaign_budget, the campaign stops with cost_budget_exhausted in the checkpoint’s last_stop and the solution’s metadata["stopped_reason"]. An estimate equal to either limit is allowed. When both limits would be exceeded, the campaign budget stop takes precedence. A slice whose recorded spend has already reached campaign_budget stops before ideating, so a resumed campaign does not build a candidate only to refuse it. Candidates that fail or pause before admission, with no estimate or measurement, have no recorded platform charge. A scored candidate with an unknown cost is rejected when cost-aware ranking is enabled. These are estimate-based controls. Actual usage can exceed an estimate, and the coding agent is not a security sandbox. Enforce provider-side limits when a workload must have a hard spending ceiling.

Measured cost and ranking

An evaluator can add a cost string to its printed manifest line:

KAPSO_EVAL_MANIFEST {"score": 0.92, "cost": "2.5 credits"}

Registered evaluators retain their normal manifest fields, including fidelity, fraction, seed, items, and total items. An unregistered evaluator may print a manifest containing only cost alongside its normal evaluation output. Units may contain spaces, for example "2.5 core hours". The measurement is stored as measured_cost and replaces estimated_cost for that candidate in every cost-aware ranking and in campaign accounting. Zero is a valid measurement. An evaluation that ran counts toward the budget even if its result is invalid. Malformed or mismatched-unit costs are rejected. The raw internal score remains unchanged:

Problem direction	Higher-is-better selection value
Maximize	`score - weight * cost`
Minimize	`-score - weight * cost`

Equal-score candidates therefore prefer the cheaper design when weight is positive. Equal-cost candidates retain the problem’s score ordering. External iteration-evaluator metrics remain observational. The same cost utility guides best-candidate selection, parent_policy: best, top-experiment queries, and the strategy template’s ranking. With weight: 0.0, caps still apply without a cost penalty in selection. With enabled: false, existing rankings and agent-facing history remain unchanged.

Where cost decisions show up

Every cost decision is a campaign event, surfaced where the time budget and the knowledge tallies already are. The X/Y used figure on the console lines and in the refusal clause is the campaign’s spend before that candidate, the same number the refusal records as campaign_cost_used.

Surface	What the surface shows
Console, at admission	`[Orchestrator] node 1 estimated 2.0 credits; campaign 2.0/3.0 used; running evaluation`
Console, at refusal	`[Orchestrator] node 2 refused: campaign_budget (estimate 2.0 credits, 2.0/3.0 used)`
Console, after a completed run	`[GenericSearch] node 1 evaluation ran in 4.0s; measured 1.5 credits`; a failed or overrun run keeps its own line
`.kapso/status.json`, recent ring	The node-completed note gains a clause: `cost: 1.5 credits (measured)`, `cost: 2.0 credits (estimate)`, `refused: campaign_budget (2.0 estimate, 2.0/3.0 used)`, or `cost: not estimated` for a build that never reached admission
`.kapso/status.json`, payload	`platform_budget: {used, budget, unit, refused}`, present only when cost is enabled; `kapso watch --json` passes it through
`kapso watch ./campaign`	A second bar under the time bar: `cost: ▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░ 2.0/3.0 credits, 1 candidate refused`
`kapso watch ./campaign --tree`	`cost: 2.0` on each node’s row after its score, measured over estimated, and `refused` as the outcome of a cost-refused node
`kapso evolve`, the completion summary	`Platform cost: 2.0/3.0 credits, 1 candidate refused`

A campaign with cost disabled prints, writes and renders none of this.

History and resume

The experiment-history tools show each candidate’s estimate, measurement, unit, and refusal when present. They report the full campaign’s budget usage once at the top, even when a query returns only one candidate. Estimate basis, metadata, feedback, and query text are passed to ideation in full. The MCP history process reads the cost policy and the problem’s scoring direction from the campaign checkpoint (.kapso/run_state.json), so it ranks with the campaign’s actual weight, unit, and direction. The estimator entry point and all cost settings are part of the run configuration fingerprint. Resume rejects a changed estimator, cap, or other cost setting by name. Keep the estimator’s implementation and its external pricing inputs stable during a campaign; the entry point does not fingerprint arbitrary caller-owned code or remote pricing data. Related pages: Parent selection · Configuration · External evaluation Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.

Getting started

Evolve system

Knowledge graph

Trajectory learning

Research

Deployment

Benchmarks

Estimate platform cost before evaluating a candidate

Configuration

Estimator contract

Admission and refusal

Measured cost and ranking

Where cost decisions show up

History and resume

​Configuration

​Estimator contract

​Admission and refusal

​Measured cost and ranking

​Where cost decisions show up

​History and resume

Configuration

Estimator contract

Admission and refusal

Measured cost and ranking

Where cost decisions show up

History and resume