Skip to main content
RelBench (Stanford/Kumo) benchmarks supervised prediction over temporal relational databases: 11 databases, 66 tasks across entity classification (AUROC), entity regression (MAE/NMAE, R²), recommendation (MAP@K), and the new v2 autocomplete family. A model receives a multi-table database plus seed rows (entity id, seed time) and predicts future outcomes, missing attributes, or ranked future links — with every feature and neighborhood censored at each row’s seed time.

​
Results

Bar chart of three RelBench panels: outcome prediction, Kapso 81.2 AUROC against KumoRFM-v2's 79.6; forecasting, Kapso 0.2476 NMAE against 0.2912, lower being better; recommendations, Kapso 18.4 MAP against the best reported 15.7. Each panel is drawn from its own truncated axis

Kapso against KumoRFM-v2 on RelBench outcome prediction, forecasting and recommendations

All scores come from the official RelBench evaluator, under the same data, hardware and time budget. Published entries live on the official RelBench leaderboard.

​
How Kapso runs it

Each experiment in the tree search is implemented by the coding agent on a git branch and then evaluated by RelBenchHandler:
  1. The candidate’s main.py runs against a sanitized, read-only RelBench cache: the database is physically truncated at the test cutoff (or, for autocomplete, post-cutoff target values are blanked), and the test task table carries only entity ids and timestamps — test labels are physically absent from the candidate’s process.
  2. The candidate writes val_predictions.npy and test_predictions.npy to $KAPSO_RUN_DATA_DIR (contract validated: shapes, dtypes, ranges).
  3. The handler computes official validation metrics — the search score — and computes test metrics privately (quarantined until final_evaluate).
  4. final_evaluate picks the best-by-validation run, audits its code for forbidden access patterns, and writes a leaderboard-ready final_report.json.
This is exactly the RelBench protocol: tune on validation, report test once. The problem context injected into ideation includes the database schema, the task’s label-generating SQL, and per-family playbooks (temporal GNNs, censored SQL feature engineering + GBDT, repeat-behavior recommendation heuristics, hybrid rankers). A starter kit of contract helpers — environment and task loading, prediction saving and validation, method choice entirely the agent’s — is seeded into every workspace under kapso_datasets/.

​
Usage

​
CLI options

​
Setup

Datasets download automatically to ~/.cache/relbench on first use. See benchmarks/relbench/README.md for the campaign plan, protocol guarantees, and per-family strategy notes.

Installation

Installing the harness

Trajectory learning

Banking what a campaign learned
Related pages: Installation · Trajectory learning · IOAI 2026 Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.