Skip to main content
RelBench (Stanford STAR) benchmarks supervised prediction over temporal relational databases: 11 databases, 66 tasks across entity classification (AUROC), entity regression (MAE/NMAE, R²), recommendation (MAP@K), and the new v2 autocomplete family. A model receives a multi-table database plus seed rows (entity id, seed time) and predicts future outcomes, missing attributes, or ranked future links — with every feature and neighborhood censored at each row’s seed time.

​
Results

Bar chart of three RelBench panels: classification, Kapso 81.2 AUROC against KumoRFM-v2's 79.3 and RT-J's 79.0; regression, Kapso 0.2476 NMAE against RT-J's 0.2711 and KumoRFM-v2's 0.2847, lower being better; recommendations, Kapso 18.4 MAP against the official leaderboard's best other entry, ID-GNN, at 9.3. The first two panels are drawn from a truncated axis, the third from zero

Kapso against the official RelBench leaderboard entries of Stanford STAR and Kumo AI

All Kapso scores come from the official RelBench evaluator on the official task sets; the comparison numbers are the methods’ own entries on the official RelBench leaderboard, read on 2026-10-11. The leaderboard is self-reported and sets no hardware or time budget, so no claim is made about either. KumoRFM-v2 is on the board as KumoRelational, the open release from the KumoRFM-2 paper. Neither it nor RT-J has a recommendation entry, so there the bar is ID-GNN, a RelBench baseline, at 9.3 mean MAP over the same ten tasks.

​
How Kapso runs it

Each experiment in the tree search is implemented by the coding agent on a git branch and then evaluated by RelBenchHandler:
  1. The candidate’s main.py runs against a sanitized, read-only RelBench cache: the database is physically truncated at the test cutoff (or, for autocomplete, post-cutoff target values are blanked), and the test task table carries only entity ids and timestamps — test labels are physically absent from the candidate’s process.
  2. The candidate writes val_predictions.npy and test_predictions.npy to $KAPSO_RUN_DATA_DIR (contract validated: shapes, dtypes, ranges).
  3. The handler computes official validation metrics — the search score — and computes test metrics privately (quarantined until final_evaluate).
  4. final_evaluate picks the best-by-validation run, audits its code for forbidden access patterns, and writes a leaderboard-ready final_report.json.
This is exactly the RelBench protocol: tune on validation, report test once. The problem context injected into ideation includes the database schema, the task’s label-generating SQL, and per-family playbooks (temporal GNNs, censored SQL feature engineering + GBDT, repeat-behavior recommendation heuristics, hybrid rankers). A starter kit of contract helpers — environment and task loading, prediction saving and validation, method choice entirely the agent’s — is seeded into every workspace under kapso_datasets/.

​
Usage

​
CLI options

​
Setup

Datasets download automatically to ~/.cache/relbench on first use. See benchmarks/relbench/README.md for the campaign plan, protocol guarantees, and per-family strategy notes.

Installation

Installing the harness

Trajectory learning

Banking what a campaign learned
Related pages: Installation · Trajectory learning · IOAI 2026 Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.