Skip to main content
RelBench (Stanford/Kumo) benchmarks supervised prediction over temporal relational databases: 11 databases, 66 tasks across entity classification (AUROC), entity regression (MAE/NMAE, R²), recommendation (MAP@K), and the new v2 autocomplete family. A model receives a multi-table database plus seed rows (entity id, seed time) and predicts future outcomes, missing attributes, or ranked future links — with every feature and neighborhood censored at each row’s seed time.

​
How Kapso runs it

Each experiment in the tree search is implemented by the coding agent on a git branch and then evaluated by RelBenchHandler:
  1. The candidate’s main.py runs against a sanitized, read-only RelBench cache: the database is physically truncated at the test cutoff (or, for autocomplete, post-cutoff target values are blanked), and the test task table carries only entity ids and timestamps — test labels are physically absent from the candidate’s process.
  2. The candidate writes val_predictions.npy and test_predictions.npy to $KAPSO_RUN_DATA_DIR (contract validated: shapes, dtypes, ranges).
  3. The handler computes official validation metrics — the search score — and computes test metrics privately (quarantined until final_evaluate).
  4. final_evaluate picks the best-by-validation run, audits its code for forbidden access patterns, and writes a leaderboard-ready final_report.json.
This is exactly the RelBench protocol: tune on validation, report test once. The problem context injected into ideation includes the database schema, the task’s label-generating SQL, per-family playbooks (temporal GNNs, censored SQL feature engineering + GBDT, repeat-behavior recommendation heuristics, hybrid rankers), and the current published state of the art for the task (benchmarks/relbench/data/sota.json). A starter kit with the official RelBench reference implementations is seeded into every workspace under kapso_datasets/.

​
Usage

​
CLI Options

​
Setup

Datasets download automatically to ~/.cache/relbench on first use. See benchmarks/relbench/README.md for the campaign plan, protocol guarantees, and per-family strategy notes.