Gym Leaderboards

🏆 Open the interactive leaderboards →

The benchmark gym ranks denoising + reconstruction algorithms on a fixed set of datasets so it is obvious, at a glance, what the best method is on each dataset and how much room is left to improve. The leaderboards are the scoreboard for that gym — built to make you want to beat them.

What you get

Running the builder produces, under results/study/leaderboards/:

File	What it is
`leaderboards.html`	Open this. A single self-contained interactive page — sortable boards, headroom bars, a density×noise regime heatmap, and a quality-vs-cost scatter. No server, works offline, opens by double-click.
`INDEX.md`	Every board's champion in one table, sorted by headroom (biggest open problems first).
`<suite>/*.md`	One markdown board per dataset/suite for git diffs and review.
`leaderboards.json`	The same data, machine-readable.

How a board is organised

An entry is one competitor: a (denoiser, reconstructor) pair. The raw none + <reconstructor> baseline is always an entry, so you can see whether denoising actually helped.
A board ranks entries on a primary metric: reconstruction → CPD, denoise → AUC-PR, pipeline → CPD (with the denoising gain shown as a secondary ΔCPD column).
Reconstruction and pipeline boards can be ranked by CPD or KNN with the "rank by" toggle (CPD = global accuracy, KNN = local neighbourhood accuracy). The champion, headroom, and ranking all follow the chosen metric — and the two often disagree (e.g. on sim_bipartite_grid, MDS wins CPD while STRND wins KNN). The index has the same toggle.
Headroom = ceiling − champion score (all primary metrics have ceiling 1.0). A big headroom means a wide-open problem.

Per-dataset vs aggregated

This is the key scoping rule:

Experimental suites → one board per network. Real networks (human_tonsil, weinstein2019, …) are not interchangeable, so each gets its own ranking. The ± you see is run variance on that network.
Simulated suites → one board per suite, averaging over the many graphs in it, plus a regime heatmap showing which method wins in each density × noise cell.

Building / refreshing the leaderboards

After a study run (results live under results/study/):

python scripts/build_leaderboards.py            # writes results/study/leaderboards/
python scripts/build_leaderboards.py --open     # ... and open the HTML

scripts/run_full_study.py builds the leaderboards automatically as its final step, so a full study refresh needs no extra command.

Useful flags:

Flag	Purpose
`--results-dir DIR`	Read results from somewhere other than `results/study`.
`--outdir DIR`	Write the leaderboards somewhere other than `<results-dir>/leaderboards`.
`--open`	Open `leaderboards.html` when done.
`--docs`	Also publish the HTML snapshot to `docs/leaderboards/` so the docs site serves it. Commit that file to update the live page.

The builder only reads the raw.csv files already written by the gym — it never recomputes metrics, so it is fast and safe to run repeatedly.

Adding a new method and putting it on the board

When you implement a new reconstructor or denoiser, three steps put it on the leaderboards (the first is the standard method-authoring flow in new_method_checklist.md):

Implement it so reconstruct(..., method="<name>") (or score_edges(..., method="<name>")) works.
Enter it by adding a spec to the relevant suite manifest(s) under data/benchmark/<suite>/manifest.json:
```
{ "_label": "<name>", "method": "<name>" }
```
Put it in reconstruction_specs (and the pipeline reconstruction_specs), or denoise_specs, of every suite you want it scored on.

Run and rebuild — the run resumes from checkpoints, so only your new entry actually computes:

python scripts/run_full_study.py --suites <suite>     # rebuilds boards at the end
# or, for a single suite into results/gym/<suite>:
python scripts/run_gym_report.py --suite <suite>
python scripts/build_leaderboards.py --results-dir results/gym

Your entry takes the crown when its mean primary metric beats the current champion's by more than its standard deviation. The INDEX (sorted by headroom) will show where your method moved the needle — and where the open problems still are.

Adding a new dataset

The boards auto-adapt to whatever results exist, but you must (re)run the gym to produce those results — nothing recomputes on its own. The builder discovers suites and datasets by scanning the per-task raw.csv files under the results directory, so a new dataset shows up as a new board automatically once its results are there.

For a new experimental network:

Register it in compare/datasets.py (EXPERIMENTAL_DATASETS) and add its name to data/benchmark/experimental/manifest.json under real_datasets.
Run the gym so its raw.csv is written, then rebuild:
```
python scripts/run_full_study.py --suites experimental
python scripts/build_leaderboards.py --docs
```
A new per-network board appears for it automatically (experimental suites split one board per graph_id).

For a new simulated suite: add data/benchmark/<suite>/manifest.json, then run run_full_study.py --suites <suite> and rebuild. A new suite board (with its regime heatmap) appears.

What does not happen automatically: the gym does not re-run when you add a dataset, and the committed docs/leaderboards/leaderboards.html snapshot does not refresh until you run build_leaderboards.py --docs and commit it. Runs resume from checkpoints, so re-running only computes the genuinely new rows.

API

The leaderboard logic lives in spatial_graph_algorithms.compare.report:

leaderboard(result, *, task, scope_cols=None, ...) — rank entries within each scope group (scope_cols=["graph_id"] for per-network boards).
board_champion(board, *, task) — the rank-1 entry with its headroom.

scripts/build_leaderboards.py wraps these into the HTML/markdown/JSON output.

Regression guard (CI)

A CI job (scripts/check_champion_regression.py) runs the small, deterministic smoke suite, finds the best score on each board for each metric (CPD and KNN for reconstruction/pipeline, AUC-PR and F1 for denoise), and compares them to a committed baseline at tests/champion_baseline.json. If any champion metric drops by more than the tolerance (default 0.05), the build fails — the quality counterpart to the benchmarks/ speed guard. Tracking KNN as well as CPD means a change that quietly wrecks local accuracy while holding global accuracy is still caught.

When a champion legitimately changes (you improved a method, or accepted a new best), regenerate and commit the baseline — that commit is the audit trail:

python scripts/check_champion_regression.py --update   # rewrites tests/champion_baseline.json

Not yet covered

Per-dataset characteristics page — a docs page describing each dataset (size, structure, ground-truth coverage, provenance). Planned; until then the board headers show node/edge counts.
Ground-truth coverage in headers for experimental boards.
CI regression check that fails when a champion metric drops.