Module: compare

File: src/spatial_graph_algorithms/compare/ Status: Experimental.

Purpose

compare orchestrates metric-first comparative studies. It is designed for questions such as:

How do mds, strnd, and landmark_mds perform on the same graph?
How does one reconstruction method behave as false-edge noise increases?
How sensitive are results to graph-generation mode, shape, or seed?

It returns a ComparisonResult — a thin wrapper around a tidy pandas.DataFrame with built-in helpers for summarising, ranking, and plotting. The raw DataFrame is always accessible via .df for full pandas flexibility.

Typical Usage

from spatial_graph_algorithms.compare import parameter_grid, run_comparison

graphs = parameter_grid(
    base={"n": 500, "dim": 2, "shape": "square", "k": 8},
    vary={
        "mode": ["knn", "delaunay_corrected"],
        "false_edge_fraction": [0.0, 0.05, 0.10],
    },
)

reconstructions = parameter_grid(
    cases=[
        {"method": "mds"},
        {"method": "strnd"},
        {"method": "landmark_mds", "n_landmarks": 32},
        {"method": "landmark_mds", "n_landmarks": 64},
    ],
)

results = run_comparison(
    graph_specs=graphs,
    reconstruction_specs=reconstructions,
    seeds=[1, 2, 3],
)

# Summarise: mean CPD and KNN per (graph condition, method)
results.summary()

# Rank: best method per graph type
results.best(metric="cpd")

# Visualise: grouped bar chart
results.plot(metric="cpd", by="method", hue="graph_label")

# Access the raw tidy DataFrame at any time
results.df.head()

Preview the planned work before running it:

from spatial_graph_algorithms.compare import dry_run_comparison

plan = dry_run_comparison(
    graph_specs=graphs,
    reconstruction_specs=reconstructions,
    seeds=[1, 2, 3],
)

plan[["graph_label", "reconstruction_label", "seed", "method"]]

Save and reload results:

results.save("results/study.csv")
from spatial_graph_algorithms.compare import ComparisonResult
reloaded = ComparisonResult.load("results/study.csv")
reloaded.summary()

Avoiding Unwanted Combinations

Use grouped grids when different graph modes need different parameters. This keeps the study explicit and avoids invalid cartesian products.

graphs = parameter_grid(
    groups=[
        {
            "base": {"n": 500, "dim": 2, "shape": "square", "mode": "knn"},
            "vary": {"k": [4, 8], "false_edge_fraction": [0.0, 0.10]},
        },
        {
            "base": {"n": 500, "dim": 2, "shape": "square", "mode": "epsilon"},
            "vary": {"epsilon": [0.10, 0.20], "false_edge_fraction": [0.0, 0.10]},
        },
        {
            "base": {"n": 500, "dim": 2, "shape": "square", "mode": "delaunay_corrected"},
            "vary": {"false_edge_fraction": [0.0, 0.10]},
        },
    ],
)

For smaller cases, where= can filter an expanded grid, and drop_none=True can remove inactive parameters after filtering.

Output: `ComparisonResult`

run_comparison() returns a ComparisonResult with one row per:

graph_spec × seed × reconstruction_spec

Built-in analysis methods

Method	What it does
`.summary(by=..., metrics=...)`	Mean metrics grouped by `["graph_label", "method"]` (default)
`.best(metric=..., by=...)`	Best method per group by one metric
`.plot(metric=..., by=..., hue=...)`	Grouped bar chart; returns `matplotlib.Figure`
`.save(path)`	Write to CSV or Parquet (inferred from extension)
`.load(path)`	Class method; restore from CSV or Parquet
`.df`	The raw `pandas.DataFrame` for full pandas access

DataFrame columns

graph_label, reconstruction_label, seed, method
graph_<param> — generation parameters with prefix
recon_<param> — reconstruction parameters with prefix
status and error
generation_seconds and reconstruction_seconds
graph metrics from metrics.evaluate()
reconstruction quality metrics: cpd, knn, and optional distortion

Failed graph generation or reconstruction is recorded in the row and the study continues.

Relationship to `verify`

Use verify.run_report() when you want a complete artifact directory with CSVs and plots for one run.

Use compare.run_comparison() when you want a single in-memory table for many parameter combinations.

Benchmark Gym

The benchmark gym adds fixed, named datasets on top of the lower-level runners. It is intended for repeatable comparisons across:

demo
simulated_easy
simulated_mid
simulated_difficult
experimental
smoke

Each suite lives under data/benchmark/<suite>/manifest.json. The manifest contains a description, rationale, graph definitions, method definitions, task metrics, and the graph paths or experimental dataset names used by the suite. Real experimental input files live under data/experimental/.

from spatial_graph_algorithms.compare.gym import list_suites, load_suite, run_suite

list_suites()
suite = load_suite("smoke")
print(suite.description)

result = run_suite("smoke", task="pipeline")
result.df.head()
result.delta()

Supported tasks:

Task	Meaning	Primary metric
`denoise`	score/filter saved synthetic graphs	`auc_pr`
`reconstruction`	reconstruct saved or experimental graphs	`cpd`
`pipeline`	raw baseline plus `denoise → reconstruct`	`delta_cpd`

Experimental suites do not support denoise-only F1/AUC because real datasets do not have edge_metadata["is_false"] labels. They can still run reconstruction and pipeline tasks; quality is computed from positioned nodes only because metrics.evaluate() ignores rows with NaN positions.

The built-in experimental suite currently registers:

human_tonsil
mouse_embryo_bipartite
weinstein2019
hu2026_c58_4

The other Hu 2026 variants (hu2026_c58_5, hu2026_c58_8, and hu2026_c58_16) are available in data/experimental/hu2026/ and registered for manual loading, but the suite uses hu2026_c58_4 by default because it has the most positioned nodes with the smallest edge list among the four variants.

Materialized Graphs

Synthetic benchmark graphs are saved as directories:

graphs/graph_000/
├── adjacency.npz
├── positions.npy
├── edge_metadata.csv
├── node_metadata.csv
└── metadata.json

metadata.json records the original graph spec, seed, difficulty tier, edge counts, false-edge counts, true:false edge ratio, and false-edge rate of total.

Only the tiny smoke materialized graphs are tracked in git. Larger benchmark graphs are regenerated locally:

python scripts/build_benchmark_datasets.py --suite simulated_easy
python scripts/build_benchmark_datasets.py --suite simulated_mid
python scripts/build_benchmark_datasets.py --suite simulated_difficult

Use demo for visual report inspection before running larger suites. It builds six n=500 graphs from three graph specifications and two seeds:

python scripts/run_gym_report.py --suite demo --build --quiet

Custom Benchmarks

To create a new benchmark, copy an existing manifest, change the name, description, rationale, graph_specs, seeds, and task method specs, then build it:

python scripts/build_benchmark_datasets.py --suite path/to/my_suite/manifest.json

Run it by passing the manifest or suite directory:

result = run_suite("path/to/my_suite/manifest.json", task="denoise")

Reports

compare.report turns one or more ComparisonResult objects into ranked tables, labelled plots, and a Markdown report.

from spatial_graph_algorithms.compare.gym import load_suite, run_suite
from spatial_graph_algorithms.compare.report import gym_report

suite = load_suite("smoke")
results = {
    "reconstruction": run_suite(suite, task="reconstruction"),
    "pipeline": run_suite(suite, task="pipeline"),
}

report_path = gym_report(results, outdir="results/gym/smoke", suite=suite)

The report directory contains:

metadata.json
report.md
plots/*.png
<task>/raw.csv
<task>/leaderboard.csv
<task>/leaderboard_by_graph.csv
<task>/status_counts.csv
<task>/failed_rows.csv
<task>/pipeline_deltas.csv   # pipeline only

leaderboard.csv is the headline table aggregated across graph instances by difficulty, so plots show one bar per method or method pair. More detailed graph-condition rankings are written separately to leaderboard_by_graph.csv.

The command-line driver can build generated suites, run selected tasks, filter method subsets in memory, and write the report:

python scripts/run_gym_report.py \
  --suite simulated_easy \
  --build \
  --tasks reconstruction denoise pipeline \
  --denoise-methods jaccard square_bipartite community \
  --reconstruction-methods mds landmark_mds strnd \
  --quiet

The method filters match either method or _label from the manifest and do not modify the committed manifest.