Benchmark#
Detailed evaluation (paper-style): Evaluation.
Regenerate all artifacts#
python scripts/run_all_benchmarks.py
Writes docs/source/_generated/*.json and *.rst for Sphinx.
Synthetic only (no tcl_test data):
python scripts/generate_benchmark_report.py
Real data only (requires tcl_test + conda env st for R reference):
python scripts/run_tcl_benchmark.py --also-docs
Real data (GSE288946)#
Sample |
Genes |
Cells |
Intercept med |diff| |
Theta med |diff| |
Intercept r |
Theta r |
Inf θ match |
Speedup |
|---|---|---|---|---|---|---|---|---|
GSM8779707 |
1995 |
2000 |
1.55e-09 |
2.44e-07 |
1.000000 |
0.028824 |
217/217 |
6.24x |
GSM8779708 |
1997 |
2000 |
2.52e-09 |
2.02e-07 |
1.000000 |
-0.000408 |
188/188 |
6.95x |
GSM8779709 |
1996 |
2000 |
6.19e-09 |
1.68e-07 |
1.000000 |
-0.000520 |
58/58 |
6.38x |
Metric |
Value |
|---|---|
Samples / total genes |
3 / 5,988 |
Intercept med |diff| (median across samples) |
2.52e-09 |
Genes with |Δ Intercept| < 0.01 (median %) |
100.0% |
Theta med |diff| (median across samples) |
2.02e-07 |
Inf θ match (pooled) |
463/463 |
Wall time Python / R (total) |
7.1s / 46.5s |
Speedup (median) |
6.38x |
Note
Tier-B alignment: Python and R use identical cells_step1 /
genes_step1 subsamples exported from Seurat vst() (seed 1448145).
Comparison is against a fresh sctransform::fit_glmGamPoi_offset
call (conda env st), not cached model_pars_step1.csv — the latter
differs from fit_glmGamPoi_offset by ~0.5 on the intercept scale.
Synthetic scaling#
Genes |
Cells |
Intercept med |diff| |
Theta med |diff| |
Intercept r |
Theta r |
Inf θ match |
Speedup |
|---|---|---|---|---|---|---|---|
100 |
200 |
0.0000 |
4.11e-07 |
1.000000 |
1.000000 |
0/0 |
164.39x |
200 |
500 |
0.0000 |
4.74e-07 |
1.000000 |
1.000000 |
0/0 |
76.89x |
500 |
1000 |
0.0000 |
5.77e-07 |
1.000000 |
1.000000 |
0/0 |
36.03x |
1000 |
1500 |
0.0000 |
6.63e-07 |
1.000000 |
1.000000 |
0/0 |
17.33x |
2000 |
2000 |
0.0000 |
6.32e-07 |
1.000000 |
1.000000 |
0/0 |
8.79x |
Tests#
pytest tests/test_theta_od_parity.py -q
pytest tests/test_parity_r.py -m parity_r # optional, needs R