Quorum

Which leaderboard gaps does the data actually support?

The table below is a synthetic leaderboard. The scores differ. Read down the observed score column and you will start ranking the models — that is what a column of numbers does to a reader.

Now look at the pairwise matrix. Every pair recomputed with an anytime-valid confidence sequence on the paired, scaffold-marginalised contrast. Red means the interval contains zero: this data cannot tell those two apart.

Models

Pairwise separation

Intervals are anytime-valid: they may be inspected after every observation with no correction for peeking. Alpha is split across all candidate pairs by Bonferroni. The “additional rollouts” figure extrapolates the observed shrinkage rate — it is a projection, not a guarantee.

How to read “not distinguishable”

It means this data cannot separate these two. It does not mean the models are equal, and it is not a criticism of either one. The demo data here is synthetic and the labels are placeholders — nothing on this page measures, ranks or disparages a real system.

Run it on your own data

This page is static, so it cannot take an upload. The same analysis runs locally in one command:

pip install -e .
quorum compare modelA modelB --benchmark synthetic-easy --budget $50