Not paper results

Pilot data

Everything on this page is test, smoke, calibration, or in-progress data (26 episodes). None of it feeds the results on the Overview, which uses only the paper pool (390 episodes). Pilot numbers exist to check the instrument and to size the real runs — treat every reading here as provisional.

What the pilot runs told us

Four things from the exploratory pool are worth knowing, none of them results:

• Capture was not ordered by capability in calibration. With 5 attacker episodes per model, a 4B model (gemma-3-4b) out-captured two frontier models. Noise-level evidence at that sample size — it is why the paper runs use far more episodes — but it is the observation that motivated the capability-ladder experiment (10_family_ladder).
• An explicit broken promise exists in the logs. In a calibration episode, gemma-3-4b wrote "I'll reciprocate next round if you're willing", received the pull, and never pulled back (view in reader). This is the exact case the promise judge will classify.
• Models did not spontaneously broker. The credit-smoke runs showed near-zero third-party credits until a nudge was added — the nudge was then removed so the paper runs observe spontaneous use only.
• A 9-episode pilot preceded the attack arms. 04_pilot ran 3 episodes per attacker (opus-4.8, gpt-5.5, qwen3.7-max) against the glm-5.2 background to shake out the attack config before the 90-episode arms; it stays in the exploratory pool and feeds no paper number.

All pilot episodes remain browsable in the reader.

Pilot

Pilot leaderboard

The same extraction measure as the overview leaderboard, computed over the pilot pool. Pools mix different games (attacker episodes, smokes, credit tests), so rows are not comparable the way paper rows are.

Rank by CI overlap: tied models share a rank (LMSYS convention).

Pilot

Test & calibration configs

Exploratory runs

Provenance

All eval files

Raw index of every .eval file behind this site — paper, in-progress, and test runs alike.