The leaderboard at the site root is the newest generation. Older generations stay here unchanged — numbers are never rewritten retroactively, criteria differences are stated per generation. Raw per-attempt data is published alongside (verify, don't trust).
| Generation | Code state | Models | Tasks | pass@k | pass@1 | pass^k | Page |
|---|---|---|---|---|---|---|---|
| 2026-09-21 | main @ 146bf36 (PR #102, untagged) | 2 | 10 | 80% | 75% | 70% | leaderboard · data.json |
| 2026-09-10 | tag v0.6.0 (c39f50f) | 5 | 13 | 52% | 42% | 31% | leaderboard · data.json |
| 2026-09-09 | pre-v0.6.0 | 2 | 13 | 31% | not computed (legacy metrics) | ||
main commit 146bf36
(PR #102) — untagged, v0.9.0 not released yet. Two new-generation
models: openai/gpt-oss-20b (MoE, MXFP4) and qwen/qwen3-30b-a3b
(MoE, Q4_K_M). Task set shrank from 13 to 10: p-regulator,
pid-antiwindup and system-id moved to the plant target and now refuse unix honestly
(no scoring detector on unix — any output would pass), so cross-generation
comparison is valid only on the 10 common tasks. One infra event of 40 attempts
(gpt-oss-20b / frame-corrupt / attempt 1: the LLM server was restarted
mid-attempt, connection refused) — excluded from rates by the standing infra rule;
its attempt 2 honestly failed at the iteration limit. No LLM timeouts this
generation: every non-infra failure is iteration-limit exhaustion (no lower-bound
caveats, unlike 2026-09-10). A pre-campaign smoke run is kept out of the tree.v0.6.0.
Caveat: qwen3.5-9b and glm-4.6v-flash numbers are lower
bounds — 22 of 130 attempts ended with «LLM error: timed out»
(the 900 s LLM timeout of a thinking model); the server was healthy (no 503/400
batch anomalies), so these count as live attempts, not infra. The other three models
have zero LLM failures. Metrics recomputed with the current aggregator, which adds
pass@1 (attempt-level) and pass^k (reliability) next to the pair-level pass@k; the
pass@k value itself is unchanged from the original 2026-09-12 report.2026-09-19 nightly bench on the live ESP32 board: uart-echo PASS 10.0 s, bme-read PASS 14.7 s — the bench the campaigns ran on is itself checked regularly; the meteo firmware is restored after every series.
results.jsonl of the current generation (per model, per task).