ironbench leaderboard - firmware-agent benchmark
| Model | pass@k | Tasks | control | fsm | io | protocol | resilience |
|---|
| glm-4.6v-flash | 54% | 13 | 4/8 (50%) | 1/2 (50%) | 4/4 (100%) | 0/8 (0%) | 2/8 (25%) |
| qwen2.5-7b-instruct | 54% | 13 | 4/8 (50%) | 2/2 (100%) | 3/4 (75%) | 0/8 (0%) | 2/8 (25%) |
| qwen3.5-4b | 46% | 13 | 1/8 (12%) | 1/2 (50%) | 4/4 (100%) | 1/8 (12%) | 3/8 (38%) |
| qwen3.5-9b | 38% | 13 | 3/8 (38%) | 0/2 (0%) | 4/4 (100%) | 0/8 (0%) | 1/8 (12%) |
| zai-org/glm-4.7-flash | 69% | 13 | 6/8 (75%) | 1/2 (50%) | 4/4 (100%) | 2/8 (25%) | 4/8 (50%) |