Radical Geek field note
The Radical Geek Local Coding Model Leaderboard
Eight downloaded coding models, one frozen SWE-bench baseline, official scoring and the runtime failures the headline scores leave out.
I wanted a leaderboard I could actually use to make routing decisions.
The model cards were useful for choosing candidates, but they could not answer the question I had on the local stack: when each downloaded model sees the same issue and the same heuristically retrieved source context, can it produce a patch that applies and passes the repository tests without Qwen repairing the answer afterwards?
I ran the models on the same five SWE-bench Verified tasks, preserved every raw
trace, checked each patch with git apply --check, then used the official
SWE-bench harness for the final result.
The result is blunt. Qwen3.6-35B-A3B Q4_K_M is the only model in the shared
matrix to produce a cleanly applicable, officially resolved patch. It scored
1/5. Every other model scored 0/5.
That headline is useful, but it is not the whole report. A zero made from five short, non-applicable patches is operationally different from a zero that consumed 98,304 completion tokens on one task. The detailed breakdown matters.
Current leaderboard
| Rank | Model and quant | Size | Resolved | Clean apply | Patch yield | P50 latency | Output tokens |
|---|---|---|---|---|---|---|---|
| 1 | Qwen3.6-35B-A3B Q4_K_M | 20.614 GiB | 1/5 | 1/5 | 5/5 | 107.0 s | 25,300 |
| =2 | DeepSeek-Coder-V2-Lite Q6_K | 13.101 GiB | 0/5 | 0/5 | 5/5 | 34.9 s | 3,248 |
| =2 | Gemma 4 E4B Q4_K_M | 4.969 GiB | 0/5 | 0/5 | 5/5 | 105.1 s | 19,212 |
| =2 | Kimi-Linear-48B-A3B Q4_K_M | 27.997 GiB | 0/5 | 0/5 | 4/5 | 47.2 s | 9,128 |
| =2 | Devstral Small 2 24B Q4_K_M | 13.350 GiB | 0/5 | 0/5 | 4/5 | 78.0 s | 1,809 |
| =2 | Ornith 1.0 35B Q4_K_M | 19.713 GiB | 0/5 | 0/5 | 3/5 | 125.5 s | 123,569 |
| =2 | GPT-OSS-20B MXFP4 | 11.278 GiB | 0/5 | 0/5 | 1/5 | 20.2 s | 5,978 |
| =2 | Laguna S 2.1 118B-A8B Q4_K_M | 68.098 GiB | 0/5 | 0/5 | 0/5 | 135.7 s | 6,773 |
The shared rank ties every model with zero solves and zero clean applies. Latency and patch yield are important diagnostics, but I am not allowing either to outrank correctness.
Laguna S 2.1: the fit baseline
Laguna’s official Q4_K_M GGUF is 68.098 GiB. The Evo exposes 30 GiB of host RAM and a 64 GiB GPU carve-out, so the weights and a 131k f16 KV cache could not fit the same envelope as the smaller models. The first attempt generated 7,836 Astropy tokens before the kernel killed the server under host-memory pressure.
I did not count that as a model failure. I kept the same quant and official sampler, then fitted the runtime honestly:
- Poolside’s
lagunallama.cpp branch - 65,536-token context
- 38 of 48 layers on the GPU
- Q8 KV cache
- thinking enabled
- temperature 1.0, top-p 1.0, top-k 20 and min-p 0
At the required 32,768-token allowance, Astropy consumed the entire budget and returned an empty patch. I increased the allowance to 40,960 and restarted all five tasks. Astropy then ended naturally at 5,944 tokens, removing the ceiling ambiguity, but it still returned an empty patch.
The final Laguna result is 0/5: five empty patches, no evaluator errors and no
40,960-token ceiling breaches. It is tied with the other zero-solve,
zero-clean-apply models on correctness, while its separate fit envelope remains
visible rather than being presented as identical hardware conditions.
The five tasks
| Project | Instance |
|---|---|
| Astropy | astropy__astropy-12907 |
| Django | django__django-10097 |
| Matplotlib | matplotlib__matplotlib-13989 |
| Pytest | pytest-dev__pytest-10051 |
| Sphinx | sphinx-doc__sphinx-10323 |
Five fixed tasks are not enough for a universal intelligence ranking. They are enough for a controlled promotion screen, especially when the raw failures are kept rather than collapsed into one percentage.
Task-by-task result
The token count in each cell is the completion length. Invalid means an
extractable patch failed the clean-apply gate. No patch means the model did
not return an extractable patch. Not resolved means the official scorer ran
but the repository tests did not pass.
| Model | Astropy | Django | Matplotlib | Pytest | Sphinx |
|---|---|---|---|---|---|
| Qwen3.6 | Resolved · 4,313 | Invalid · 5,578 | Invalid · 4,810 | Invalid · 4,935 | Invalid · 5,664 |
| DeepSeek Coder V2 Lite | Invalid · 618 | Invalid · 652 | Invalid · 832 | Invalid · 260 | Invalid · 886 |
| Gemma 4 E4B | Invalid · 7,152 | Invalid · 2,355 | Not resolved · 3,691 | Invalid · 3,743 | Invalid · 2,271 |
| Kimi Linear | No patch · 5,515 | Invalid · 748 | Invalid · 1,353 | Invalid · 983 | Invalid · 529 |
| Devstral Small 2 | Invalid · 326 | Invalid · 579 | No patch · 167 | Not resolved · 182 | Not resolved · 555 |
| Ornith 1.0 | Invalid · 16,533 | Invalid · 6,361 | Ceiling/invalid · 98,304 | No patch · 2,003 | No patch · 368 |
| GPT-OSS-20B | No patch · 741 | Invalid · 5,026 | No patch · 42 | No patch · 109 | No patch · 60 |
| Laguna S 2.1 | No patch · 5,944 | No patch · 99 | No patch · 165 | No patch · 380 | No patch · 185 |
The baseline I froze
The shared seven-model matrix used:
SWE-bench/SWE-bench_Verified- one model for the whole generation
- heuristic retrieval capped at 80,000 source characters and 12 files
- 131,072-token context
- parallel 1
- maximum GPU offload
- no model-assisted repair
git apply --check- the official SWE-bench scorer
I selected the best already-downloaded quant that fitted the hardware. This is fit parity, not quant parity. Forcing every model to Q4 would have downgraded DeepSeek even though Q6 fitted. Forcing Q6 across the board would have excluded the larger candidates. The benchmark records the exact file size and SHA-256 hash instead.
The 32k output rule changed the run
The first allowance was 32,768 tokens. Qwen spent the allowance in reasoning without returning a final patch, so I increased the allowance and restarted the matrix from the beginning.
At 65,536 tokens, Ornith reached the ceiling on Sphinx and still did not produce a usable patch. That made the run unsuitable for comparison, so I restarted again.
The final shared allowance was 98,304 tokens. This is the largest comparable output budget under GPT-OSS’s 131,072 native context once the prompt is included. Ornith then consumed all 98,304 tokens on Matplotlib and failed the patch gate. There was no honest higher shared setting to try, so that result is recorded as a maximum-context ceiling failure.
The output limit was not being treated as a target. It was a diagnostic ceiling, and the restart rule did its job.
Quant and sampler settings
| Model | Quant | Generation settings |
|---|---|---|
| Qwen3.6-35B-A3B | Q4_K_M | temperature 0.6, top-p 0.95, top-k 20, min-p 0, repetition penalty 1 |
| Gemma 4 E4B | Q4_K_M | temperature 1.0, top-p 0.95, top-k 64 |
| Kimi-Linear-48B-A3B | Q4_K_M | temperature 1.0 |
| Devstral Small 2 24B | Q4_K_M | temperature 0.15 |
| GPT-OSS-20B | MXFP4 | temperature 1.0, reasoning effort high |
| DeepSeek-Coder-V2-Lite | Q6_K | temperature 0, top-p 0.95, top-k 50 |
| Ornith 1.0 35B | Q4_K_M | temperature 1.0, top-p 0.95, top-k 20 |
| Laguna S 2.1 | Q4_K_M | temperature 1.0, top-p 1.0, top-k 20, min-p 0, thinking enabled |
I checked the current model documentation before freezing these settings. For
Laguna in particular, Poolside recommends thinking for agentic coding, its
laguna llama.cpp branch with --jinja, and publishes Q4_K_M as the local GGUF
route. The optional DFlash draft model was not used because it is not among the
downloaded models being tested:
What each result actually says
Qwen is still the control
Qwen did not produce five beautiful patches. It produced one verified solve and four invalid patches. That is not a reason to pretend the baseline is stronger than it is.
It is enough to retain Qwen as the control. It was the only model to cross all three gates: emit a patch, apply cleanly and pass the repository tests. Every challenger now has a simple target.
DeepSeek was fast and wrong
DeepSeek’s 34.9-second median looks good on a latency chart. It also emitted five patches. None applied cleanly. Fast invalid output is still invalid output.
Gemma remains a supporting model
Gemma emitted a patch for every task and reached the official test stage on Matplotlib, but it resolved nothing. That fits the evidence from the wider stack: Gemma remains interesting as a classifier, reviewer and judge, not the owner of the final patch.
Kimi and Devstral remain specialists
Kimi and Devstral both produced useful structure in earlier planning and review work. This isolated baseline says neither earned patch ownership. I will keep them available for role-specific experiments without making them a tax on every coding request.
GPT-OSS needs a tighter job
GPT-OSS had the fastest median because several tasks ended with almost no usable output. It produced one extractable patch from five tasks. A narrow judge role may still suit it; this implementer baseline did not.
Ornith failed the output-control gate
Ornith is no longer unranked. The isolated run is now complete and the result is
0/5, with no cleanly applicable patch. Its 98,304-token Matplotlib failure is
the clearest diagnostic in the matrix. The previous apparent Ornith success
came from a Qwen repair stage; this run removes that attribution error.
Why this is intentionally harsh
The best production workflow may still use several models. A planner can find the right files, a reviewer can spot a missed constraint, and Qwen can repair an otherwise useful candidate.
That is a different benchmark.
This run measures patch ownership. I removed planning diversity, reviewer feedback and model-assisted repair so that a resolved task belongs to the model named in the row. The lower score is worth the cleaner attribution.
What comes next
The next version should use 20–30 stratified tasks and at least three generations for stochastic settings. It should keep Qwen in every batch as the control, retain this heuristic-only retrieval baseline, publish confidence intervals and score planner, reviewer and repair roles separately.
For now the operational decision is straightforward:
Keep Qwen3.6-35B-A3B Q4_K_M as the default local implementer and repairer. Do not promote another model to final patch ownership from this evidence.
That decision can change. The next model has to change it by producing patches that apply and pass, not by looking impressive in its own model card.