Every 256K-capable local LLM on my Mac Studio M3 Ultra, ranked. The winner decodes at 60.45 tokens/sec.

Architecture beats parameter count, decisively

The top three are all mixture-of-experts with linear or hybrid attention. A 30B-A3B beats an 80B-A3B by 2.3x on decode while using roughly half the memory (38.7 GB against 71.1 GB), and a 4B sits between them. Meanwhile the dense 27B, running the same weights on the fastest engine available to it, decodes at 7.89.

The reason is that dense full attention pays O(n squared) per layer, and n here is a quarter of a million. The hybrid and linear-attention models only pay that on a minority of their layers. You can watch the term appear directly: the same 7B model at 262,268 tokens prefills at 191.6 tok/s and at 955,201 tokens prefills at 50.2, so 3.64x more tokens cost 13.9x more time.

The engine comparison lands in a similar place. On the one model where I ran all four engines head to head on an identical prompt, oMLX has the fastest prefill and the slowest decode, mlx-lm decodes 1.49x faster than llama.cpp, and all four score 8/8. Recall at this depth is engine-independent. Speed is not. I have since taken that 1.49x apart, and it is a single run per arm that only just clears my own threshold for trusting one.

Why the winner wins: it degrades slowly

The table above says rank 1 wins. It does not say why, and the answer turned out not to be speed. I ran the winner and a dense Qwen2.5-14B-Instruct-1M across four depths on the same needles, and the gap between them grows by 5.8x as the context does.

Across that range the winner's decode falls 1.83x. The 14B's falls 10.68x. The mechanism is visible in the allocator: the winner grows from 37.00 to 38.67 GB, a factor of 1.045, while the 14B grows from 16.69 to 69.67 GB, a factor of 4.17. At a quarter of a million tokens a model is mostly its KV cache, and this one barely has one.

So the honest statement of the result is not that rank 1 is fast. A 2B model beats it at 2k. It is that rank 1 is flat, and flatness is what a long context actually rewards. That also means the ranking on this page is a ranking at one depth, and would reorder at a shorter one.

The recall gate never excluded a model for failing to retrieve

This is the result I would most like other people to steal, because it says a common benchmark practice is broken.

I published five recall scores below 8/8 and treated each as a model property. Every one was wrong, and every one came back 8/8 when re-run. Qwen3.6-35B-A3B went 2/8 to 8/8. Qwopus3.5-4B went 1/8 to 8/8. Qwen3.5-2B went 1/8 to 8/8. Qwen3.6-27B went 2/8, then 4/8, then 8/8. In each case exactly one thing changed: the generation budget went from 300 tokens to 3,000 or 6,000.

The mechanism is that these are reasoning models. Given a short budget they are still inside their scratchpad when the cap hits, so the scorer reads an empty answer and records a retrieval failure. I was measuring my own harness. My first explanation was that the chat template triggered it, and that was also wrong: Qwopus, handed a raw file with no template at all, emitted a think tag as its very first token. Reasoning mode is a property of the model, not of the prompt format.

# The rule that decides whether a recall score means anything.
# It keys on the GENERATION, not the prompt: models enter reasoning
# mode unprompted, so a raw untemplated prompt is not a defence.
think_open   = "<think>"  in generation
think_closed = "</think>" in generation
hit_cap      = generated_tokens >= max_gen

if think_open and not think_closed and hit_cap:
    verdict = "NOT REPORTABLE"   # you measured the budget, not the model

Four models do genuinely fail, and here is the useful distinction: all four fail by degenerating, not by missing the needles. Falcon-H1-7B emits token soup. Base Qwen3.8-27B retrieves perfectly and then cannot stop, emitting 74,278 characters of verbatim haystack without ever closing its think block, even at a 16,000-token budget, while answering cleanly at 2k, 8k and 32k. Qwen2.5-7B-Instruct-1M repeats the filler sentence to the cap. GLM-5.3-Flash, described below, is the fourth and fails the same way as the base 27B. Retrieval has not once been the binding constraint at this depth.

One intervention repaired a genuine failure, which nothing else in this project managed. The abliterated build of Qwen3.8-27B, same base revision and quantization, stops on its own after 533 tokens with a closed think block and scores 8/8, using 3% of the budget the base weights exhausted. Abliteration is a refusal-direction weight edit aimed at censorship, and it happened to fix termination. I would not have predicted the direction.

"1M-native" is a claim about training, not about behaviour

Qwen2.5-7B-Instruct-1M advertises a trained context of 1,010,000 tokens. I built a 955,201-token needle prompt, deliberately inside that ceiling, with the needles at the same proportional depths as the 262k one so the two tests measure the same thing.

The machine is not the limitation. It ingested all 955,201 tokens with the truncation guard passing, in 5.29 hours of prefill, peaking at 151.8 GiB with zero swap. A 512GB M3 Ultra can hold and process a million-token context.

The model is the limitation, and given my own history of budget artifacts the distinguishing check matters: it never opened a think block, so there was no scratchpad to truncate. The output is verbatim repetition of the haystack filler to the cap. The model works perfectly at a quarter of its advertised context and degenerates at 95% of it, with no warning in between and nothing in its configuration marking the boundary. The only way to know a model's usable context is to measure it at the depth you intend to use.

Three acceleration paths, one base model, one winner

The last stretch tested three published accelerations against the same base model at the same depth, so they are comparable to each other and to one baseline rather than to marketing.

DFlash2 is the only one that survives to this depth, and it beats its own published claim of 2.7 to 3.4x, which was measured on an M5 Max at short context. I had predicted the opposite: I expected speculative decoding to shrink at 262k because every forward pass is bound by reading the KV cache. It helps precisely because that KV read is amortized across several accepted tokens instead of repeated per token. The cost is a 36% prefill regression, because the 3.6 GB drafter pushes peak memory from 115.7 to 133.4 GiB and that drops the sustainable clock from 957 to 732 MHz. The trade pays back after roughly 4,100 generated tokens.

The dual-ANE recipe delivers nothing, and I could have known before measuring. Its kernel is fixed-shape at exactly 2,048 tokens and accelerates MLP layers rather than attention, and the recipe's own published results are all at a prompt length of 2,048. At 262k, attention dominates and the shape does not match, so the Neural Engine drew 0.00 W across all 636 telemetry samples in both arms, and the baseline arm was faster than the accelerated one.

The 2B beats the winner, and it is still not the pick

Qwen3.5-2B tops both charts on raw numbers: 2026.6 tok/s prefill, 65.31 tok/s decode, 8/8 recall, and it is the one model small enough to finish before the firmware limiter engages. If you rank by this benchmark alone, it wins.

I am not shipping it, and I want to be precise about why, because one of my two reasons is weaker than it first looked. The reason I originally gave was that its 8/8 is on the 234,158-token raw prompt while the winner's is on the harder 262,223-token templated one. That argument is weak, because second place is measured at 234,158 too, and I kept it on the board.

The reason that actually holds is about the metric. Needle-in-a-haystack measures retrieval, not comprehension. Copying eight literal 8-digit codes out of filler text is the weakest long-context task there is, and a 2B scoring 8/8 on it tells you nothing about whether it can reason over a quarter-million tokens. That is a reason to distrust this benchmark as a sole metric, which is a conclusion about my measurement rather than about the model. The honest statement is that the 2B wins the test I ran, and the test is not sufficient.

The other flagged row is my own engine, surge, a C and Metal inference engine I have been building. Its 0.537 tok/s is real but stale: it was measured before a decode-attention rewrite that is 28.3x faster on this exact shape in isolation. That does not license moving it up the table, because attention is one term of a decode step and nobody has measured the end-to-end result. Re-measuring costs about 31 hours of GPU time, roughly 99% of it in a prefill path that work did not touch, so the row stays as measured and labelled.

The largest model I have run here, and why it is not ranked

GLM-5.3-Flash is 320B total and 18B active across 45 layers, 181.9 GB of weights. It is by a wide margin the largest thing this machine has ever run: 270.11 GB of allocator peak against a 366.7 GiB system peak, on a box with 512 GB. The prefill alone took 92 minutes.

It scored 8/8 retrieval and 8/8 association at 234,143 tokens. That is genuine perfect recall, and on the hardest configuration here. It is also not on the leaderboard, because it never terminated: 6001 tokens generated against a 6000-token cap, a unique-word ratio of 0.15 where clean prose is 1.0, and a tail visibly looping back over codes it had already found. That is the base Qwen3.8-27B failure exactly, so it gets the same treatment.

Worth flagging that my own guard did not catch this. The automated inconclusive check read False, purely because there is no unterminated think tag in this model's output to key on. It keyed on the marker instead of the artifact, which is the same mistake described in the retractions below, surfacing in a different guard.

Two caveats on its numbers. Its 42.3 tok/s prefill is limiter-shaped and must not be compared with clean rows: median 504 MHz with 11% of samples at or under 400, touching the 338 MHz floor, which is what a 366.7 GiB footprint predicts given the clock-versus-footprint bound above. And its correctness rests on a third-party fork of the runtime whose claim of 1e-6 parity against transformers is the author's, tested by the author, not verified here. Its depth reads 234,143 rather than the 234,158 elsewhere in the table because it is the same prompt file measured with a different tokenizer.

The protocol, and why it is this paranoid

A firmware power limiter on this machine clamps the GPU to 338 MHz under sustained load and understates a run by 2x to 5x. Every row above had to clear the same gate.

# What every published row here had to clear, in order.
pgrep bench                     # never contend for the GPU
sudo fanpro mode curve
sleep 8                         # SET_MODE race: without this, the next line is dropped
sudo fanpro set all 3625
sudo fanpro fans                # VERIFY 3400-3760 rpm; fanpro can latch monitor-only
sleep 120                       # let the limiter release BEFORE gating (see below)
gemm_gate                       # > 20.5 TFLOPS, or the GPU is still clamped
# ... run ...
# decode by least-squares slope, not wall-clock division
# truncation guard: built == saw, truncated == False
# macmon: median MHz AND percent of samples <= 400 MHz

The order in that snippet is load-bearing, and I had it backwards for the whole project. I wrote it as "gate, then fans" and only caught it when a run aborted twice, reporting 19.55 and 18.23 TFLOPS on an idle machine that I had hand-probed at 21.35 a minute earlier. Gating with the fans on auto measures a partially clamped GPU, for the reason in the next paragraph, so the gate ends up answering "are the fans spun up" rather than "is the GPU fresh". With the fans pinned and 120 seconds of idle first, the same probe reads 21.80 and the run proceeds. The dangerous part is that the abort is indistinguishable from a genuinely busy machine, so an automated loop retries forever. It can only cause a false failure, never a false pass, so no number above is affected.

Two findings from this project change how the machine has to be used. First, the limiter tracks fan speed, not die temperature: I quietened the fans mid-run to check, and the GPU went from 0% clamped at 60.6 C to 77% clamped at 58.3 C. It clamped while getting cooler, and restoring the fans released it. This machine cannot be quiet and produce valid measurements at the same time.

Second, the achievable clock is bounded by the whole machine's memory footprint, not by cooling. Runs under about 93 GiB reached about 1377 MHz; runs over about 114 GiB never exceeded 1153, across cooldowns from 0 to 300 seconds. A deliberate 300-second cooldown produced one model's slowest run. This makes cross-model clock-matching impossible, and it means the winner's small footprint is part of why the machine runs fast on it. It also means a resident model server ruins your benchmark: an idle server holding 31.7 GB moved one run from 1379 MHz to a hard clamp.

Ten retractions, and the single pattern behind them

I withdrew ten published claims during this work. Listing them is more useful than any single number above, because every one came from attributing a difference to a mechanism without isolating the variable.

"llama.cpp is the fastest engine" was written while the comparison row was limiter-shaped. "surge is 2.15x slower end to end" was GPU drift, and I should have noticed that prefill fell in the same pair while prefill was untouched by the change. "The recall collapse is real" was the generation-budget artifact, five times over. "The decode column mixes metrics, which explains the gaps" was refuted by measuring both metrics from one generation: they agree to 1%, and the gaps were ordinary run-to-run variance. "Nemotron wins on every axis" was false; it loses the memory axis on both bases.

Two more came after this post first went up, and both are the same shape. I published that "three of five models are slower on prose than on code" and withdrew it when two of the three differences turned out to be run-to-run variance rather than a property of the prompt. And I published a 1.52x coding-quality lead for the winner over a 4B coder, then retracted the headline entirely: it largely measured who ran out of generation budget, because 64% of the 4B's completions produced no extractable code against the winner's 24%. Among completions that did produce code the ranking inverts. Conditioning on that is biased too, so this project currently has no defensible verdict on coding at all.

Two of the ten were caught by arithmetic rather than by measurement. I published a correlation as differing between two memory unit bases when neither variable in it was the memory figure, which is impossible on its face. And I introduced a duplicate finding number that silently broke every cross-reference after it. Both were found by re-reading, not by re-running.

The real noise scale on this machine, for anyone calibrating their own: about 1% from the choice of metric, about 12% from re-running an identical configuration, and up to 1.5x when the clock differs. Prefill reproduces to 0.08%. Any comparison tighter than 12% needs repeated runs, and I have the receipts for what happens when it does not get them.

Caveats, honestly

This is one machine, a Mac Studio M3 Ultra with 512GB and 80 GPU cores, and one benchmark family. Needle-in-a-haystack is a retrieval test, so nothing here measures whether any of these models can reason over long context; the section on the 2B is the honest consequence of that.

Nothing here ranks coding ability. I did measure it, on 182 LiveCodeBench problems, and then retracted the headline: the run was confounded by generation budget, and among the completions that produced any code at all the ranking reversed. A re-run at an adequate budget is the only thing that would settle it, and the two attempts at that so far have not been servable on this machine. So the honest position is that coding is measured and undecided, not that it favours the winner.

The depth column is not uniform: rows read 234,158 or about 262,2xx depending on whether the prompt went through a chat template, and the difference is real though small. Two rows carry specific defects that I have marked rather than smoothed: the abliterated 27B's prefill is clock-contaminated and not reportable, and the 955,201-token run had its fans reduced mid-run, so its speed figures are not fully protocol-compliant while its recall verdict is unaffected. Run counts differ by row, from one to seven, and only the top two have enough repeats to quote a variance. The oMLX rows have an indirect truncation guard: those runs could not prove full ingestion directly, so it was inferred from recall, wall time and peak memory. I have since closed that on two of those rows, and the result is worth stating because it corrects me rather than the software. Both return 234,158 of 234,158 tokens served, not truncated, so the inference was right. The cause of the gap was mine: OpenAI-style streaming omits the token-usage block unless the client asks for it, and my client never did. I re-ran the 35B-A3B row second on purpose, because it is the only one carrying a superlative (the 696.7 tok/s fastest prefill above), where a short-served prompt would have collapsed the claim rather than nudged it. It came back at full clock and 7% faster, 745.7 tok/s, which is inside this machine's run-to-run noise, so the table keeps the published figure. Three rows keep the indirect label; none owns a superlative. And the median clock is missing for two of them because I had not yet started recording it.

Data provenance

Every figure comes from runs on my own machine between 10 and 27 August 2026, each with its own log directory, per-token timestamps and a per-second macmon telemetry trace. One exception to the "my own machine, my own stack" rule is flagged where it appears: the GLM-5.3-Flash row runs on a third-party fork of the runtime, so its correctness rests on the fork author's parity testing rather than on mine. Decode is a least-squares slope over per-token timestamps rather than wall-clock division; both metrics were measured from the same generation on three models and agree to within 1.2%. Prefill and recall come from the runs' own JSON output. Recall is scored against the built prompt's own needle record, never against a source file, after that mistake produced a spurious score once. The generation-truncation guard keys on the model's output, not on the prompt. Numbers here are transcribed from a single reconciled summary table in my working notes rather than re-derived per row, because re-deriving is how several of the retractions above happened.