A one-layer EAGLE-3 head reads Qwen3-1.7B's own hidden states and guesses up to eight tokens, as many as it's sure of, and Qwen3-1.7B checks them all in one forward pass. Guesses up to the first miss are kept, and the big model adds one token of its own. With greedy decoding the text is exactly what the big model writes alone, just sooner. Every day Figure 1 writes up a top Hacker News story that way on GitHub's CPUs and races it against the big model writing alone, and the button below runs one on your GPU with two guesses a pass.
Each of these runs on this GPU. The number on a prompt is how many times as fast as alone it came out on an RTX 4070 laptop GPU, where the model alone writes about 160 tokens a second. How much the guesses help depends on how easy the answer is to guess, and a head that has cost more time than it saved after 24 passes is switched off for the rest of the answer.
IBM Granite 3.0 1B-A400M has 32 experts in each of its 24 layers, and a small router sends every token to 8 of them. Figure 2 shows the router's real decisions from one forward pass over each sentence. Pick a sentence, then hover or tap its tokens.
Downloads 1.5 GB once, then your browser keeps it. The 8-bit copy picks the same 8 experts as the full model at 94.8% of token and layer pairs.
Qwen3-0.6B ships its weights in bf16, and every token it writes reads all of them from memory. Storing them in 8 or 4 bits makes each token cheaper, and the price is rounding. Figure 3 snaps one real row of weights to a grid, then shows what each format does to the text the whole model writes.
A group of weights shares one grid, stretched from its smallest weight to its largest. One outlier stretches the grid for its whole group, so the rest fall onto a few levels near zero. Smaller groups keep that damage local, at the cost of more scales to store.
| Whole model in | Bits a weight | Picks bf16's next token | KL from bf16, nats | Tokens before it splits |
|---|
A model only commits to a token at its last layer, but every layer's output can be read the same way, through the final norm and the output head. Figure 4 does that for each of Qwen3-0.6B's 28 layers at every position of a prompt. This is the logit lens. Hover or tap a cell to see its five likeliest tokens.
For the French prompt, the layers just before the end guess 犬, the Chinese and Japanese word for dog, before the last layer settles on "ch" for chien. Studies of other multilingual models have seen the same thing, with the middle layers working in the language they saw most in training.