How small can attention windows get?

The idea

In normal (global) attention, token i looks at every token before it. In sliding-window (local) attention it only looks at the last W: tokens i − W + 1 through i. A local layer never needs to cache more than W keys and values, however long the sequence gets.

Gemma 3 mixes the two: five local layers with a 512-token window for every one global layer. Mistral 7B used windows everywhere. I wanted to know how much quality a window really costs, and how the pattern of local and global layers changes that.

The experiment

Same control model as Part 1 (6 layers, 3.4M parameters, 256-token context, 50M tokens), with one change per variant: a layer pattern string and a window size. G is a global layer, L a local one, so the control is GGGGGG. Everything else, including the learning rate tuned for the control, stays identical. That setup slightly favours the control.

Before running anything I wrote down predictions: W = 64 and 128 would match the control, 16 and 32 would be clearly worse, and a hybrid with two global layers would recover most of the gap.

Results

1. All-local: loss rises steadily as the window shrinks

W = 128 matches the control. W = 64 is slightly but consistently worse: all three seeds landed above all three control seeds. W = 32 and W = 16 are clearly worse, far outside the noise.

2. Hybrids beat full attention

This was the surprise. LLGLLG at W = 32 didn't just recover the gap, it beat the control: all three hybrid seeds landed below all three control seeds, the worst hybrid beating the best control by 0.011. That's −0.020 nats with 3× less KV memory at 4,096 tokens.

3. One global layer is enough

LLLLLG, Gemma 3's 5:1 ratio, landed within 0.002 of the two-global hybrid at half its long-context memory (single seed). Adding the first global layer to an all-local W = 32 model moved the loss by 0.073; the second added about 0.002.

−0.020

nats vs full attention (LLGLLG, 3 seeds)

5.8×

less KV cache at 4K tokens (LLLLLG)

0.073

loss recovered by the first global layer

0.002

recovered by the second

4. With a global layer, the window barely matters for memory

At 4,096 tokens the global layer's cache dominates. LLLLLG at W = 16 would need 6.1 MB against 6.2 MB at W = 32. The number of global layers sets the cost. I think this is why Gemma can afford a generous 512-token window: once there's a global layer, a bigger window is nearly free.

Where windows help, and where they hurt

Averages hide the mechanism, so I split the validation loss by how much context each token had, and compared every variant to the control at each distance.

• Local attention has two opposite effects. Every all-local model is better than the control with little context and worse with a lot. Restricting attention to nearby tokens seems to make short-range patterns easier to learn, and it costs whatever lies out of reach.
• The penalty starts after about 2W, not W. Six stacked local layers can relay information backwards (a receptive field of 6 · (W − 1)), but relaying is lossy, so the penalty grows with distance.
• The hybrids keep the short-range advantage everywhere. The global layer covers what the local layers can't reach, so they're about 0.02 better at every distance.

Is the hybrid just learning faster?

A fair suspicion: maybe hybrids only learn faster, and the control would catch up with more training. Tracking the gap through training:

Partly, yes. The hybrids lead by 0.06 at 2M tokens, and that shrinks over the first 20M. But for the last 30M tokens it holds steady at −0.020 to −0.021. All-local models start ahead too, then fall behind once the control learns to use long-range context, from about 4M tokens on. At this budget a stable gap remains. Whether it survives much longer training, I haven't tested.

A detour: generation speed

I also measured tokens per second while generating with a rolling KV cache. All-local models generated about 2.3× faster, and adding any global layer brought the speed back down to the control's.

But the control is as fast at 4,096 tokens as at 256, so the difference isn't attention arithmetic. My hypothesis (untested): a global layer's cache grows by one entry every step, so its tensors change shape every step, and Apple's GPU backend prepares work per tensor shape. A rolling local cache stops changing shape once it's full. Production systems avoid this with a pre-allocated, fixed-size cache. So at this scale the speed difference is an implementation effect. The architectural win here is memory.

Predictions vs outcome

What not to conclude

"Hybrids beat full attention" holds for one dataset, 256-token context, 3.4M parameters and 50M tokens, not in general. W = 16, 32, 128 and LLLLLG are single-seed; their gaps (except LLLLLG's) are far outside the noise, and LLLLLG matches LLGLLG but isn't confirmed with extra seeds yet. Training speed isn't reported, because masked attention still computes the full matrix, so windows don't make training faster in this code.

Next

Windows have a known failure mode. If you stream text past the training length and evict the oldest tokens from the cache, models can collapse, because they've been using the very first tokens as a place to dump spare attention. Part 3 builds a streaming evaluator and tests the fixes: StreamingLLM's kept "sink" tokens, and gpt-oss's learned per-head sink.

All numbers come from the run folders and result CSVs in the repo. Every run records its config, git commit and data hashes.