Writing
Notes from building things
Experiment write-ups, mostly about how LLMs work under the hood. Numbers come from runs I can reproduce; caveats are stated.
Series
Tiny Frontier
Rebuilding the memory tricks of modern LLMs at tiny scale, one controlled experiment at a time.
- Part 1
Shrinking the frontier: rebuilding modern LLM tricks from scratch
The question, the rules for fair comparisons, a 3.4M-parameter control model, and proving the code is right by matching Google's Gemma 3 270M.
September 2026 · 11 min read
- Part 2
How small can attention windows get?
Sliding windows, local:global layer patterns, and what they cost in quality versus what they save in KV-cache memory.
October 2026 · 10 min read
- Part 3
Attention sinks and streaming
Why a windowed model breaks when you stream past its training length, and whether attention sinks fix it.
Coming soon
- Part 4
Sinks across scale
When sinks emerge during training, whether that depends on model size, and whether a learned sink allows a smaller window.
Coming soon