# Anush Deokar

> Anush Deokar is a Full Stack + AI Engineer, founding AI engineer at Meridian. RAG pipelines, MCP servers, agents, and small-scale LLM experiments.

I'm a founding AI engineer at Meridian, where I build the core AI platform end to end: RAG pipelines and MCP servers underneath, multi-agent flows in the middle, and the chat and voice agents people actually talk to on top. Before that I spent years as a frontend and full-stack engineer, so I own the whole path, from raw data to the interface.

Outside work I run experiments and build side projects: taking ideas apart, testing them carefully, and writing up what I learn.

## Experience

### Founding AI Engineer, Meridian (Jun 2025 – Present)

Founding engineer on the AI platform behind Meridian's products, designing systems end to end, from ingestion and retrieval to multi-agent flows in production.

- Architected the RAG pipelines at the centre of Hermes, Meridian's core AI platform: ingestion, retrieval and grounding that every product agent builds on.
- Built the Atlas MCP server, a standalone multi-tenant MCP server that exposes lead and lead-journey data to any MCP-compatible agent, with JWT and service-account auth, tenant resolution shared with hermes-core, and role-aware actions.
- Designed and shipped multi-agent flows on Hermes: production chat agents that hold real conversations with customers, plus core contributions to the voice agent.
- Built agents that make decisions, not just replies: judging candidates in AI interviews and qualifying leads in conversation, with results delivered to partner systems in real time.

Stack: Python, TypeScript, LangGraph, MCP, RAG, Node.js

### Software Engineer, Trek Health (Mar 2024 – Present)

- Led core development on turning terabytes of complex healthcare data into accessible, actionable insights for clients.
- Built a responsive, pixel-perfect product UI on top of systems designed for large-scale data operations, through the product's growth from 0 to $2M+ ARR.

Stack: TypeScript, Next.js, Node.js, Snowflake, AWS

### Software Engineer, matrice.ai (Jul 2023 – Apr 2024)

- Led and maintained the main product's React + TypeScript frontend, and backend microservices for image annotation.
- Integrated ML models into the product and coordinated testing with the ML team.

Stack: React, TypeScript, Go, Node.js, AWS

## Education

B.Tech, Computer Science and Engineering, IIT Jodhpur (2023)

## Skills

Python, TypeScript, LangGraph, MCP, RAG, Node.js, Next.js, Snowflake, AWS, React, Go

## Projects

### Tiny Frontier: Cheap memory in small LLMs (current)

How do modern LLMs keep memory cheap? I'm rebuilding sliding-window attention and attention sinks from scratch, testing them with controlled multi-seed experiments, all on a $0 compute budget.

- **1.8e-4**: max logit difference loading Google's Gemma 3 270M into my own code, with identical greedy decoding
- **5.8×**: less KV cache at 4K tokens for a 5:1 local:global model that matches a hybrid beating full attention
- **32 / 32**: buckets where evicting an attention sink hurt, after one change to the training data switched the sink on

Code: https://github.com/AnushDeokar/tiny-frontier  
Write-up: https://www.anushdeokar.xyz/writing  
Stack: PyTorch, From scratch, Multi-seed experiments

### Typeform clone

Form builder with workspaces and publishing, server actions and zod validation. 100+ users.

Stack: Next.js 14, Drizzle, Clerk, Stripe  
Live: https://typeform-clone-three.vercel.app/signin  
Code: https://github.com/AnushDeokar/typeform-clone

### Csgrid

Blogging platform with cover uploads, tags, and search and filtering by tag.

Stack: Next.js, Clerk, Prisma, ReactQuill  
Live: https://csgrid.vercel.app/  
Code: https://github.com/AnushDeokar/Csgrid

## Writing: Tiny Frontier

Rebuilding the memory tricks of modern LLMs at tiny scale, one controlled experiment at a time.

- [Part 1: Shrinking the frontier: rebuilding modern LLM tricks from scratch](https://www.anushdeokar.xyz/writing/tiny-frontier/background.md): The question, the rules for fair comparisons, a 3.4M-parameter control model, and proving the code is right by matching Google's Gemma 3 270M.
- [Part 2: How small can attention windows get?](https://www.anushdeokar.xyz/writing/tiny-frontier/local-global-attention.md): Sliding windows, local:global layer patterns, and what they cost in quality versus what they save in KV-cache memory.
- [Part 3: Attention sinks and streaming](https://www.anushdeokar.xyz/writing/tiny-frontier/attention-sinks.md): Streaming 16× past the training length with a 64-token cache. The predicted collapse didn't happen, and one controlled run showed why: sinks come from the training data format.

## Links

- Website: https://www.anushdeokar.xyz
- GitHub: https://github.com/AnushDeokar
- LinkedIn: https://www.linkedin.com/in/anushdeokar/
- X: https://x.com/AnushDeokar
- Email: deokaranush@gmail.com

---

# Writing (full text)

## Shrinking the frontier: rebuilding modern LLM tricks from scratch

> Modern LLMs use a handful of tricks to keep memory cheap. Papers make them sound obvious. I wanted to see them move, so I'm rebuilding them from scratch at tiny scale, for $0.

[Tiny Frontier](https://www.anushdeokar.xyz/writing) · Part 1 of 3 · by Anush Deokar · September 2026 · 11 min read  
Canonical: https://www.anushdeokar.xyz/writing/tiny-frontier/background  
More about the author: https://www.anushdeokar.xyz/llms.txt

Every model that handles long context has the same problem: the **KV cache**. To generate the next token, a transformer needs the keys and values of every token before it, and that cache grows with every token you add. Gemma 3, Mistral and gpt-oss all ship with tricks to keep it small: attention that only looks back a fixed number of tokens, a mix of cheap "local" and expensive "global" layers, and something called an attention sink.

I could explain each of these from the papers. What I couldn't do was predict what happens when you change one: how much quality you lose for how much memory, where the loss comes from, or when a trick quietly breaks. **Tiny Frontier** is my attempt to find out by building them myself, at a scale where I can run a careful experiment in an evening.

### The story in three sentences

1. **Windows cut memory.** A model that only attends to its last W tokens needs a much smaller KV cache (Part 2).
2. **Windows only break on long streams if the model relies on a sink,** a token it parks spare attention on (Part 3).
3. **Whether a sink forms depends on how the model was trained.** Mine formed one only when every training chunk started with the same token (Part 3).

### The constraints

One Apple M4 machine, PyTorch on Apple's GPU backend, no rented GPUs, and no run longer than about 45 minutes, so nothing has to run overnight. That rules out anything close to real model sizes. It also forces the question I actually care about: does the *direction* of each effect show up when everything else is held fixed?

To make the answer trustworthy, I set a few ground rules before writing any model code:

- **One control model.** Every experiment changes exactly one thing against it. Data, tokenizer, optimiser, width, depth, context, token budget and eval slice stay fixed.
- **Seeds where they matter.** The control gets three (1337, 1338, 1339) to measure the noise. A variant gets extra seeds when its gap to the control is ambiguous (0.01–0.03 nats). Single-seed results are labelled as single-seed.
- **Report the cost with every loss:** parameters, wall-clock, tokens per second, and KV-cache bytes.
- **Every run leaves a folder** with its config, git commit, seed, data hash, loss log, a sample and a summary.
- **Negative results are results.** If something doesn't work, it goes in the write-up.

### The setup

The data is [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories), a dataset of simple children's stories that small models can actually learn. I trained a byte-level BPE tokenizer with a vocabulary of 4,096, pre-tokenised the first 100M training tokens, and fixed a validation slice of 491,520 tokens that every run is scored on.

The control is a Llama-style decoder written from scratch: pre-norm RMSNorm, causal attention with RoPE, a SwiGLU feed-forward, tied embeddings and no biases.

|   | Control model |
| --- | --- |
| Layers / width / heads | 6 / 192 / 6 (head dim 32) |
| Parameters | 3,443,136 (2,656,704 non-embedding) |
| Context | 256 tokens |
| Optimiser | AdamW (0.9, 0.95), weight decay 0.1, clip 1.0 |
| Schedule | 200-step warmup, cosine to 10% of peak |
| Budget | 50M tokens (6,103 steps of 32 × 256) |
| Tests | RoPE, causality, KV-cache equivalence, GQA, regression, … |

#### Tuning the learning rate first

An untuned baseline makes every comparison meaningless, so the first real experiment was a learning-rate sweep: five peak LRs, 10M tokens each.

![Validation loss against peak learning rate, U-shaped with a minimum near 6e-3](https://www.anushdeokar.xyz/writing/tiny-frontier/lr_sweep.png)

*Final validation loss after 10M tokens, one seed per learning rate.*

The curve is a U, and it's lopsided: halving the LR from 6e-3 cost 0.021 nats, doubling it cost 0.138. Too high hurts far more than too low. A parabola through the three best points puts the minimum near 4.7e-3, so I picked `5e-3`, rounding towards the safer side because the real runs are five times longer. The LR moved the loss by up to 0.18, about twenty times the seed noise below. Getting it wrong would have swamped every effect I was trying to measure.

#### Three seeds, and the noise floor

![Validation loss curves for three seeds of the control model, nearly overlapping](https://www.anushdeokar.xyz/writing/tiny-frontier/control_seeds.png)

*The control model, three seeds, 50M tokens each (about 27 minutes per run).*

The three seeds finished at 1.8830, 1.8743 and 1.8821 nats per token: a mean of **1.8798** and a spread of **0.0087**. That spread is the most important number in the project. It's the noise floor: a difference under about 0.01 is indistinguishable from changing the seed, and anything above 0.02 that holds across seeds is a real effect.

The samples read like what you'd hope for:

> Once upon a time, there was a little boy named Tim. Tim loved to play with his toy cars. One day, he found a big tree. The tree was very tall and had many branches that he wanted to touch…

### Proving the code is right: Gemma 3 270M

A tiny model producing nice stories doesn't prove the implementation is correct. So before running any experiments, I loaded Google's real **Gemma 3 270M** weights into my own codebase and compared its outputs to Hugging Face's `transformers`.

Gemma 3 is the same kind of decoder with nine differences. I added each one as a config flag on the control model (not a forked model), guarded by a test that the trained control still gives exactly its recorded loss with every flag off.

![Gemma 3 270M architecture diagram, with the parts that differ from Llama outlined](https://www.anushdeokar.xyz/writing/tiny-frontier/gemma3_architecture.svg)

*Gemma 3 270M. Dashed outlines mark what differs from a Llama-style block: sandwich norms, GeGLU, QK-norm, multi-query attention, and a 5:1 local:global layer pattern.*

| Difference | Control | Gemma 3 270M |
| --- | --- | --- |
| RMSNorm scale | x̂ · w | x̂ · (1 + w) |
| Embeddings | unscaled | × √640 |
| QK-norm | none | RMSNorm per q/k head |
| Norms per block | 2 (pre) | 4 (pre + post) |
| MLP | SwiGLU | GeGLU (tanh GELU) |
| Heads | 6 Q / 6 KV, dim 32 | 4 Q / 1 KV, dim 256 |
| Layer pattern | all global | 5 local : 1 global |
| Window | — | 512 tokens |
| RoPE base | 10k | 10k local · 1M global |

The method was boring on purpose: read the reference source instead of trusting memory, check the parameter count (exactly 268,098,176, Google's published number), match each feature against Hugging Face's own module on random weights, then match a tiny random Gemma end to end, and only then load the real checkpoint.

- **1.8e-4**: max |logit diff| over 262,144 vocab entries
- **100%**: top-1 token agreement at every position
- **5 / 5**: prompts with identical greedy decoding
- **800**: token prompt to exercise the 512 window

> **Result:** Logits matched to about 1e-4 (the target was 1e-3), and greedy generation was token-for-token identical. It matched on the first run with real weights, because every piece had already been matched on random ones.

The remaining difference is fp32 rounding from doing operations in a different order across 18 layers. The 800-token prompt matters most: the short prompts never exceed the 512-token window, so they don't test the sliding window at all.

#### The traps

These are the bugs this process was built to catch, and the ones I'd check first in anyone's port:

- **`(1 + w)` norms.** Gemma stores norm weights near zero. Read with the Llama convention, every activation gets scaled by roughly zero, silently.
- **Same name, different job.** `post_attention_layernorm` is the norm *before* the MLP in Llama and the norm on the attention *output* in Gemma 3. I mapped weights by function, not by name.
- **`head_dim` ≠ width / heads.** Attention runs in 4 × 256 = 1,024 dimensions while the residual stream is 640 wide.
- **The GELU variant.** Exact GELU differs from the tanh approximation by up to 1e-3, enough to break parity.
- **Random-weight tests must randomise the norms too.** At zero-init, `w` and `1 + w` are indistinguishable from a Llama norm.
- **Compare in fp32.** The checkpoint is bf16, whose ~3 significant digits would hide real bugs.

One fun fact from counting: 63% of the "270M" is the embedding table (262,144 × 640). The 18 transformer layers hold about 100M parameters.

### The number to beat

With a correct, tuned control in hand, here is what it costs. Every layer is global, so the KV cache grows linearly with context:

| Context | KV cache (fp32, one sequence) |
| --- | --- |
| 256 tokens | 2.25 MB |
| 4,096 tokens | 36.0 MB |

*2 (K and V) × 6 layers × T tokens × 6 KV heads × 32 dims × 4 bytes.*

That 36 MB is tiny in absolute terms, but it scales the same way it does in a real model. **Part 2** asks how small the attention windows can get before quality suffers, and how much of that 36 MB they save.

---

Caveats, stated once: one dataset, one model size, fp32 on one machine, and three seeds give a rough noise estimate rather than a confidence interval. These are toy versions of the real architectures, not reproductions. The code, configs and every result CSV are on [GitHub](https://github.com/AnushDeokar/tiny-frontier).

---

## How small can attention windows get?

> If each layer only looks at its last W tokens, the KV cache stops growing. How small can W get before the model notices, and what happens if you keep one layer that sees everything?

[Tiny Frontier](https://www.anushdeokar.xyz/writing) · Part 2 of 3 · by Anush Deokar · October 2026 · 10 min read  
Canonical: https://www.anushdeokar.xyz/writing/tiny-frontier/local-global-attention  
More about the author: https://www.anushdeokar.xyz/llms.txt

> **The headline:** A hybrid with two global layers and four 32-token sliding-window layers *beat* full attention by 0.02 nats across 3 seeds. Gemma 3's 5:1 pattern matched it with 5.8× less KV cache at 4,096 tokens (1 seed).

### The idea

In normal (global) attention, token i looks at every token before it. In **sliding-window** (local) attention it only looks at the last W: tokens i − W + 1 through i. A local layer never needs to cache more than W keys and values, however long the sequence gets.

*Diagram: global: sees everything*

*Diagram: local, W = 4: sees the last 4*

Gemma 3 mixes the two: five local layers with a 512-token window for every one global layer. Mistral 7B used windows everywhere. I wanted to know how much quality a window really costs, and how the *pattern* of local and global layers changes that.

### The experiment

Same control model as Part 1 (6 layers, 3.4M parameters, 256-token context, 50M tokens), with one change per variant: a layer pattern string and a window size. `G` is a global layer, `L` a local one, so the control is `GGGGGG`. Everything else, including the learning rate tuned for the control, stays identical. That setup slightly favours the control.

Before running anything I wrote down predictions: W = 64 and 128 would match the control, 16 and 32 would be clearly worse, and a hybrid with two global layers would recover most of the gap.

### Results

| Variant | Seeds | Val loss | vs control | KV @ 4,096 |
| --- | --- | --- | --- | --- |
| `GGGGGG` | 3 | 1.8798 | — | 36.0 MB |
| `LLLLLL W=128` | 1 | 1.8806 | +0.001 | 1.13 MB |
| `LLLLLL W=64` | 3 | 1.8951 | +0.015 | 0.56 MB |
| `LLLLLL W=32` | 1 | 1.9340 | +0.054 | 0.28 MB |
| `LLLLLL W=16` | 1 | 1.9969 | +0.117 | 0.14 MB |
| `LLGLLG W=32` | 3 | 1.8596 | −0.020 | 12.2 MB |
| `LLLLLG W=32` | 1 | 1.8613 | −0.019 | 6.2 MB |

*Lower is better. Seed noise on the control is 0.0087. KV is the fp32 cache for one sequence, measured during generation and equal to the formula to the byte.*

![Scatter of validation loss against KV cache size at 4,096 tokens; the hybrid models sit lower and further left than the control](https://www.anushdeokar.xyz/writing/tiny-frontier/exp1_memory_tradeoff.png)

*Memory against quality. Lower-left is better. The two hybrids sit below the control at a fraction of its memory.*

#### 1. All-local: loss rises steadily as the window shrinks

W = 128 matches the control. W = 64 is slightly but consistently worse: all three seeds landed above all three control seeds. W = 32 and W = 16 are clearly worse, far outside the noise.

![Validation loss against window size for all-local models, falling towards the control band as the window grows; hybrids sit below the band](https://www.anushdeokar.xyz/writing/tiny-frontier/exp1_loss_vs_window.png)

*All-local models approach the control (grey band) as W grows. The hybrids at W = 32 sit below it.*

#### 2. Hybrids beat full attention

This was the surprise. `LLGLLG` at W = 32 didn't just recover the gap, it beat the control: all three hybrid seeds landed below all three control seeds, the worst hybrid beating the best control by 0.011. That's −0.020 nats with 3× less KV memory at 4,096 tokens.

#### 3. One global layer is enough

`LLLLLG`, Gemma 3's 5:1 ratio, landed within 0.002 of the two-global hybrid at half its long-context memory (single seed). Adding the first global layer to an all-local W = 32 model moved the loss by 0.073; the second added about 0.002.

- **−0.020**: nats vs full attention (LLGLLG, 3 seeds)
- **5.8×**: less KV cache at 4K tokens (LLLLLG)
- **0.073**: loss recovered by the first global layer
- **0.002**: recovered by the second

#### 4. With a global layer, the window barely matters for memory

At 4,096 tokens the global layer's cache dominates. `LLLLLG` at W = 16 would need 6.1 MB against 6.2 MB at W = 32. The number of global layers sets the cost. I think this is why Gemma can afford a generous 512-token window: once there's a global layer, a bigger window is nearly free.

### Where windows help, and where they hurt

Averages hide the mechanism, so I split the validation loss by how much context each token had, and compared every variant to the control at each distance.

![Loss difference to the control by position: all-local models are better at short context and worse at long context; hybrids are better everywhere](https://www.anushdeokar.xyz/writing/tiny-frontier/exp1_loss_by_position.png)

*Loss minus control, by tokens of context. Below zero is better than the control.*

- **Local attention has two opposite effects.** Every all-local model is *better* than the control with little context and worse with a lot. Restricting attention to nearby tokens seems to make short-range patterns easier to learn, and it costs whatever lies out of reach.
- **The penalty starts after about 2W, not W.** Six stacked local layers can relay information backwards (a receptive field of 6 · (W − 1)), but relaying is lossy, so the penalty grows with distance.
- **The hybrids keep the short-range advantage everywhere.** The global layer covers what the local layers can't reach, so beyond the first few tokens they're about 0.02 better at every distance.

#### Is the hybrid just learning faster?

A fair suspicion: maybe hybrids only learn faster, and the control would catch up with more training. Tracking the gap through training:

![Gap to the control over 50M training tokens; hybrids start far ahead, narrow, then hold steady near −0.02](https://www.anushdeokar.xyz/writing/tiny-frontier/exp1_gap_over_training.png)

*Loss minus control over training. The hybrids' lead narrows over the first ~20M tokens, then holds.*

Partly, yes. The hybrids lead by 0.06 at 2M tokens, and that shrinks over the first 20M. But for the last 30M tokens it holds steady at −0.020 to −0.021. All-local models start ahead too, then fall behind once the control learns to use long-range context, from about 4M tokens on. At this budget a stable gap remains. Whether it survives much longer training, I haven't tested.

### A detour: generation speed

I also measured tokens per second while generating with a rolling KV cache. All-local models generated about **2.3× faster**, and adding any global layer brought the speed back down to the control's.

| Variant | 256 tokens | 4,096 tokens |
| --- | --- | --- |
| `GGGGGG` | 301 tok/s | 310 tok/s |
| `LLGLLG W=32` | 302 tok/s | 291 tok/s |
| `LLLLLL W=64` | 528 tok/s | 700 tok/s |
| `LLLLLL W=16` | 623 tok/s | 719 tok/s |

*Greedy decoding on the M4 GPU, benchmarked control-last to rule out warm-up effects.*

But the control is as fast at 4,096 tokens as at 256, so the difference isn't attention arithmetic. My hypothesis (untested): a global layer's cache grows by one entry every step, so its tensors change shape every step, and Apple's GPU backend prepares work per tensor shape. A rolling local cache stops changing shape once it's full. Production systems avoid this with a pre-allocated, fixed-size cache. So at this scale the speed difference is an implementation effect. The architectural win here is memory.

### Predictions vs outcome

| Prediction (before running) | Outcome |
| --- | --- |
| W = 64 and 128 ≈ control | 128 ✓ · 64 slightly worse (+0.015) |
| W = 16 and 32 clearly worse | ✓ |
| LLGLLG W=32 recovers most of the gap | Exceeded: it beats the control |
| Risk: TinyStories too local to show anything | Didn't happen |

### What not to conclude

"Hybrids beat full attention" holds for one dataset, 256-token context, 3.4M parameters and 50M tokens, not in general. W = 16, 32, 128 and `LLLLLG` are single-seed; their gaps (except `LLLLLG`'s) are far outside the noise, and `LLLLLG` matches `LLGLLG` but isn't confirmed with extra seeds yet. Training speed isn't reported, because masked attention still computes the full matrix, so windows don't make training faster in this code.

### Next

Windows have a known failure mode. If you stream text past the training length and evict the oldest tokens from the cache, models can collapse, because they've been using the very first tokens as a place to dump spare attention. That's what the papers report. **Part 3** builds a streaming evaluator and tests the fixes: StreamingLLM's kept "sink" tokens, and gpt-oss's learned per-head sink. My model didn't behave the way the papers predict, and working out why turned out to be the interesting part.

---

All numbers come from the run folders and result CSVs in the [repo](https://github.com/AnushDeokar/tiny-frontier). Every run records its config, git commit and data hashes.

---

## Attention sinks and streaming

> A model trained on 256 tokens has to stream far past that with a small cache. StreamingLLM says evicting the first tokens breaks it. Mine didn't break, and finding out why was the most interesting result of the project.

[Tiny Frontier](https://www.anushdeokar.xyz/writing) · Part 3 of 3 · by Anush Deokar · October 2026 · 10 min read  
Canonical: https://www.anushdeokar.xyz/writing/tiny-frontier/attention-sinks  
More about the author: https://www.anushdeokar.xyz/llms.txt

> **The headline:** Attention sinks come from the training data, not from attention itself. My model formed none until every training chunk started with the same token. Then it formed one immediately, and evicting it from the cache cost +0.035 nats in 32 of 32 buckets.

### The problem: streaming past the training length

Part 2 was about shrinking the KV cache inside the training length. This part is about generating *past* it. A chatbot or an agent keeps producing tokens long after 256, so something has to happen to the cache. There are two obvious options:

- **Keep everything** (a full cache). Memory grows with every token, and the model meets positions it never saw in training.
- **Keep only the last W tokens** (a rolling cache). Memory is fixed, and the kept tokens are renumbered 0 to W − 1, so positions never go out of range.

The rolling cache sounds like the clear winner. [StreamingLLM](https://arxiv.org/abs/2309.17453) (Xiao et al., 2023) showed it isn't: once the first few tokens leave the cache, large models fall apart. Their explanation is the **attention sink**.

Softmax forces every head's attention weights to sum to 1, even when nothing in the context is relevant. The model needs somewhere to put that spare attention, and the first token is a natural choice: every later token can see it. So models learn to park attention there. Evict it, and that attention gets spread over tokens that do matter, which scrambles everything downstream.

Two fixes exist, and I built both:

- **Keep the sink tokens** (StreamingLLM): always keep the first 4 tokens, plus the last W.
- **Train in a sink** (gpt-oss): give each head one extra learned logit in the softmax with no value vector behind it. Weight that goes there goes nowhere, so a head can attend to "nothing".

### The experiment

I took 32 streams of 4,096 validation tokens, 16× the training length, and fed them through the model one token at a time, scoring the prediction at every position. The model and the text stay the same in every case; only the cache changes.

| Case | Model | Cache |
| --- | --- | --- |
| a | control (3 seeds) | full: keep every token |
| b | control (3 seeds) | rolling: last 64 tokens |
| c | control (3 seeds) | sink: first 4 + last 64 |
| d | control + learned sink (1 seed) | rolling: last 64 |
| e | all-local W = 64 from Part 2 (3 seeds) | rolling: last 64 |

The learned-sink model is the control with one change, trained the same way. It scored 1.8845 against 1.8830 for the matching control seed, so the sink costs nothing in quality.

My predictions, written before running: (a) breaks past 256 tokens, (b) collapses once the first tokens are evicted, (c) and (d) fix it, and (e) does best, because it never learned to look further back than 64 tokens.

### Result 1: only the full cache breaks

![Loss along a 4,096-token stream: all caches match up to 256 tokens, then the full cache climbs to about 4.7 while the rolling caches stay flat near 1.7](https://www.anushdeokar.xyz/writing/tiny-frontier/exp2_post_stream.png)

*Loss at each position of the stream (32-token moving average). The full cache breaks right after the training length; the rolling caches stay flat to 4,096.*

Up to position 256 every case is the same. After that, the full cache climbs from about 1.6 to 4.7, which is worse than the 4.0 the model scores with no context at all. Memory isn't the problem here, positions are: past 256 the model meets rotary position angles it was never trained on. Every 64-token cache stays flat at about 1.74 out to 4,096 tokens.

### Result 2: the collapse didn't happen

| Case | Mean loss, positions 256–4,095 |
| --- | --- |
| a full cache | 4.503 |
| b rolling(64) | 1.738 |
| c sink(4) + rolling(64) | 1.742 |
| d learned sink, rolling(64) | 1.744 |
| e trained with W = 64, rolling(64) | 1.718 |

*Lower is better. b, c and d are within 0.006 of each other, below the 0.01 noise floor.*

Case b didn't collapse. Keeping the first four tokens (c) made no difference, and neither did the learned sink (d). Prediction (b) was simply wrong for this model. So either my cache was broken, or the model never had a sink to lose.

The cache isn't the issue: it's unit-tested, and the same harness caught case a breaking. So I measured where the attention actually goes.

### Why: there was no sink to lose

For every layer and head, I measured how much attention token 0 gets, and divided it by what it would get if attention were spread evenly. 1× means token 0 is treated like any other token, above 1× means it's a sink, and below 1× means it's being ignored.

![Three heatmaps over 6 layers and 6 heads: the control's attention on token 0 is all blue, below even; the BOS-trained model is deep red, up to 33 times even; the learned-sink model sends 20 to 91 percent of layer 0's attention to nothing](https://www.anushdeokar.xyz/writing/tiny-frontier/exp2_post_attention.png)

*Left: the control. Middle: the same model trained with <s> first (next section). Right: how much attention the learned-sink model sends to 'nothing'. Queries at positions 16–255 of normal 256-token chunks.*

Look at the left panel first. Across all 108 heads (6 layers × 6 heads × 3 seeds), token 0 never gets more than an even share. Most layers give it a third of that or less. There is no sink, so evicting token 0 takes away nothing the model relies on.

But the right panel shows the model *does* have spare attention to dump. Given a learned sink, layer 0 sends 20–91% of its attention to "nothing", and later layers send 11–55%. The need for a sink is real. The control just never chose token 0 as the place to put it.

My guess was the training data. I train on random 256-token slices of one long stream of stories, so a chunk starts mid-sentence and position 0 holds a different token every time: "the", a name, a full stop. Llama-style models instead put the same token, `<s>` (beginning of sequence), at position 0 of every training sequence. A sink needs to be the same thing every time, and only that setup gives the model one.

### The test: one more training run

I trained the control again with exactly one change: `<s>` at the start of every training chunk. It reached 1.8835, the same as the control, so the change costs nothing on its own. Then I streamed it both ways: rolling(64), which evicts `<s>` at position 64, and sink(4) + rolling(64), which keeps it.

**A sink formed immediately.** In the middle panel above, every layer-0 head puts 19–33× an even share on `<s>`, and later layers go up to 17.5×. Same model, same data, same budget: the only difference is a fixed token at position 0.

**And now evicting it hurts.** Because the two streams use the same model and the same text, I can subtract them bucket by bucket. Seed noise and text noise cancel out, leaving only the cost of evicting the first tokens:

![Extra loss from evicting the first tokens, per 128-token bucket: about zero for the control, about +0.035 in every bucket for the BOS-trained model](https://www.anushdeokar.xyz/writing/tiny-frontier/exp2_post_eviction_cost.png)

*Loss with a rolling cache minus loss with the first 4 tokens kept, for the same model on the same 32 streams.*

- **33×**: an even share of attention on <s> (layer 0)
- **+0.035**: nats from evicting it (vs −0.003 without <s>)
- **32 / 32**: buckets where evicting it hurt
- **64**: the exact position the gap starts

The timing is the cleanest evidence. The two runs are identical for positions 1–63 (1.974 each) and split at position 64, the moment `<s>` leaves the cache (1.630 against 1.605 over the next 64 tokens). Keeping it, StreamingLLM-style, puts the model level with the control again (1.733 against 1.742).

> **What this shows:** In this setup, whether a model develops an attention sink depends on how it was trained. Without a fixed first token, a rolling cache is safe. With one, you have to keep it.

### Why 0.035, not an explosion

StreamingLLM's models blow up to perplexities in the hundreds or thousands without their sink tokens. Mine lose about 3.5% perplexity. I have two plausible reasons, neither tested. First, my sink is moderate: `<s>` takes 22–38% of layer 0's attention and much less elsewhere. Second, the model is tiny (3.4M parameters) and trained briefly (50M tokens). Whether the dependence on the sink grows with model size is the next question I'd test.

### What about the learned sink?

At this scale it doesn't change the loss: d matches b and c, and its training loss matches the control. It does get used heavily, and it has one visible effect. In the first ~10 tokens of a stream its loss is clearly lower (3.5 against 3.75 at position 1). With almost no context, attending to nothing beats spreading weight over junk.

### So which model streams best?

The all-local W = 64 model from Part 2 (e), by about 0.02 nats, consistently across seeds. It never learned to depend on tokens more than 64 back, so a 64-token cache removes nothing it uses. It also generated up to 2.3× faster in Part 2's benchmark, with a quarter of the cache. The model that was best inside the training length isn't the one you want for long streams.

### Predictions vs outcome

| Prediction (before running) | Outcome |
| --- | --- |
| (a) full cache breaks past 256 | ✓, badly: 1.6 → 4.7 |
| (b) rolling cache collapses | ✗ for the control (no sink formed) · ✓ mildly once training started with `<s>` |
| (c) keeping 4 sink tokens fixes b | Nothing to fix for the control · fully fixes the BOS model |
| (d) learned sink fixes b | ≈ b and c |
| (e) trained-with-window model is best | ✓ |

### What not to conclude

This is one dataset and one model size. TinyStories stories are about 200 tokens long, so a 4,096-token stream is about 20 stories in a row. That tests whether the model stays stable, not whether it uses long-range context. The BOS and learned-sink models are single-seed. The BOS result rests on a paired comparison that was positive in 32 of 32 buckets, plus a 19–33× effect in the attention data, but a second seed would make it stronger. I also measured attention on normal 256-token chunks, not inside the streams. And "sinks come from the data format" is shown for a 3.4M-parameter model; StreamingLLM saw sinks in larger models trained without a dedicated sink token, so scale probably matters too.

### Where this leaves the series

Across three parts: a from-scratch model that matches Gemma 3 to 1e-4; a 5:1 local:global pattern that matches a 3-seed hybrid which beats full attention, at 5.8× less KV cache; and an attention sink I could switch on and off by changing one token in the training data. The open question is scale: at what model size does evicting a sink go from a 3.5% cost to a collapse? That's the experiment I'd run next.

---

All numbers come from the run folders and result CSVs in the [repo](https://github.com/AnushDeokar/tiny-frontier). The streaming evaluator is `scripts/exp2_stream.py`.