Every transformer model has a hard limit on how much it can hold in mind at once. This isn't a bug to be patched — it's a fundamental architectural constraint with profound implications for what AI can and cannot do.

Ask a language model to summarize a 500-page book. It will struggle — not because it lacks reading ability, but because it cannot hold the entire book in its attention at once. Give it a codebase of 200,000 lines and ask it to refactor a module that depends on patterns scattered throughout. It will miss things — not from carelessness, but because much of the codebase lies beyond the edge of what it can see.

This constraint has a name: the context window. It is the single most consequential architectural limitation in current AI systems — and arguably the thing that separates today's impressive but bounded tools from the kind of AI that could genuinely act as a persistent collaborator across long projects and complex domains.

What a Context Window Actually Is

When a transformer processes text, it computes attention weights between every token and every other token in the sequence. This is what gives the model its remarkable ability to track long-range dependencies — a pronoun 200 tokens later referring back to a noun introduced 200 tokens earlier, for instance. But this all-pairs attention computation has a cost that scales quadratically with sequence length.

Double the context window, and the attention computation doesn't double — it quadruples. This is why early transformers had context windows of 512 or 1,024 tokens, why GPT-3 operated at 2,048, and why even today's 128,000-token or 1-million-token context windows represent enormous engineering achievements that consume orders of magnitude more compute than the same model running on shorter sequences.

2K
Tokens in GPT-3's original context window — roughly 1,500 words, or about 6 pages of text
1M
Token context window of Gemini 1.5 Pro — enough to hold roughly 750,000 words, or a stack of novels
O(n²)
Computational complexity of standard attention — the fundamental reason context extension is expensive
Partner · Affiliate

Your product name here

One or two sentences on what this product does and why it fits a reader of this journal. Keep the tone of the surrounding writing — it reads better and converts better.

Learn more →

What Gets Lost

The context window isn't just a technical limitation — it's a cognitive one. Anything outside the window is simply not available to the model. It doesn't fade or blur; it vanishes entirely. This makes the behavior of AI systems near their context boundary particularly interesting to study.

Research has documented what's become known as the “lost in the middle” phenomenon: models disproportionately attend to information at the beginning and end of long contexts, while information buried in the middle is systematically under-weighted. The implication is troubling. Even within the stated context window, not all tokens are equal. The edges of memory are both the boundary and the sweet spot.

The context window is not working memory. It's closer to a spotlight. And the spotlight has a hard edge.

— Attributed to a researcher at NeurIPS 2025, paraphrased

The Human Memory Analogy

Human memory operates very differently. We have multiple overlapping systems: working memory (what we're actively holding in mind, severely limited in capacity), episodic memory (specific past experiences, retrievable by cue), semantic memory (general knowledge, diffuse and associative), and procedural memory (skills and habits, largely inaccessible to conscious inspection).

Transformers have none of this architecture. Their “memory” comes in two forms: the context window (temporary, discarded after each inference call) and the weights themselves (fixed after training, encoding statistical patterns from billions of documents). There is no hippocampus performing consolidation. No episodic store you can query. No long-term memory that accumulates across conversations. Each conversation starts, in a real sense, from nothing.

Human Memory
Multi-system, persistent
Working memory, episodic, semantic, procedural. Experiences accumulate across a lifetime. Relevant memories are retrieved associatively, not sequentially. Consolidation happens during sleep. The system evolves continuously.
Transformer “Memory”
Two systems, both limited
Context window (temporary, quadratic cost) and model weights (frozen after training). Nothing persists between conversations. No consolidation. No retrieval from past experience. Each session is a clean slate.

What Researchers Are Trying

The memory problem is one of the most active areas of AI research. The approaches cluster around three strategies: extend the context window, add external memory, or fundamentally rethink the architecture.

01
Efficient Attention Variants
Sparse attention (attending to a subset of tokens), linear attention (approximating full attention at linear cost), sliding window attention, and Flash Attention (a hardware-efficient implementation) all aim to reduce the O(n²) cost of full attention. These make longer contexts tractable but don't eliminate the fundamental trade-off.
02
Retrieval-Augmented Generation (RAG)
Rather than fitting everything in the context window, retrieve only what's relevant. A vector database stores embeddings of documents; at inference time, the most relevant chunks are retrieved and inserted into context. This sidesteps the context limit but introduces new problems: what to retrieve, how to rank relevance, how to handle edge cases where relevant information spans multiple chunks.
03
State Space Models
Architectures like Mamba replace attention with a recurrent state representation that updates continuously as the sequence is processed. The cost scales linearly with sequence length. The trade-off: the compressed state inevitably loses information. Whether SSMs can match transformer quality at long contexts remains an active debate — early results are promising but not conclusive.
04
External Memory Systems
Give the model access to persistent, writable storage it can explicitly read from and write to. This is the approach taken by AI agent frameworks. The model reasons about what to store, when to retrieve, and how to integrate retrieved information with its current reasoning. Closest in spirit to human episodic memory — and furthest from being solved.

Why It Matters Beyond Engineering

The context window problem is often framed as a technical challenge — a matter of making attention faster, or building better retrieval systems. But it has deeper implications for what AI systems can be.

A doctor builds up a detailed model of a patient over years of visits. A lawyer understands a client's legal situation through a web of interconnected matters stretching back decades. A collaborator on a long creative project carries an evolving, shared understanding of its themes, constraints, and history. These kinds of persistent, accumulated understanding are precisely what current AI systems cannot do. They can simulate them within a conversation. They cannot sustain them across time.

Implication This is not just a capability gap. It is a trust gap. We extend trust to professionals and collaborators partly because they remember — they have context, history, and accumulated judgment. An AI that forgets everything the moment a session ends cannot, in the same way, be trusted as a genuine long-term partner. Solving the memory problem isn't only about making AI more capable. It's about making it a different kind of thing entirely.

The context window is, in this sense, the most legible marker of the distance between what AI is now and what it might become. Every token beyond the current limit is a reminder that intelligence, for now, still has an edge — and that beyond that edge, everything disappears.