The transformer architecture wasn't designed to model the brain. Its inventors were solving a translation problem. And yet neuroscientists keep finding that it replicates patterns of human cognition that took evolution millions of years to produce.
In 2017, a team at Google Brain published “Attention Is All You Need” — a paper that replaced recurrent networks with a mechanism called self-attention. The goal was efficiency. Train faster. Translate better. No one was thinking about cortical columns or hippocampal replay. And yet, in the years since, researchers in neuroscience and cognitive science have repeatedly found themselves saying the same quiet, unsettling thing: this looks familiar.
Attention and Selective Focus
The transformer's attention mechanism allows every token in a sequence to “attend” to every other token — weighting relevance dynamically, in context. This is not how earlier neural networks worked. And it is strikingly similar to how neuroscientists describe selective attention in the human brain.
When you read a sentence, your visual cortex doesn't process every word with equal weight. Your brain assigns relevance dynamically, suppressing irrelevant inputs and amplifying what matters. The thalamus acts as a kind of gating mechanism — deciding what reaches conscious processing. Parallel Transformers implement a mathematical version of exactly this: a learned gating function that selects what matters, in context.
The thalamus filters sensory signals before they reach the cortex — amplifying relevant inputs and suppressing noise. Attention is not uniform; it is dynamically allocated based on task demands and prior context.
Each attention head computes a weighted sum over all positions in a sequence. Weights are learned and context-dependent — a token attends more to some tokens than others based on the current task. The mechanism is mathematically analogous to dynamic filtering.
Your product name here
One or two sentences on what this product does and why it fits a reader of this journal. Keep the tone of the surrounding writing — it reads better and converts better.
Learn more →Layers as Hierarchical Processing
Transformer models are deep — stacked layers of attention and feedforward computation, each one transforming the representation produced by the layer below. Early layers tend to capture surface syntax. Middle layers: semantics. Later layers: abstract reasoning and task-specific inference.
This layered specialization is not programmed in. It emerges from training. And it maps with uncomfortable precision onto the hierarchical organization of the visual cortex — where V1 detects edges, V2 detects contours, V4 detects objects, and IT cortex handles abstract categorical recognition. The hierarchy wasn't designed into either system. Both evolved it — one through millions of years of natural selection, the other through gradient descent on text.
“The fact that a model trained to predict words develops the same hierarchical feature representations as a cortex trained to predict the world is not a coincidence. It's a constraint.”
Ila Fiete, MIT Neuroscience, paraphrased from 2024 lecture
The Memory Problem
One of the most striking parallels concerns memory. Transformers have a context window — a finite working memory within which all information must be available for attention to operate. Beyond that window, information is lost unless explicitly stored. This is architecturally similar to the distinction between working memory and long-term memory in the brain.
The hippocampus consolidates short-term experiences into long-term storage during sleep. Transformers have no equivalent. But researchers studying how transformers store factual knowledge in their weights have found that the feedforward layers function like a kind of implicit long-term memory — encoding facts in the weights during training, retrievable during inference.
What the Parallels Do Not Mean
It is tempting to read these parallels as evidence that transformers are, in some meaningful sense, brain-like. They are not. The brain operates on electrochemical signals, modulated by dozens of neurotransmitters, across continuous time. Transformers operate on floating-point arithmetic, in discrete steps, without any continuous temporal dynamics.
Caution What the parallels suggest is something more interesting and more modest: that certain computational solutions to certain problems may be convergent. Evolution found them. Gradient descent found them. This says something profound about the geometry of the problem space, not about the nature of mind.