From GPT-2 to Kimi K3: The Seven-Year Evolution of Large Models, Essentially a Memory Arms Race

According to Perceiving AI monitoring, Baseten engineer Ali published a long article breaking down from GPT-2 to Kimi K3. In terms of parameters, one K3 is equivalent to about 22,580 GPT-2 models. However, he believes that the real change in the past seven years is not simply scaling up models, but continuously transforming the way models store, update, and retrieve information.

GPT-2 uses full attention. Each token leaves behind its key and value, which can be revisited at any time. While information retention is most complete, the longer the context, the higher the KV cache and computational cost.

Linear attention compresses the entire history into a fixed-size state. The cost no longer escalates with the expanding context, but the trade-off is that different information gets crowded into the same space. As the content grows, old memories start to interfere with each other.

Subsequent architectural innovations aim to add management rules to this limited memory. DeltaNet allows the model to first read old content and then overlay with the differences between old and new. Gated DeltaNet introduces overall forgetting, allowing old information to decay with changing contexts.

KDA further refines forgetting control down to each channel. Different information can have different retention times: long-term facts are preserved longer, while temporary instructions and local variables decay faster. Ali believes this is the most critical advancement in Kimi Linear.

However, a fixed state cannot retain all details. Therefore, K3 does not solely rely on KDA. Instead, it arranges 23 sets of four-layer structures into "three layers of KDA plus one layer of MLA," with a final additional layer of MLA. KDA is responsible for cost-effective maintenance of long-term memory, while MLA periodically revisits the complete context to retrieve precise information such as numbers, code, and early instructions.

K3 also introduces Attention Residuals. Regular models continuously add results from each layer, causing early extracted information to be diluted in the 93 layers of computation. K3 divides the network into 12-layer blocks, allowing later layers to selectively access intermediate representations formed in earlier stages without needing to derive from the original text every time.

Therefore, the key to K3 is not a sudden appearance of black technology. Its true uniqueness lies in combining recurrent memory, global retrieval, inter-layer hindsight, and expert routing into a layered information system.

The architectural evolution of models over the past seven years fundamentally teaches models how to choose what is worth retaining, what should be overwritten, and when they must refer back to the original text.