Why is Kimi K3 catching up to Fable 5? Moonside simultaneously rewrites Attention, Residuals, and MoE

According to Watchful Beating monitoring, the Dark Side of the Moon has released the Kimi K3 Technical Report. K3 has a total of 2.8 trillion parameters, with each Token activating 104 billion parameters, and the underlying architecture has undergone a comprehensive upgrade from K2.

Parameter enlargement is just one part of it. The Dark Side of the Moon stated that the new architecture, data, and training methods have made K3's overall scaling efficiency about 2.5 times higher than K2. Following its Scaling Law curve, the training computational volume required to achieve the same validation loss is approximately 40% of K2.

K3 first rewrote attention. It uses KDA to handle long sequences, with a global MLA layer added every three layers. KDA compresses previous content into a fixed-size state, reducing the computation and cache pressure on millions of Tokens' contexts; MLA is responsible for retaining global information exchange.

It also introduced Attention Residuals. While traditional models can only continuously superimpose information from all previous layers, K3 allows each layer to directly select necessary content from earlier network blocks. As the model becomes deeper, early information is less likely to be diluted.

MoE was also redesigned. K3 has 896 routing experts, with each Token activating 16, double that of K2. The routing experts first perform calculations in a compressed space and then return to the model backbone. New activation functions and load-balancing algorithms are used to prevent uncontrolled training of super-large models.

What truly enhances Agent capability is post-training. The Dark Side of the Moon trains general, Agent, and code models separately, then trains three levels of thinking intensity for each direction, and finally merges the nine experts. The training task can last for thousands of tool calls and continuously retain file, application, and virtual machine states.

K3 has approached or surpassed Fable 5 and GPT-5.6 Sol in various code, search, and tool call tests. Its surge is not solely due to parameter stacking but rather pushing the limits with a larger foundation, a new architecture, and a million Token Agent reinforcement learning.