Cursor Reordering GPU Pipeline, MoE Model Training Accelerated by 41%

According to Dynamic Beating monitoring, Cursor has open-sourced the Mixture-of-Kittens (MoK) to accelerate MoE large-scale model training. It combines the previously separate GPU data transfer and computation into a single kernel (a low-level program running directly on the GPU).

MoE decomposes the model into numerous "experts," with only a subset called upon at each instance. These experts are spread across different GPUs, requiring frequent data movement, sometimes occupying more than half of the training time.

MoK enables GPUs to perform both data transfer and computation simultaneously, reducing waiting time. During actual training using 512 GB300 GPUs, the overall throughput increased by 41%. When specifically testing the MoE layer, it was up to 2.37 times faster than the public baseline solution.

MoK has been utilized in Composer training on tens of thousands of Cursor GPUs and has been open-sourced under the Apache 2.0 license. Currently, it only supports NVIDIA Blackwell GPUs and is primarily aimed at institutions with GB200 and GB300 NVL72 clusters.