AMD has released the fully open-source MoE Large Model, Instella-MoE, with a parameter scale of 16B, challenging mainstream open-source models.

July 25th, AMD announced the launch of the fully open-source Mixtures of Experts (MoE) model, Instella-MoE, with a total of 16 billion parameters, each Token activating 2.8 billion parameters, claiming to have leading performance among open-source language models of the same scale.

AMD stated that Instella-MoE was entirely trained from scratch based on its own AMD Instinct MI300X and MI325X GPUs, as well as the ROCm software stack, and leveraged architecture innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective to enhance training and inference efficiency.

Performance tests showed that the Instella-MoE-16B-A3B base model achieved an average score of 76.7, leading among open-source models, surpassing models like SmolLM3-3B, OLMo-3-7B, and competing with larger-scale models with only 2.8 billion activated parameters.

Furthermore, the model supports 64K Token-length context processing and completes the full training process, including pre-training, mid-term training, long-context extension, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning (RL).

AMD has simultaneously released all model weights, training configurations, data ratios, intermediate checkpoints, and inference code for Instella-MoE, promoting open AI model research and reproducibility.

AMD stated that Instella-MoE demonstrates the ability to conduct large-scale MoE model training based on AMD hardware and an open software ecosystem, and will continue to advance the development of open-source language models with larger scales, stronger inference capabilities, and higher efficiency in the future.