Ant Group and vLLM Jointly Open-Source Humming: A High-Performance, Flexible Quantized Inference Kernel

October 21 - Ant Group and the vLLM community have jointly open-sourced Humming, a general-purpose, high-performance quantized inference operator library. Humming features: broad device support, covering a variety of GPUs as well as CUDA-programming-compatible devices; rich quantization support, supporting a wide range of weight and activation format combinations in Dense and MoE through a unified architecture, covering multiple integer and floating-point types, bit widths, and scaling methods. Its outstanding execution performance significantly outperforms existing implementations such as vLLM Marlin, FlashInfer, and DeepGEMM across various typical workloads.