Quantizing a model is only part of making it faster. The weights may occupy fewer bytes, but the inference runtime still has to unpack them, apply scales, and execute matrix multiplications efficiently on the GPU that is actually available. Different weight formats and GPU generations have traditionally meant different kernels, preprocessing paths, and performance trade-offs.

On October 21, Ant Group and the vLLM community introduced Humming, an open-source CUDA kernel library that tries to consolidate that fragmented implementation work. It targets quantized dense matrix multiplication and mixture-of-experts (MoE) computation, including low-bit weights paired with 16-bit, 8-bit, or 4-bit activations where the hardware and format allow it.

The interesting result is not a new quantization algorithm. Humming is an execution layer for already-quantized computations, with a common description of quantization formats and multiple GPU-specific ways to run them. Its source code is available under Apache 2.0, and vLLM exposes it as a selectable backend.

One quantization description, several execution paths

A label such as “4-bit weights” leaves out much of what a kernel needs to know. The weights might be integers or a floating-point format such as MXFP4 or NVFP4. Scales can apply to an entire tensor, individual output channels, or small blocks. Zero points, secondary scales, activation precision, and the physical layout of packed weights also affect the calculation.

Humming separates these properties from the GPU instructions used to perform the multiplication. Its configuration documentation describes three layers:

LayerConfig captures the weight representation, tensor dimensions, scales, zero points, and target GPU architecture. Transforming the weights establishes this layout; changing a storage property generally requires transforming them again.

ComputeConfig describes the operation, including whether the matrix multiplication is dense or uses an indexed or grouped MoE layout. Those layouts matter because an MoE router sends different numbers of tokens to different experts, so a kernel cannot always assume one uniform matrix shape.

TuningConfig selects execution details such as tile sizes, pipeline stages, warps, memory-transfer instructions, and scheduling. A deployment can select a different compatible execution strategy without necessarily changing its stored quantized weights.

The implementation assembles CUDA components for data movement, decoding and scaling, Tensor Core computation, and output handling, then compiles the selected kernel with NVRTC or NVCC. Compiled artifacts can be cached. This is a useful distinction from maintaining a completely separate kernel family for every combination of weight format, activation type, and GPU.

There is still hardware-specific code underneath. Humming does not make all low-precision instructions available on every card. Its published support matrix covers NVIDIA architectures from Turing (SM75) onward, but the efficient activation and weight combinations vary by generation. A newer GPU's native FP4 path is not equivalent to converting FP4 weights into a format an older GPU can execute.

Why the GPU and batch size change the answer

A single kernel configuration is unlikely to be optimal for every request shape. During token-by-token decoding, matrix multiplications can be small and memory traffic matters greatly. During prefill or high-concurrency serving, larger operations may benefit from different tiling, transfer instructions, and Tensor Core pipelines.

Humming currently uses device-specific heuristics to choose configurations across workload sizes. The maintainers describe paths using asynchronous copies and conventional MMA instructions for some smaller workloads, and TMA/UMMA-based pipelines for larger ones on suitable hardware. Automatic empirical tuning is listed as future work, not an already completed feature.

Compilation cost matters too. In one first-party test on an H200 with CUDA 13.4, the authors report 1.82 seconds to compile a single W4A16 Humming kernel with NVCC, versus 23.35 seconds for the tested CUTLASS compilation, with caches disabled. That is a comparison of specific compilation paths, not a general statement about every kernel or end-to-end model startup.

For local deployment, this flexibility is relevant even when a large datacenter GPU is not available. Supporting multiple NVIDIA generations with one library can reduce the number of format-specific backends a serving stack needs. It does not remove the need to check whether a particular quantized checkpoint, GPU, and activation type are compatible.

What the benchmarks actually establish

The launch measurements show substantial kernel-level improvements against Marlin, FlashInfer, and DeepGEMM on selected formats and hardware. Those plots cover dense and MoE operations, and the claimed advantage varies with matrix dimensions, quantization format, and GPU generation. They should not be read as a universal speedup.

There is also a more useful end-to-end test. On eight NVIDIA H20 GPUs with tensor parallelism set to eight, the authors served DeepSeek-V4.1-Flash while keeping its MXFP4 weights unchanged. They compared Marlin W4A16 MoE computation, using BF16 activations, against Humming W4A8 computation, using tokenwise FP8 activations.

Across the reported concurrency settings from one to 64, output throughput improved by approximately 14% to 43%. At concurrency 64, it rose from 442.37 to 633.94 output tokens per second. Mean time to first token fell from 5,467.35 to 3,882.94 milliseconds, and mean time per output token fell from 123.25 to 85.70 milliseconds.

These are first-party results, not an independent reproduction. More importantly, the comparison changes both the kernel backend and the activation precision. It shows what that combined deployment choice achieved under the reported conditions. It does not isolate how much of the gain comes from Humming's implementation rather than the move from BF16 to FP8 activations, and it does not by itself establish equivalent output quality.

The repository contains Humming microbenchmark scripts and a Marlin comparison harness. These use generated tensors, GPU synchronization, and Triton's benchmarking utility with explicit warmup and repetition settings. That makes the kernel comparison more inspectable than an unsupported performance claim, but a full model-serving reproduction still needs the exact model, checkpoint, GPU setup, software versions, workload, and quality checks.

No GPU benchmarks were independently run for this article.

Integration is real; portability has boundaries

Humming is packaged as humming-kernels, with PyTorch 2.7 or newer specified in its project dependencies and CUDA compilation tooling needed for on-demand kernels. In vLLM, the kernel configuration includes humming among the available dense and MoE backends. The launch post documents selecting it with --linear-backend humming or --moe-backend humming.

That does not mean every model will use Humming for every layer. The vLLM quantization documentation also describes per-quantization backend overrides, which are useful when a model mixes quantization schemes. The actual kernel choice still depends on supported operations and formats.

There are several boundaries worth keeping visible. Quantized arithmetic can change numerical results, particularly when activation precision changes. Weight transformation has its own cost and hardware-dependent layout. Just-in-time compilation can add startup latency even when cached runs are fast. Some of the strongest published speedups are microbenchmarks, not complete model measurements, and the end-to-end result covers one model and one eight-GPU configuration.

Humming is worth following because it addresses a durable systems problem: low-bit inference is not one kernel, and format proliferation makes maintaining fast implementations expensive. The released code and vLLM integration provide a concrete way to test the idea. Whether it is the best choice for a particular local GPU or serving workload remains a measurement question, not something the headline speedups can settle.

Sources

• Ant Group and vLLM Community, “Announcing Humming: Fast, Flexible Kernels for Quantized Inference” (October 21, 2026)
• Humming source repository and supported-format matrix
• Humming configuration documentation
• Humming README and hardware support matrix
• Humming benchmark implementation
• Marlin comparison benchmark implementation
• Humming package metadata and dependencies
• vLLM kernel backend configuration
• vLLM quantization and per-backend selection documentation