Local AI, model serving, performance, and efficient ML.
A new H100 and H200 measurement study finds that speculative decoding's energy advantage depends on batch size, draft acceptance, and the cost of verifying rejected tokens.
The new open-source CUDA kernel library combines quantization formats and GPU-specific execution paths; vLLM benchmarks show useful gains with important precision caveats.
An independent reproduction finds exact Orthrus trajectories in every tested FP32 case, but only 43–45% under BF16.
