A new H100 and H200 measurement study finds that speculative decoding's energy advantage depends on batch size, draft acceptance, and the cost of verifying rejected tokens.
Speculative decoding is usually described as a way to make language models respond faster. A smaller or auxiliary system proposes several tokens, and the main model checks them in parallel. When enough proposals survive verification, the model produces more output for each expensive pass through its weights.
The energy story is less straightforward. Verifying a proposed token consumes work even if that token is rejected, and a busy GPU may already be sharing its weight reads across many requests. A preprint submitted on October 29, Only Accepted Tokens Count, measures that trade-off across several model architectures and speculative decoding methods. Its central finding is conditional: speculation can substantially reduce joules per generated token, but a long draft can become an energy liability as serving batches grow.
The result is useful because it separates two objectives that are often bundled together: lowering energy per token and shortening the time between tokens seen by an individual user.
In ordinary autoregressive decoding, a model repeatedly reads its weights to generate one new token for each active request. At small batch sizes, the GPU often spends more time moving weights than performing arithmetic. Speculation exploits that imbalance by verifying several candidate tokens during one pass.
The target model does not accept every candidate. It retains a valid prefix of the proposal, then corrects the first disagreement. The remaining proposed tokens still incurred drafting and verification work. That makes accepted tokens per verification more informative than a headline draft length or acceptance percentage alone.
Batching changes the calculation. When 32 or 64 requests run together, one model pass already serves many tokens from different users. Adding speculative tokens increases the arithmetic performed by that pass. Eventually the verifier can move from a memory-bandwidth bottleneck toward a compute bottleneck. The extra rejected work then costs time and energy rather than riding almost for free on a weight read.
This is not a new warning in itself. Fernandez and colleagues' 2025 energy study found that speculation saved energy at smaller batches but consumed more at sufficiently large ones in its tested configurations. The October paper extends the question to more recent models, different attention architectures, multiple drafting strategies, and direct measurements of where the verifier crosses the GPU's arithmetic-to-memory balance point.
The new study evaluates 11 models spanning grouped-query attention, sliding-window attention, multi-head latent attention, and hybrids with recurrent Gated DeltaNet layers. Five are mixture-of-experts models. It compares n-gram lookup, multi-token-prediction heads, EAGLE3, DFlash, and DSpark where compatible checkpoints exist.
The authors use SGLang with CUDA graphs enabled on NVIDIA H100 and H200 GPUs. Their experiments include batches up to 64, greedy decoding, BF16 weights and KV caches, 512 generated tokens per request, and at least three seeds per measurement cell. Each speculative run has a non-speculative baseline with the same model, batch, prompt pool, and engine settings. Energy is measured using the GPU's cumulative board-energy counter.
At batch size one, the best available speculative scheme for each model saves energy; the median of those per-model best savings is 54% of decode energy. For example, the reported best saving on gemma-4-12B-it is 75% at batch one, falling to 25% at batch 64. On Qwen3-8B, the best saving falls from 62% to 24% across the same batch sizes.
These figures are not claims that every drafter saves energy. At batch 64, the best scheme still saves between 15% and 52% on the models that could run that batch, but several other schemes lose energy. The choice of drafter matters as much as whether speculation is enabled.
One particularly revealing comparison concerns DFlash blocks. On Qwen3-4B, a 16-token verification step reaches the H100's measured compute-to-memory balance point around batch 32 and the H200's around batch 16. Its energy advantage disappears around those transitions. DSpark, verifying seven tokens per step, remains beneficial on the H100 at batch 64 but loses its energy advantage on the H200 there. A draft length that works on one accelerator therefore cannot be assumed to be optimal on another.
Draft acceptance is not an intrinsic property of a model-drafter pair. It depends on the text.
The researchers test code, conversation, long documents, and matched English and Romansh news prompts. Every evaluated drafter accepts fewer tokens per verification on conversation than on code. For the DFlash drafter paired with Qwen3-Coder-30B-A3B-Instruct, accepted tokens fall from 5.9 on code to roughly 2.0–3.0 on conversation. At batch 64, that configuration saves 51% of decode energy on code but costs energy on conversation.
Longer drafts can look attractive at low concurrency because they produce more accepted tokens per weight read. At higher concurrency, a shorter multi-token-prediction head can win even while accepting fewer tokens: it also asks the target to verify fewer rejected candidates.
The distinction becomes sharper for recurrent hybrids. Rejected speculative tokens require restoring the correct recurrent state, not just dropping unused KV-cache entries. The paper reports cases where long-block drafting could not start at larger batches because of rollback-state memory. A deployment budget must account for this state as well as model weights and ordinary caches.
One of the paper's strongest contributions is methodological. The authors show that disabling CUDA graph capture can exaggerate the apparent energy benefit of speculation by 19 to 39 percentage points on tested mixture-of-experts models at single stream. The non-speculative baseline is especially sensitive to kernel-launch overhead, so comparing an uncaptured baseline with a captured or differently optimized speculative path can be misleading.
They also document less obvious hazards: an engine may admit fewer requests than the configured batch; repeated prompts can accidentally activate prefix caching; missing chat templates can alter the generated text; and profiling windows can include prefill rather than decode. Each can change an energy result without producing an obvious error.
The measurements also clarify what speculation buys under a latency constraint. At single stream, the fastest schemes improve per-user token cadence by 1.3 to 4.3 times in the tested models. Under a mean target of 25 milliseconds per output token, the lowest-energy qualifying speculative configuration uses 18% to 69% less energy than a qualifying non-speculative configuration on 10 of the 11 models; the remaining model meets that target only with speculation in the evaluated configurations. Batching alone cannot provide the same per-user speed improvement.
This is an arXiv preprint, not an independently reproduced benchmark. Earlier work supports the broad observation that inference energy depends strongly on batch and workload, and a peer-reviewed EACL 2026 study also examines how speculative decoding energy varies with model family, method, and dataset. Neither independently verifies the October paper's specific H100 and H200 numbers.
The authors measure GPU-board energy, not full server or facility energy. Most prompt contexts are short, the batch sweep stops at 64, and latency is reported as a mean rather than a tail percentile. The default engine verifies a fixed number of tokens for every request, even when fewer might suffice; a different adaptive verification policy could change the trade-off. The roofline comparison is measured on Qwen3-4B rather than every model in the study.
Those limits matter particularly for local inference, where consumer GPUs, CPUs, shared-memory devices, and quantized models can have very different memory and arithmetic ratios. The paper does not establish an optimal speculative configuration for those devices.
Its practical lesson is more durable than any single percentage: benchmark the actual model, drafter, hardware, batch, and prompt distribution together. Report accepted tokens, tokens verified, energy per output token, and per-user latency under the same execution settings. A faster decoder is not automatically a more energy-efficient one, and the best choice can change when the server gets busy.
