--- title: "Orthrus Shows Why 'Lossless' Speculative Decoding Depends on Precision" description: "An independent reproduction finds exact Orthrus trajectories in every tested FP32 case, but only 43–45% under BF16." date: 2026-09-14 topic: inference tags: ["speculative-decoding","numerical-precision","diffusion-decoding"] production: ai-assisted source: https://notes.suffro.com/posts/orthrus-lossless-decoding-bf16 --- # Orthrus Shows Why 'Lossless' Speculative Decoding Depends on Precision A new independent reproduction of Orthrus puts a useful qualification on one of the strongest promises in speculative decoding: "lossless" behavior can depend on the numerical precision used at inference time. In a [preprint posted on September 14](https://arxiv.org/abs/2609.15504), Ilya Koziev, Leonid Sinev, and Ivan Oseledets report that Orthrus exactly matched the reference autoregressive trajectory on only 45% of prompts with the authors' checkpoint and 43% with an independently trained model when inference ran in BF16. The evaluation covered 1,190 prompts from 12 domains. Repeating the trajectory test in FP32 produced exact matches on every evaluated prompt. That does not mean Orthrus stops being useful in BF16. The same reproduction found no systematic degradation on downstream `lm-eval-harness` benchmarks. Instead, it exposes an important distinction: preserving task-level quality is not the same thing as reproducing the exact token trajectory of the reference model. ## How Orthrus is supposed to stay lossless [Orthrus](https://arxiv.org/abs/2605.12825), introduced in May, is designed to accelerate autoregressive generation without replacing the base language model's distribution with a separately trained parallel model. The method freezes an autoregressive Transformer and adds a lightweight diffusion attention module. During prefill, the autoregressive path builds the normal high-fidelity KV cache. The diffusion path then uses that same cache to propose a block of future tokens in parallel instead of producing them one by one. The key step is verification. Orthrus sends the proposed block through the frozen autoregressive path, which can score all populated positions in parallel. For greedy decoding, a proposed token is accepted only while it agrees with the autoregressive model's argmax prediction from left to right. At the first disagreement, Orthrus falls back to the autoregressive correction and starts another speculative cycle. For nonzero-temperature sampling, the paper uses rejection sampling to align the proposal with the target distribution. This is why the original work describes the method as strictly lossless. Its paper reports speedups of up to 7.8×, about 16% additional trainable parameters, and constant historical KV-cache overhead because the two views share the same cache. Those performance figures are results reported by the Orthrus authors; the September reproduction is primarily a test of output fidelity, not an independent re-benchmark of the claimed speedups. ## What the reproduction changes The new work asks a narrower question than a conventional benchmark: if Orthrus is run beside its reference autoregressive model, do both executions produce exactly the same sequence? Under BF16, often they do not. Across 1,190 prompts, exact trajectory agreement was 45% for the released Orthrus checkpoint and 43% for the independently trained reproduction. The authors also found a strong association between exact matching and the response-conditional perplexity of the reference model. In other words, trajectories were less stable in cases where the reference model was less confident about the generated continuation. The most important control is FP32. When the researchers repeated the trajectory evaluation at full 32-bit floating-point precision, every evaluated prompt matched exactly. That makes the result substantially more informative than a generic reproduction failure: the reported divergence changes with inference precision. At the same time, the BF16 divergence did not translate into a systematic drop on the downstream `lm-eval-harness` tasks tested by the authors. An implementation can therefore diverge from the exact reference trajectory while remaining practically indistinguishable on coarse task metrics. ## "Lossless" has always had a numerical boundary The issue is not unique to Orthrus. The original 2023 [speculative sampling paper](https://arxiv.org/abs/2302.01318) states its distribution-preservation result explicitly as holding "within hardware numerics." That qualifier matters because speculative decoding makes acceptance decisions from model probabilities or token rankings. Reduced-precision arithmetic can slightly perturb intermediate activations and logits. When competing tokens are close enough, a small numerical change may alter an argmax or another decision used by the decoding procedure. Once one token differs, every later autoregressive state is conditioned on a different prefix, so the trajectories can continue to separate. The Orthrus reproduction does not establish that this is the exact mechanism behind every BF16 mismatch, and it does not show that reduced precision makes speculative decoding generally invalid. What it does show is that mathematical distribution-preservation arguments and bit-for-bit trajectory equivalence are different properties in a real finite-precision implementation. It also means that a statement such as "lossless speculative decoding" is incomplete unless the intended notion of losslessness is clear. It may mean preservation of the target distribution under the algorithmic assumptions, equality of greedy outputs under a particular numerical configuration, or simply no measurable loss on downstream benchmarks. Those are not interchangeable tests. ## Why this matters for local inference Local and edge inference makes this distinction especially relevant because reduced precision is often part of the deployment strategy. BF16, FP16, INT8, and lower-bit formats exist precisely because memory capacity, bandwidth, latency, and energy are constrained. A speculative-decoding implementation that promises exact reference behavior should therefore test that property in the same precision, kernels, model configuration, and hardware path that users will actually run. Exact trajectory comparison should be reported separately from benchmark accuracy and from speed. For users, the practical decision depends on the goal. If the requirement is identical greedy output to a reference run, the numerical configuration becomes part of the correctness contract. If the requirement is faster inference without measurable degradation on downstream tasks, exact token-for-token identity may be unnecessarily strict. The reproduction is also a useful reminder for benchmark design. A single task score can hide low-level behavioral divergence, while exact sequence equality can exaggerate differences that do not matter to the application. Efficient inference work benefits from reporting both. ## Limits of the result The September study is an arXiv preprint, not a peer-reviewed result. Its FP32 experiment found zero mismatches on the evaluated prompts, but that is evidence for those tests rather than a universal proof that FP32 will always reproduce every Orthrus trajectory. The findings are also specific to Orthrus and the evaluated setup. They should not be generalized to every speculative decoder or every reduced-precision backend without separate measurements. Still, the central lesson is durable: when an inference method makes an exactness claim, numerical precision is not merely a performance setting. It can be part of what "exact" means. ## Sources - [Koziev, Sinev, and Oseledets, "How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus"](https://arxiv.org/abs/2609.15504) - [Nguyen et al., "Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion"](https://arxiv.org/abs/2605.12825) - [Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling"](https://arxiv.org/abs/2302.01318)