We perform three microbenchmarks to validate our hypothesis and analyze decoding throughput: First, we fix the total model parameters at 350M and see how changing layer width and depth would affect …

Feb 5, 2024 · In this work, we perform a detailed study comprising over 350 experiments with LLAMA-65B and OPT-66B using speculative decoding and de-lineate the factors that affect the performance …

The goal of SuffixDecoding is to enable fast, adaptive speculative decoding over long sequences, particularly suited for agentic applications where repeated inference calls often contain highly …

We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm.

No doubt some total misunderstandings of this kind do exist. Bet the vast range must contain same degree of reciprocity between encoding and decoding moments, otherwise we could not speak of an …

In this paper, we introduce a novel framework named Big Little Decoder (BiLD) that can be applied to various text generation scenarios to reduce inference latency without additional training iterations or …

We introduce Retrieval-Based Speculative De- coding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that …