On October 24, 2026, a paper titled “RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training” was published on the arXiv platform (arXiv ID: 2610.31098v1), introducing RIFT—a novel KV cache compression framework designed for draft models in speculative decoding.
The paper points out that the traditional KV cache selection method, which relies on pre‑filled information, is prone to “decoding staleness” during long‑output generation: as the number of generated tokens increases, the relevance of earlier cache entries diminishes, leading to a decline in the draft model’s accuracy. RIFT addresses this issue by dynamically refreshing the KV cache selection during token generation, balancing pre‑filled information with the most recently generated content, thereby significantly improving draft quality—though at the cost of additional decoding latency. To mitigate this, the authors further devise an adaptive refresh strategy to reduce the computational overhead of scoring.
Additionally, this work introduces a “compressibility-driven draft model training” phase, leveraging sparse attention masks to optimize the model’s ability to adapt to the compressed KV cache. In evaluations where Qwen3.5-4B serves as the draft model and Qwen3.5-122B as the target model, RIFT, combined with this training method, can still maintain over 98% of the full cache‑enabled input length under stringent conditions: a 32K‑token input, a 4K‑token output, and a KV cache budget of only 8K tokens.
Please note: This work is currently an arXiv preprint and has not yet undergone peer review; all experimental results are based on the setup described in the paper and do not report the generalization performance of other models or hardware platforms; the paper does not disclose engineering metrics such as absolute inference latency or GPU memory savings, nor does it discuss RIFT’s impact on the verifier model.