Feature Request: Slim Attention (lossless 2x reduction in KV cache size)

[Mushoz]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Slim attention can reduce KV cache size by a factor of 2 without a loss of accuracy as it's lossless: https://arxiv.org/pdf/2503.05840

Motivation

Allows you to run with larger context sizes at the same (V)RAM usage or allows you to cram the same context into less (V)RAM. Furthermore, it improves performance at long context sizes.

Possible Implementation

No response

[NilsGraf]

Discussion on reddit:
https://www.reddit.com/r/LocalLLaMA/comments/1j9wkc2/slim_attention_cut_your_context_memory_in_half

Github of slim attention:
https://github.com/OpenMachine-ai/transformer-tricks

[Xonar92]

Also seems compatible with FA, so a huge (total) reduction of VRAM usage for long context might be feasible in the near future

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[VL4DST3R]

Bump. This would a great addition, don't let it fall to the wayside.