New preprint out with colleagues from MIT and IBM Research
text
New preprint out with colleagues from MIT and IBM Research
new_line
new_line
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
https://huggingface.co/papers/2405.12981
2405.12981
paper
resource
https://huggingface.co/papers/2405.12981
new_line
new_line
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
text
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
New preprint out with colleagues from MIT and IBM Research

https://huggingface.co/papers/2405.12981

We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
👍
Jaward DmitryRyumin osanseviero alielfilali01 whitebill
6032802e1f993496bc14d9e3
https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
Omar Sanseviero
osanseviero
user