Weclome to use MInference, which leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for million tokens LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with the optimal custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy with 1M tokens.
text
Weclome to use MInference, which leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for million tokens LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with the optimal custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy with 1M tokens.
new_line
new_line
For more detail please check, 
text
For more detail please check, 
new_line
project page: 
text
project page: 
https://aka.ms/MInference
https://aka.ms/MInference
link
new_line
code: 
text
code: 
https://github.com/microsoft/MInference
https://github.com/microsoft/MInference
link
new_line
paper: 
text
paper: 
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via
  Dynamic Sparse Attention (2407.02490)
https://huggingface.co/papers/2407.02490
2407.02490
paper
resource
https://huggingface.co/papers/2407.02490
new_line
hf demo: 
text
hf demo: 
https://huggingface.co/spaces/microsoft/MInference
microsoft/MInference
space
resource
https://huggingface.co/spaces/microsoft/MInference
Weclome to use MInference, which leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for million tokens LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with the optimal custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy with 1M tokens.

For more detail please check, 
project page: https://aka.ms/MInference
code: https://github.com/microsoft/MInference
paper: https://huggingface.co/papers/2407.02490
hf demo: https://huggingface.co/spaces/microsoft/MInference
🔥
osanseviero