HF community survey: What is an acceptable Perplexity (PPL) degradation?
text
HF community survey: What is an acceptable Perplexity (PPL) degradation?
new_line
new_line
An area of personal research is to find ways to shrink the size of LLMs without incurring in a noticeable loss of capability. All the models in my repo have been generated by quantizing different tensors at different levels based on how much they influence the inference process (see the model's card for more details). This approach produces, on average, a ~10% size reduction with <  1% of PPL penalty.
text
An area of personal research is to find ways to shrink the size of LLMs without incurring in a noticeable loss of capability. All the models in my repo have been generated by quantizing different tensors at different levels based on how much they influence the inference process (see the model's card for more details). This approach produces, on average, a ~10% size reduction with <  1% of PPL penalty.
new_line
new_line
I'm now focusing on pruning (whole layer removal), as a way to achieve better size reduction, but this comes at the cost of a much higher PPL degradation.
text
I'm now focusing on pruning (whole layer removal), as a way to achieve better size reduction, but this comes at the cost of a much higher PPL degradation.
new_line
new_line
So, the question for the HF community is: what is the lowest/worst PPL correlation coefficient (𝜌PPL) you'd consider acceptable for a quantized model? (e.g. 99%? 95%? 90%? etc)
text
So, the question for the HF community is: what is the lowest/worst PPL correlation coefficient (𝜌PPL) you'd consider acceptable for a quantized model? (e.g. 99%? 95%? 90%? etc)
new_line
new_line
To clarify, by 𝜌PPL I mean the Cor(ln(PPL(Q)), ln(PPL(base))) statistic generated by llama-perplexity.
text
To clarify, by 𝜌PPL I mean the Cor(ln(PPL(Q)), ln(PPL(base))) statistic generated by llama-perplexity.
HF community survey: What is an acceptable Perplexity (PPL) degradation?

An area of personal research is to find ways to shrink the size of LLMs without incurring in a noticeable loss of capability. All the models in my repo have been generated by quantizing different tensors at different levels based on how much they influence the inference process (see the model's card for more details). This approach produces, on average, a ~10% size reduction with <  1% of PPL penalty.

I'm now focusing on pruning (whole layer removal), as a way to achieve better size reduction, but this comes at the cost of a much higher PPL degradation.

So, the question for the HF community is: what is the lowest/worst PPL correlation coefficient (𝜌PPL) you'd consider acceptable for a quantized model? (e.g. 99%? 95%? 90%? etc)

To clarify, by 𝜌PPL I mean the Cor(ln(PPL(Q)), ln(PPL(base))) statistic generated by llama-perplexity.
🚀
John6666
65d3e4b45bea2700d1f7b89e
https://cdn-avatars.huggingface.co/v1/production/uploads/65d3e4b45bea2700d1f7b89e/xJhcJFQlo-_TvAPbCxyqE.jpeg
John Leimgruber III
ubergarm
user
658f24cb35c41262d64af1a2
https://cdn-avatars.huggingface.co/v1/production/uploads/658f24cb35c41262d64af1a2/vUK_c6K821tq61AxSeV4i.png
Ed Addario
eaddario
user