Question: Quantization through GPTQ 
text
Question: Quantization through GPTQ 
new_line
new_line
Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 
text
Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 
new_line
new_line
quantization_config = GPTQConfig(
text
quantization_config = GPTQConfig(
new_line
     bits=4,
text
     bits=4,
new_line
     group_size=128,
text
     group_size=128,
new_line
     dataset="wikitext2",
text
     dataset="wikitext2",
new_line
     batch_size=16,
text
     batch_size=16,
new_line
     desc_act=False
text
     desc_act=False
new_line
new_line
)
text
)
new_line
new_line
1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
text
1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
new_line
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)
text
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)
new_line
new_line
I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 
text
I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 
new_line
new_line
 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.
text
 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.
new_line
new_line
https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51
https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51
link
text
new_line
new_line
Could you kindly suggest?  Appreciate your kind support. 
text
Could you kindly suggest?  Appreciate your kind support. 
new_line
Thanks
text
Thanks
new_line
new_line
Question: Quantization through GPTQ 

Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 

quantization_config = GPTQConfig(
     bits=4,
     group_size=128,
     dataset="wikitext2",
     batch_size=16,
     desc_act=False

)

1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)

I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 

 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.

https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51  

Could you kindly suggest?  Appreciate your kind support. 
Thanks


👀
victor
6553a92df60d0d9678f45daf
/avatars/a8ae31b23226e85b2f3d8e1c173da5f3.svg
KR
KashyapR
user