Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained as well?

🤗Transformers
1

Hello,
Is the multiple-choice head for the pre-trained LongformerForMultipleChoice model pre-trained as well?
I am asking because like for the pre-trained GPT2DoubleHeadsModel, the main body of the model is trained but its multiple-choice head is not.

Is the multiple-choice head for the pre-trained LongformerForMultipleChoice model untrained, just like the case for the GPT2DoubleHeadsModel?

Thank you,

2

Hi @h56cho
Yes , multiple-choice head is randomly initialized, you need to train these heads according to your tasks.