I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the simpletransformers package, but it has memory issues at the moment and after a few epochs my colab session crashes.

How pretrain ELECTRA on custom dataset?

valhalla September 6, 2020, 10:53am 6

1 Like

Topic		Replies	Views
ELECTRA training reimplementation and discussion Research	14	6750	September 17, 2023
PreTrain ELECTRA from scratch in Portuguese Flax/JAX Projects	2	1345	November 8, 2021
How hard is it to finetune an ELECTRA model for multihead regression & classification? 🤗Transformers	0	318	October 15, 2021
ELECTRA: Accounting for mask tokens that are correctly predicted by MLM 🤗Transformers	9	1307	May 15, 2021
Retrain Electra model with different embedings from scratch Beginners	0	328	April 1, 2022