# Positional Encoding error, Protein Bert Model

**URL:** <https://discuss.huggingface.co/t/positional-encoding-error-protein-bert-model/1737>\
**Category:** Intermediate\
**Created:** [October 25, 2020, 8:54am UTC](https://discuss.huggingface.co/t/positional-encoding-error-protein-bert-model/1737 "2020-10-25T08:54:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![donal](https://avatars.discourse-cdn.com/v4/letter/d/a9adbd/32.png) [@donal](https://discuss.huggingface.co/u/donal)\
**Post date:** [October 25, 2020, 8:54am UTC](https://discuss.huggingface.co/t/positional-encoding-error-protein-bert-model/1737/1 "2020-10-25T08:54:56Z")

</div>

Hi Guys,

This seems very obivious but I can’t seem to find an answer anywhere. I’m trying to build a very basic roberta protein model similar to ProTrans. It’s just Roberta but I need to use a very long positional encodings of 40\_000, because protein seqeunces are about 40,000 amino acids long. But anytime I change the max postional embeddings to 40k I keep getting an CUDA error: device-side assert triggered error.

(additidonal question I save the linebylinetext to a variable. From what I understand linebyline splits the sentence into chunks, but how can I index in to see each chunk?)  
(also other than a lambda function is there any more efficient way I can preprocess and tokenise this dataset?)

Steps

1. Preprocess a Uniref50 sequence into a single space text document. Treating each Amino acid as a word and each protein as a sentence.
2. Tokenise, I can load in the tokens from ProTrans model, tokenises fine.
3. Use line by line text dataset to split each into 40\_0000 blocks,
4. Step up the config, important changes to roberta vocab size 30, max\_positional\_embeddings= 40\_000
5. Run through datacollator, model setup, train setup, get error.

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](https://discuss.huggingface.co/u/Zhylkaaa)\
**Post date:** [October 25, 2020, 3:15pm UTC](https://discuss.huggingface.co/t/positional-encoding-error-protein-bert-model/1737/2 "2020-10-25T15:15:43Z")

</div>

Hi @donal I guess it’s near impossible to put 40 000 tokens in any transformer model that uses full attention. Maybe a better choice is [Reformer](https://arxiv.org/pdf/2001.04451.pdf) or LongFormer(I am not sure that LongFormer can handle 40 000 tokens too). The main problem of BERT architecture is that it’s memory and computational complexity is O(L^2) where L is max\_seq\_len, so I guess about 2048 tokens is upper limit.  
Or maybe consider using something like sliding window or some kind of memory banks 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![donal](https://avatars.discourse-cdn.com/v4/letter/d/a9adbd/32.png) [@donal](https://discuss.huggingface.co/u/donal)\
**Post date:** [October 25, 2020, 9:44pm UTC](https://discuss.huggingface.co/t/positional-encoding-error-protein-bert-model/1737/3 "2020-10-25T21:44:00Z")

</div>

I figured it out, but now I keep getting freezes with a mutli Gpu set up. Anyone else having this problem?