--- datasets: - multimolecule/gencode language: rna library_name: multimolecule license: agpl-3.0 mask_token: pipeline_tag: fill-mask tags: - Biology - RNA - 3' UTR widget: - example_title: microRNA 21 mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: CAUG score: 0.999251 - label: CAGG score: 0.000442 - label: UAUG score: 5.1e-05 - label: AAUG score: 3.3e-05 - label: CAGA score: 2.5e-05 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: UAGCUUAUCAGAUGUUGA - example_title: microRNA 146a mask_index: 10 mask_index_1based: 11 masked_char: A output: - label: UGAU score: 0.988037 - label: UCAU score: 0.00972 - label: ACAU score: 0.000208 - label: CCAU score: 0.000143 - label: UAAU score: 0.000129 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: UGAGAACUGACAUGGGUU - example_title: microRNA 155 mask_index: 15 mask_index_1based: 16 masked_char: A output: - label: GUGG score: 0.80526 - label: GUGA score: 0.089648 - label: GUGC score: 0.06432 - label: GUGU score: 0.037719 - label: GUAA score: 0.000345 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: UUAAUGCUAAUCGUGGGGUU - example_title: RNA component of mitochondrial RNA processing endoribonuclease mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: CUCU score: 0.363528 - label: CUGU score: 0.329001 - label: CCCU score: 0.294664 - label: GCCU score: 0.001334 - label: CGCU score: 0.001205 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: GGUUCGUGCUGCCUGUAUCCUAGGCUACACACUGAGGACUCUGUUCCUCCCCUUUCCGCCUAGGGGAAAGUCCCCGGACCUCGGGCAGAGAGUGCCACGUGCAUACGCACGUAGACAUUCCCCGCUUCCCACUCCAAAGUCCGCCAAGAAGCGUAUCCCGCUGAGCGGCGUGGCGCGGGGGCGUCAUCCGUCAGCUCCCUCUAGUUACGCAGGCAGUGCGUGUCCGCGCACCAACCACACGGGGCUCAUUCUCAGCGCGGCUGUAAAAAAAAA - example_title: 7SK small nuclear RNA mask_index: 13 mask_index_1based: 14 masked_char: A output: - label: GCGC score: 0.992816 - label: GGGC score: 0.006837 - label: UGGC score: 8.7e-05 - label: CGGC score: 7.5e-05 - label: AGGC score: 5.5e-05 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: GGAUGUGAGGGCGGGCUGCGACAUCUGUCACCCCAUUGAUCGCCAGGGUUGAUUCGGCUGAUCUGGCUGGCUAGGCGGGUGUCCCCUUCCUCCCUCACCGCUCCAUGUGCGUCCCUCCCGAAGCUGCGCGCUCGGUCGAAGAGGACGACCAUCCCCGAUAGAGGAGGACCGGUCUUCGGUCAAGGGUAUACGAGUAGCUGCGCUCCCCUGCUAGAACCUCCAAACAAGCUCUCAAGGUCCAUUUGUAGGAGAACGUAGGGUAGUCAAGCUUCCAAGACUCCAGACACAUCCAAAUGAGGCGCUGCAUGUGGCAGUCUGCCUUUCUUUU - example_title: telomerase RNA component mask_index: 23 mask_index_1based: 24 masked_char: A output: - label: GGUG score: 0.996049 - label: GGGG score: 0.003613 - label: GGCG score: 7.8e-05 - label: CGUG score: 7.0e-05 - label: GGGC score: 4.9e-05 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: GGGUUGCGGAGGGUGGGCCUGGGGUGGUGGCCAUUUUUUGUCUAACCCUAACUGAGAAGGGCGUAGGCGCCGUGCUUUUGCUCCCCGCGCGCUGUUUUUCUCGCUGACUUUCAGCGGGCGGAAAAGCCUCGGCCUGCCGCCUUCCACCGUUCAUUCUAGAGCAAACAAAAAAUGUCAGCUGCUGGCCCGUUCGCCCCUCCCGGGGACCUGCGGCGGGUCGCCUGCCCAGCCCCCGAACCCCGCCUGGAGGCCGCGGUCGGCCCGGGGCUUCUCCGGAGGCACCCACUGCCACCGCGAAGAGUUGGGCUCUGUCAGCCGCGGGUCUCUCGGGGGCGAGGGCGAGGUUCAGGCCUUUCAGGCCGCAGGAAGAGGAACGGAGCGAGUCCCCGCGCGCGGCGCGAUUCCCUGAGCUGUGGGACGUGCACCCAGGACUCGGCUCACACAUGC - example_title: vault RNA 2-1 mask_index: 12 mask_index_1based: 13 masked_char: A output: - label: GCAA score: 0.501475 - label: GUAA score: 0.313931 - label: GUUA score: 0.147817 - label: GGAA score: 0.008565 - label: GUGA score: 0.004199 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: CGGGUCGGAGUUCAAGCGGUUACCUCCUCAUGCCGGACUUUCUAUCUGUCCAUCUCUGUGCUGGGGUUCGAGACCCGCGGGUGCUUACUGACCCUUUUAUGCAA - example_title: brain cytoplasmic RNA 1 mask_index: 18 mask_index_1based: 19 masked_char: A output: - label: CCUG score: 0.767475 - label: CUCG score: 0.144532 - label: CUUG score: 0.064397 - label: ACUG score: 0.006056 - label: GCUG score: 0.006033 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: GGCCGGGCGCGGUGGCUCCUGUAAUCCCAGCUCUCAGGGAGGCUAAGAGGCGGGAGGAUAGCUUGAGCCCAGGAGUUCGAGACCUGCCUGGGCAAUAUAGCGAGACCCCGUUCUCCAGAAAAAGGAAAAAAAAAAACAAAAGACAAAAAAAAAAUAAGCGUAACUUCCCUCAAAGCAACAACCCCCCCCCCCCUUU - example_title: HIV-1 TAR-WT mask_index: 13 mask_index_1based: 14 masked_char: A output: - label: GCAG score: 0.553846 - label: GUAG score: 0.409701 - label: GUUG score: 0.022455 - label: ACAG score: 0.001498 - label: GUGG score: 0.00124 pipeline_tag: fill-mask sequence_type: ncRNA task: fill-mask text: GGUCUCUCUGGUUCAGAUCUGAGCCUGGGAGCUCUCUGGCUAACUAGGGAACC - example_title: prion protein (Kanno blood group) mask_index: 21 mask_index_1based: 22 masked_char: A output: - label: UGGG score: 0.999484 - label: UUGG score: 0.000287 - label: UGGC score: 5.5e-05 - label: UGGA score: 4.6e-05 - label: UGGU score: 3.7e-05 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGGCGAACCUUGGCUGCUGGUGGUUCUCUUUGUGGCCACAUGGAGUGACCUGGGCCUCUGC - example_title: interleukin 10 mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: CCUG score: 0.710925 - label: CUUG score: 0.213729 - label: CUCG score: 0.053865 - label: GCUG score: 0.009599 - label: CUGG score: 0.003499 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGCACAGCUCCUGCUCUGUUGCCUGGUCCUCCUGACUGGGGUGAGGGCC - example_title: Zaire ebolavirus mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: AAGU score: 0.580672 - label: AACU score: 0.244028 - label: AUGU score: 0.145776 - label: AAAU score: 0.003751 - label: AGGU score: 0.002252 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AAUGUUCAAACUGUGAAGCUCUGUUAGCUGAUGGUCUUGCUAAAGCAUUUCCUAGCAAUAUGAUGGUAGUCACAGAGCGUGAGCAAAAAGAAAGCUUAUUGCAUCAAGCAUCAUGGCACCACACAAGUGAUGAUUUUGGUGAGCAUGCCACAGUUAGAGGGAGUAGCUUUGUAACUGAUUUAGAGAAAUACAAUCUUGCAUUUAGAUAUGAGUUUACAGCACCUUUUAUAGAAUAUUGUAACCGUUGCUAUGGUGUUAAGAAUGUUUUUAAUUGGAUGCAUUAUACAAUCCCACAGUGUUAU - example_title: SARS coronavirus mask_index: 14 mask_index_1based: 15 masked_char: A output: - label: CUUU score: 0.999988 - label: UUUU score: 4.0e-06 - label: AUUU score: 2.0e-06 - label: GUUU score: 1.0e-06 - label: CUUC score: 1.0e-06 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGUUUAUUUUCUUUUUCUUACUCUCACUAGUGGUAGUGACCUUGACCGGUGCACCACUUUUGAUGAUGUUCAAGCUCCUAAUUACACUCAACAUACUUCAUCUAUGAGGGGGGUUUACUAUCCUGAUGAAAUUUUUAGAUCAGACACUCUUUAUUUAACUCAGGAUUUAUUUCUUCCAUUUUAUUCUAAUGUUACAGGGUUUCAUACUAUUAAUCAUACGUUUGACAACCCUGUCAUACCUUUUAAGGAUGGUAUUUAUUUUGCUGCCACAGAGAAAUCAAAUGUUGUCCGUGGUUGGGUUUUUGGUUCUACCAUGAACAACAAGUCACAGUCGGUGAUUAUUAUUAACAAUUCUACUAAUGUUGUUAUACGAGCAUGUAACUUUGAAUUGUGUGACAACCCUUUCUUUGCUGUUUCUAAACCCAUGGGUACACAGACACAUACUAUGAUAUUCGAUAAUGCAUUUAAAUGCACUUUCGAGUACAUAUCU - example_title: insulin mask_index: 12 mask_index_1based: 13 masked_char: A output: - label: UGCC score: 0.99504 - label: UGGC score: 0.0047 - label: GGCC score: 0.000155 - label: UGUC score: 2.0e-05 - label: AGCC score: 2.0e-05 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGGCCCUGUGGGCCUCCUGCCCCUGCUGGCGCUGCUGGCCCUCUGGGGACCUGACCCAGCCGCAGCCUUUGUGAACCAACACCUGUGCGGCUCACACCUGGUGGAAGCUCUCUACCUAGUGUGCGGGGAACGAGGCUUCUUCUACACACCCAAGACCCGCCGGGAGGCAGAGGACCUGCAGGUGGGGCAGGUGGAGCUGGGCGGGGGCCCUGGUGCAGGCAGCCUGCAGCCCUUGGCCCUGGAGGGGUCCCUGCAGAAGCGUGGCAUUGUGGAACAAUGCUGUACCAGCAUCUGCUCCCUCUACCAGCUGGAGAACUACUGCAACUAG - example_title: cyclin dependent kinase inhibitor 2A mask_index: 18 mask_index_1based: 19 masked_char: A output: - label: GGCA score: 0.997962 - label: GGGA score: 0.00186 - label: AGCA score: 3.7e-05 - label: CGCA score: 3.7e-05 - label: UGCA score: 2.4e-05 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGGAGCCGGCGGCGGGGGCAUGGAGCCUUCGGCUGACUGGCUGGCCACGGCCGCGGCCCGGGGUCGGGUAGAGGAGGUGCGGGCGCUGCUGGAGGCGGGGGCGCUGCCCAACGCACCGAAUAGUUACGGUCGGAGGCCGAUCCAGGUCAUGAUGAUGGGCAGCGCCCGAGUGGCGGAGCUGCUGCUGCUCCACGGCGCGGAGCCCAACUGCGCCGACCCCGCCACUCUCACCCGACCCGUGCACGACGCUGCCCGGGAGGGCUUCCUGGACACGCUGGUGGUGCUGCACCGGGCCGGGGCGCGGCUGGACGUGCGCGAUGCCUGGGGCCGUCUGCCCGUGGACCUGGCUGAGGAGCUGGGCCAUCGCGAUGUCGCACGGUACCUGCGCGCGGCUGCGGGGGGCACCAGAGGCAGUAACCAUGCCCGCAUAGAUGCCGCGGAAGGUCCCUCAGACAUCCCCGAUUGA - example_title: human papillomavirus type 16 E6 mask_index: 10 mask_index_1based: 11 masked_char: A output: - label: AAAC score: 0.999972 - label: CAAC score: 6.0e-06 - label: GAAC score: 3.0e-06 - label: UAAC score: 3.0e-06 - label: AAAA score: 2.0e-06 pipeline_tag: fill-mask sequence_type: mRNA task: fill-mask text: AUGCACCAAAAACUGCAAUGUUUCAGGACCCACAGGAGCGACCCAGAAAGUUACCACAGUUAUGCACAGAGCUGCAAACAACUAUACAUGAUAUAAUAUUAGAAUGUGUGUACUGCAAGCAACAGUUACUGCGACGUGAGGUAUAUGACUUUGCUUUUCGGGAUUUAUGCAUAGUAUAUAGAGAUGGGAAUCCAUAUGCUGUAUGUGAUAAAUGUUUAAAGUUUUAUUCUAAAAUUAGUGAGUAUAGACAUUAUUGUUAUAGUUUGUAUGGAACAACAUUAGAACAGCAAUACAACAAACCGUUGUGUGAUUUGUUAAUUAGGUGUAUUAACUGUCAAAAGCCACUGUGUCCUGAAGAAAAGCAAAGACAUCUGGACAAAAAGCAAAGAUUCCAUAAUAUAAGGGGUCGGUGGACCGGUCGAUGUAUGUCUUGUUGCAGAUCAUCAAGAACACGUAGAGAAACCCAGCUGUAA - example_title: NRAS proto-oncogene mask_index: 36 mask_index_1based: 37 masked_char: A output: - label: UUGG score: 0.650678 - label: UCGG score: 0.254423 - label: UUCG score: 0.062639 - label: GCGG score: 0.007477 - label: UGGG score: 0.005193 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: GGGGCCGGAAGUGCCGCUCCUUGGUGGGGGCUGUUCCGGUUCCGGGGUCUCCAACAUUUUUCCCGGCUGUGGUCCUAAAUCUGUCCAAAGCAGAGGCAGUGGAGCUUGAGGUUCUUGCUGGUGUGAA - example_title: amyloid beta precursor protein mask_index: 15 mask_index_1based: 16 masked_char: A output: - label: GGUA score: 0.962511 - label: GGCA score: 0.035258 - label: GGCC score: 0.000472 - label: GGGA score: 0.000449 - label: GGCU score: 0.000266 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: GUCAGUUUCCUCGGCGUAGGCGAGAGCACGCGGAGGAGCGUGCGCGGGGGCCCCGGGAGACGGCGGCGGUGGCGGCGCGGGCAGAGCAAGGACGCGGCGGAUCCCACUCGCACAGCAGCGCACUCGGUGCCCCGCGCAGGGUCGCG - example_title: RUNX family transcription factor 1 mask_index: 15 mask_index_1based: 16 masked_char: A output: - label: CUCA score: 0.992773 - label: CACA score: 0.006718 - label: AACA score: 7.0e-05 - label: CCCA score: 6.1e-05 - label: UACA score: 4.6e-05 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: ACUUCUUUGGGCCUCACAACCACAGAACCACAAGUUGGGUAGCCUGGCAGUGUCAGAAGUCUGAACCCAGCAUAGUGGUCAGCAGGCAGGACGAAUCACACUGAAUGCAAACCACAGGGUUUCGCAGCGUGGUAAAAGAAAUCAUUGAGUCCCCCGCCUUCAGAAGAGGGUGCAUUUUCAGGAGGAAGCG - example_title: fragile X messenger ribonucleoprotein 1 mask_index: 15 mask_index_1based: 16 masked_char: A output: - label: CUCG score: 0.996774 - label: CCCG score: 0.002748 - label: CGCG score: 0.000108 - label: GCCG score: 7.3e-05 - label: CUCU score: 7.1e-05 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: CUCAGUCAGGCGCUCCCGUUUCGGUUUCACUUCCGGUGGAGGGCCGCCUCUGAGCGGGCGGCGGGCCGACGGCGAGCGCGGGCGGCGGCGGUGACGGAGGCGCCGCUGCCAGGGGGCGUGCGGCAGCGCGGCGGCGGCGGCGGCGGCGGCGGCGGCGGAGGCGGCGGCGGCGGCGGCGGCGGCGGCGGCUGGGCCUCGAGCGCCCGCAGCCCACCUCUCGGGGGCGGGCUCCCGGCGCUAGCAGGGCUGAAGAGAAG - example_title: MYC proto-oncogene mask_index: 10 mask_index_1based: 11 masked_char: A output: - label: UGUU score: 0.99518 - label: UAUU score: 0.003561 - label: UCUU score: 0.000147 - label: UUUU score: 0.000118 - label: GAUU score: 0.000101 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: AACUCGCUGUAUUCCAGCGAGAGGCAGAGGGAGCGAGCGGGCGGCCGGCUAGGGUGGAAGAGCCGGGCGAGCAGAGCUGCGCUGCGGGCGUCCUGGGAAGGGAGAUCCGGAGCGAAUAGGGGGCUUCGCCUCUGGCCCAGCCCUCCCGCUGAUCCCCCAGCCAGCGGUCCGCAACCCUUGCCGCAUCCACGAAACUUUGCCCAUAGCAGCGGGCGGGCACUUUGCACUGGAACUUACAACACCCGAGCAAGGACGCGACUCUCCCGACGCGGGGAGGCUAUUCUGCCCAUUUGGGGACACUUCCCCGCCGCUGCCAGGACCCGCUUCUCUGAAAGGCUCUCCUUGCAGCUGCUUAGACG - example_title: activating transcription factor 4 mask_index: 20 mask_index_1based: 21 masked_char: A output: - label: CCUG score: 0.518725 - label: CAUG score: 0.255265 - label: CCCG score: 0.190836 - label: AAUG score: 0.007829 - label: GAUG score: 0.00523 pipeline_tag: fill-mask sequence_type: 5' UTR task: fill-mask text: CAUUUCUACUUUGCCCGCCCAUGUAGUUUUCUCUGCGCGUGUGCGUUUUCCCUCCUCCCCGCCCUCAGGGUCCACGGCCACCAUGGCGUAUUAGGGGCAGCAGUGCCUGCGGCAGCAUUGGCCUUUGCAGCGGCGGCAGCAGCACCAGGCUCUGCAGCGGCAACCCCCAGCGGCUUAAGCCAUGGCGCUUCUCACGGCAUUCAGCAGCAGCGUUGCUGUAACCGACAAAGACACCUUCGAAUUAAGCACAUUCCUCGAUUCCAGCAAAGCACCGCAAC - example_title: Human GPI protein p137 mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: AGAU score: 0.99574 - label: AGGU score: 0.003927 - label: GGAU score: 5.1e-05 - label: UGAU score: 5.1e-05 - label: AGGC score: 5.0e-05 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: UUUUUAAAAGGGAUACCAAAUGCCUGCUGCUACCACCCUUUUCAAUUGCUAUGUUUUGAAAGGCACCAGUAUGUGUUUUAGAUUGAUUUAAAUGUUUCAUUUAAAUCACGGACAGUAGUUUCAGUUCUGAUGGUAUAAGCAAAACAAAUAAAACGUUUAUAAAAGUUGUAUCUUGAAACACUGGUGUUCAACAGCUAGCAGCUUAUGUGAUUCACCCCAUGCCACGUUAGUGUCACAAAUUUUAUGGUUUAUCUCCAGCAACAUUUCUCUAGUACUUGCACUUAUUAUCUGAAUUC - example_title: nucleophosmin 1 mask_index: 11 mask_index_1based: 12 masked_char: A output: - label: UUUU score: 0.413889 - label: UUAU score: 0.408688 - label: UAAU score: 0.104135 - label: UUGU score: 0.00231 - label: UUUA score: 0.002118 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: GAAAAUAGUUUAAUUUGUUAAAAAAUUUUCCGUCUUAUUUCAUUUCUGUAACAGUUGAUAUCUGGCUGUCCUUUUUAUAAUGCAGAGUGAGAACUUUCCCUACCGUGUUUGAUAAAUGUUGUCCAGGUUCUAUUGCCAAGAAUGUGUUGUCCAAAAUGCCUGUUUAGUUUUUAAAGAUGGAACUCCACCCUUUGCUUGGUUUUAAGUAUGUAUGGAAUGUUAUGAUAGGACAUAGUAGUAGCGGUGGUCAGACAUGGAAAUGGUGGGGAGACAAAAAUAUACAUGUGAAAUAAAACUCAGUAUUUUAAUAAAGUAGCACGGUUUCUAUUGA - example_title: superoxide dismutase 1 mask_index: 12 mask_index_1based: 13 masked_char: A output: - label: UGGU score: 0.987416 - label: UAGU score: 0.011428 - label: AAGU score: 0.000144 - label: GAGU score: 0.000135 - label: UUGU score: 0.0001 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: ACAUUCCCUUGGAGUCUGAGGCCCCUUAACUCAUCUGUUAUCCUGCUAGCUGUAGAAAUGUAUCCUGAUAAACAUUAAACACUGUAAUCUUAAAAGUGUAAUUGUGUGACUUUUUCAGAGUUGCUUUAAAGUACCUGUAGUGAGAAACUGAUUUAUGAUCACUUGGAAGAUUUGUAUAGUUUUAUAAAACUCAGUUAAAAUGUCUGUUUCAAUGACCUGUAUUUUGCCAGACUUAAAUCACAGAUGGGUAUUAAACUUGUCAGAAUUUCUUUGUCAUUCAAGCCUGUGAAUAAAAACCCUGUAUGGCACUUAUUAUGAGGCUAUUAAAAGAAUCCAAAUUCAAACUAAA - example_title: hemoglobin subunit alpha 2 mask_index: 13 mask_index_1based: 14 masked_char: A output: - label: GGUU score: 0.999995 - label: UGUU score: 2.0e-06 - label: AGUU score: 2.0e-06 - label: GGUC score: 1.0e-06 - label: CGUU score: 0.0 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: CUGGAGCCUCGGUGUUCCUCCUGCCCGCUGGGCCUCCCAACGGGCCCUCCUCCCCUCCUUGCACCGGCCCUUCCUGGUCUUUGAAUAAAGUCUGAGUGGGCAGCA - example_title: BRAF proto-oncogene mask_index: 12 mask_index_1based: 13 masked_char: A output: - label: GUGU score: 0.987349 - label: GAGU score: 0.011629 - label: AAGU score: 0.000134 - label: UAGU score: 7.5e-05 - label: CAGU score: 5.2e-05 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: AACAAAUGAGUGAGUUCAGGAGAGUAGCAACAAAAGGAAAAUAAAUGAACAUAUGUUUGCUUAUAUGUUAAAUUGAAUAAAAUACUCUCUUUUUUUUUAAGGUGAACCAAAGAACACUUGUGUGGUUAAAGACUAGAUAUAAUUUUUCCCCAAACUAAAAUUUAUACUUAACAUUGGAUUUUUAACAUCCAAGGGUUAAAAUACAUAGACAUUGCUAAAAAUUGGCAGAGCCUCUUCUAGAGGCUUUACUUUCUGUUCCGGGUUUGUAUCAUUCACUUGGUUAUUUUAAGUAGUAAACUUCAGUUUCUCAUGCAACUUUUGUUGCCAGCUAUCACAUGUCCACUAGGGACUCCAGAAGAAGACCCUACCUAUGCCUGUGUUUGCAGGUGAGAAGUUGGCAGUCGGUUAGCCUGGG - example_title: H3 clustered histone 1 mask_index: 17 mask_index_1based: 18 masked_char: A output: - label: CUCC score: 0.996808 - label: UUCC score: 0.002306 - label: CUGC score: 0.000562 - label: AUCC score: 0.000192 - label: GUCC score: 9.1e-05 pipeline_tag: fill-mask sequence_type: 3' UTR task: fill-mask text: UUACUGUGGUCUCUCUGUCCAAGCAAAGGCUCUUUUCAGAGCCACCACCUUUUC --- # 3UTRBERT Pre-trained model on 3’ untranslated region (3’UTR) using a masked language modeling (MLM) objective. ## Disclaimer This is an UNOFFICIAL implementation of the [Deciphering 3’ UTR mediated gene regulation using interpretable deep representation learning](https://doi.org/10.1101/2023.09.08.556883) by Yuning Yang, Gen Li, et al. The OFFICIAL repository of 3UTRBERT is at [yangyn533/3UTRBERT](https://github.com/yangyn533/3UTRBERT). > [!TIP] > The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation. **The team releasing 3UTRBERT did not write this model card for this model so this model card has been written by the MultiMolecule team.** ## Model Details 3UTRBERT is a [bert](https://huggingface.co/google-bert/bert-base-uncased)-style model pre-trained on a large corpus of 3’ untranslated regions (3’UTRs) in a self-supervised fashion. This means that the model was trained on the raw nucleotides of RNA sequences only, with an automatic process to generate inputs and labels from those texts. Please refer to the [Training Details](#training-details) section for more information on the training process. ### Variants - **[multimolecule/utrbert-3mer](https://huggingface.co/multimolecule/utrbert-3mer)**: The 3UTRBERT model pre-trained on 3-mer data. - **[multimolecule/utrbert-4mer](https://huggingface.co/multimolecule/utrbert-4mer)**: The 3UTRBERT model pre-trained on 4-mer data. - **[multimolecule/utrbert-5mer](https://huggingface.co/multimolecule/utrbert-5mer)**: The 3UTRBERT model pre-trained on 5-mer data. - **[multimolecule/utrbert-6mer](https://huggingface.co/multimolecule/utrbert-6mer)**: The 3UTRBERT model pre-trained on 6-mer data. ### Model Specification
Variants Num Layers Hidden Size Num Heads Intermediate Size Num Parameters (M) FLOPs (G) MACs (G) Max Num Tokens
3UTRBERT-6mer 12 768 12 3072 98.05 96.86 48.32 512
3UTRBERT-5mer 88.45
3UTRBERT-4mer 86.53
3UTRBERT-3mer 86.14
### Links - **Code**: [multimolecule.utrbert](https://github.com/DLS5-Omics/multimolecule/tree/master/multimolecule/models/utrbert) - **Data**: [multimolecule/gencode-human](https://huggingface.co/datasets/multimolecule/gencode-human) - **Paper**: [Deciphering 3’ UTR mediated gene regulation using interpretable deep representation learning](https://doi.org/10.1101/2023.09.08.556883) - **Developed by**: Yuning Yang, Gen Li, Kuan Pang, Wuxinhao Cao, Xiangtao Li, Zhaolei Zhang - **Model type**: [BERT](https://huggingface.co/google-bert/bert-base-uncased) - **Original Repository**: [yangyn533/3UTRBERT](https://github.com/yangyn533/3UTRBERT) ## Usage The model file depends on the [`multimolecule`](https://multimolecule.danling.org) library. You can install it using pip: ```bash pip install multimolecule ``` ### Direct Use #### Masked Language Modeling > [!WARNING] > Default transformers pipeline does not support K-mer tokenization. You can use this model directly with a pipeline for masked language modeling: ```python import multimolecule # you must import multimolecule to register models from transformers import pipeline predictor = pipeline("fill-mask", model="multimolecule/utrbert-3mer") output = predictor("gguccugguuagaccagaucugagccu")[1] ``` ### Downstream Use #### Extract Features Here is how to use this model to get the features of a given sequence in PyTorch: ```python from multimolecule import RnaTokenizer, UtrBertModel tokenizer = RnaTokenizer.from_pretrained("multimolecule/utrbert-3mer") model = UtrBertModel.from_pretrained("multimolecule/utrbert-3mer") text = "UAGCUUAUCAGACUGAUGUUG" input = tokenizer(text, return_tensors="pt") output = model(**input) ``` #### Sequence Classification / Regression > [!NOTE] > This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression. Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch: ```python import torch from multimolecule import RnaTokenizer, UtrBertForSequencePrediction tokenizer = RnaTokenizer.from_pretrained("multimolecule/utrbert-3mer") model = UtrBertForSequencePrediction.from_pretrained("multimolecule/utrbert-3mer") text = "UAGCUUAUCAGACUGAUGUUG" input = tokenizer(text, return_tensors="pt") label = torch.tensor([1]) output = model(**input, labels=label) ``` #### Token Classification / Regression > [!NOTE] > This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for token classification or regression. Here is how to use this model as backbone to fine-tune for a nucleotide-level task in PyTorch: ```python import torch from multimolecule import RnaTokenizer, UtrBertForTokenPrediction tokenizer = RnaTokenizer.from_pretrained("multimolecule/utrbert-3mer") model = UtrBertForTokenPrediction.from_pretrained("multimolecule/utrbert-3mer") text = "UAGCUUAUCAGACUGAUGUUG" input = tokenizer(text, return_tensors="pt") label = torch.randint(2, (len(text), )) output = model(**input, labels=label) ``` #### Contact Classification / Regression > [!NOTE] > This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for contact classification or regression. Here is how to use this model as backbone to fine-tune for a contact-level task in PyTorch: ```python import torch from multimolecule import RnaTokenizer, UtrBertForContactPrediction tokenizer = RnaTokenizer.from_pretrained("multimolecule/utrbert-3mer") model = UtrBertForContactPrediction.from_pretrained("multimolecule/utrbert-3mer") text = "UAGCUUAUCAGACUGAUGUUG" input = tokenizer(text, return_tensors="pt") label = torch.randint(2, (len(text), len(text))) output = model(**input, labels=label) ``` ## Training Details 3UTRBERT used Masked Language Modeling (MLM) as the pre-training objective: taking a sequence, the model randomly masks 15% of the tokens in the input then runs the entire masked sentence through the model and has to predict the masked tokens. This is comparable to the Cloze task in language modeling. ### Training Data The 3UTRBERT model was pre-trained on human mRNA transcript sequences from [GENCODE](https://gencodegenes.org). GENCODE aims to identify all gene features in the human genome using a combination of computational analysis, manual annotation, and experimental validation. The GENCODE release 40 used by this work contains 61,544 genes, and 246,624 transcripts. 3UTRBERT collected the human mRNA transcript sequences from GENCODE, including 108,573 unique mRNA transcripts. Only the longest transcript of each gene was used in the pre-training process. 3UTRBERT only used the 3’ untranslated regions (3’UTRs) of the mRNA transcripts for pre-training to avoid codon constrains in the CDS region, and to reduce increased complexity of the entire mRNA transcripts. The average length of the 3’UTRs was 1,227 nucleotides, while the median length was 631 nucleotides. Each 3’UTR sequence was cut to non-overlapping patches of 510 nucleotides. The remaining sequences were padded to the same length. Note [`RnaTokenizer`][multimolecule.RnaTokenizer] will convert "T"s to "U"s for you, you may disable this behaviour by passing `replace_T_with_U=False`. ### Training Procedure #### Preprocessing 3UTRBERT used masked language modeling (MLM) as the pre-training objective. The masking procedure is similar to the one used in BERT: - Mask rate: 15% - Replacement: `` for 80% of masked tokens - Replacement: random token for 10% of masked tokens - Replacement: unchanged token for 10% of masked tokens Since 3UTRBERT used k-mer tokenizer, it masks the entire k-mer instead of individual nucleotides to avoid information leakage. For example, if the k-mer is 3, the sequence `"UAGCGUAU"` will be tokenized as `["UAG", "AGC", "GCG", "CGU", "GUA", "UAU"]`. If the nucleotide `"C"` is masked, the adjacent tokens will also be masked, resulting `["UAG", "", "", "", "GUA", "UAU"]`. #### Pre-training The model was trained on 4 NVIDIA Quadro RTX 6000 GPUs with 24GiB memories. - Batch size: 128 - Steps: 200,000 - Optimizer: AdamW(β1=0.9, β2=0.98, e=1e-6) - Learning rate: 3e-4 - Learning rate scheduler: Linear - Learning rate warm-up: 10,000 steps - Weight decay: 0.01 ## Citation ```bibtex @article {yang2023deciphering, author = {Yang, Yuning and Li, Gen and Pang, Kuan and Cao, Wuxinhao and Li, Xiangtao and Zhang, Zhaolei}, title = {Deciphering 3{\textquoteright} UTR mediated gene regulation using interpretable deep representation learning}, elocation-id = {2023.09.08.556883}, year = {2023}, doi = {10.1101/2023.09.08.556883}, publisher = {Cold Spring Harbor Laboratory}, abstract = {The 3{\textquoteright}untranslated regions (3{\textquoteright}UTRs) of messenger RNAs contain many important cis-regulatory elements that are under functional and evolutionary constraints. We hypothesize that these constraints are similar to grammars and syntaxes in human languages and can be modeled by advanced natural language models such as Transformers, which has been very effective in modeling protein sequence and structures. Here we describe 3UTRBERT, which implements an attention-based language model, i.e., Bidirectional Encoder Representations from Transformers (BERT). 3UTRBERT was pre-trained on aggregated 3{\textquoteright}UTR sequences of human mRNAs in a task-agnostic manner; the pre-trained model was then fine-tuned for specific downstream tasks such as predicting RBP binding sites, m6A RNA modification sites, and predicting RNA sub-cellular localizations. Benchmark results showed that 3UTRBERT generally outperformed other contemporary methods in each of these tasks. We also showed that the self-attention mechanism within 3UTRBERT allows direct visualization of the semantic relationship between sequence elements.Competing Interest StatementThe authors have declared no competing interest.}, URL = {https://www.biorxiv.org/content/early/2023/09/12/2023.09.08.556883}, eprint = {https://www.biorxiv.org/content/early/2023/09/12/2023.09.08.556883.full.pdf}, journal = {bioRxiv} } ``` > [!NOTE] > The artifacts distributed in this repository are part of the MultiMolecule project. > If MultiMolecule supports your research, please cite the MultiMolecule project as follows: ```bibtex @software{chen_2024_12638419, author = {Chen, Zhiyuan and Zhu, Sophia Y.}, title = {MultiMolecule}, doi = {10.5281/zenodo.12638419}, publisher = {Zenodo}, url = {https://doi.org/10.5281/zenodo.12638419}, year = 2024, month = may, day = 4 } ``` ## Contact Please use GitHub issues of [MultiMolecule](https://github.com/DLS5-Omics/multimolecule/issues) for any questions or comments on the model card. Please contact the authors of the [3UTRBERT paper](https://doi.org/10.1101/2023.09.08.556883) for questions or comments on the paper/model. ## License This model implementation is licensed under the [GNU Affero General Public License](license.md). For additional terms and clarifications, please refer to our [License FAQ](license-faq.md). ```spdx SPDX-License-Identifier: AGPL-3.0-or-later ```