Hugging Face's logo

NbAiLab
/
nb-tokenizers-spm

Model card Files
xet
Community
versae commited on
Commit
0ed8b17
·
1 Parent(s): 79c0482

Adding tokenizers models and vocabs

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. oscar/da_16000_bpe.100extra.sp.model +3 -0
  2. oscar/da_16000_bpe.100extra.sp.vocab +0 -0
  3. oscar/da_16000_bpe.sp.model +3 -0
  4. oscar/da_16000_bpe.sp.vocab +0 -0
  5. oscar/da_16000_unigram.100extra.sp.model +3 -0
  6. oscar/da_16000_unigram.100extra.sp.vocab +0 -0
  7. oscar/da_16000_unigram.sp.model +3 -0
  8. oscar/da_16000_unigram.sp.vocab +0 -0
  9. oscar/da_32000_bpe.100extra.sp.model +3 -0
  10. oscar/da_32000_bpe.100extra.sp.vocab +0 -0
  11. oscar/da_32000_bpe.sp.model +3 -0
  12. oscar/da_32000_bpe.sp.vocab +0 -0
  13. oscar/da_32000_unigram.100extra.sp.model +3 -0
  14. oscar/da_32000_unigram.100extra.sp.vocab +0 -0
  15. oscar/da_32000_unigram.sp.model +3 -0
  16. oscar/da_32000_unigram.sp.vocab +0 -0
  17. oscar/da_64000_bpe.100extra.sp.model +3 -0
  18. oscar/da_64000_bpe.100extra.sp.vocab +0 -0
  19. oscar/da_64000_bpe.sp.model +3 -0
  20. oscar/da_64000_bpe.sp.vocab +0 -0
  21. oscar/da_64000_unigram.100extra.sp.model +3 -0
  22. oscar/da_64000_unigram.100extra.sp.vocab +0 -0
  23. oscar/da_64000_unigram.sp.model +3 -0
  24. oscar/da_64000_unigram.sp.vocab +0 -0
  25. oscar/de_16000_bpe.100extra.sp.model +3 -0
  26. oscar/de_16000_bpe.100extra.sp.vocab +0 -0
  27. oscar/de_16000_bpe.sp.model +3 -0
  28. oscar/de_16000_bpe.sp.vocab +0 -0
  29. oscar/de_16000_unigram.100extra.sp.model +3 -0
  30. oscar/de_16000_unigram.100extra.sp.vocab +0 -0
  31. oscar/de_16000_unigram.sp.model +3 -0
  32. oscar/de_16000_unigram.sp.vocab +0 -0
  33. oscar/de_32000_bpe.100extra.sp.model +3 -0
  34. oscar/de_32000_bpe.100extra.sp.vocab +0 -0
  35. oscar/de_32000_bpe.sp.model +3 -0
  36. oscar/de_32000_bpe.sp.vocab +0 -0
  37. oscar/de_32000_unigram.100extra.sp.model +3 -0
  38. oscar/de_32000_unigram.100extra.sp.vocab +0 -0
  39. oscar/de_32000_unigram.sp.model +3 -0
  40. oscar/de_32000_unigram.sp.vocab +0 -0
  41. oscar/de_64000_bpe.100extra.sp.model +3 -0
  42. oscar/de_64000_bpe.100extra.sp.vocab +0 -0
  43. oscar/de_64000_bpe.sp.model +3 -0
  44. oscar/de_64000_bpe.sp.vocab +0 -0
  45. oscar/de_64000_unigram.100extra.sp.model +3 -0
  46. oscar/de_64000_unigram.100extra.sp.vocab +0 -0
  47. oscar/de_64000_unigram.sp.model +3 -0
  48. oscar/de_64000_unigram.sp.vocab +0 -0
  49. oscar/en_16000_bpe.100extra.sp.model +3 -0
  50. oscar/en_16000_bpe.100extra.sp.vocab +0 -0
oscar/da_16000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce32f8bc474ae09e067d58b1dbaa83eb83fe20ae00b30d0f8ea83519c9be1f86
3
+ size 471306
oscar/da_16000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_16000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9cd7bc00cd948bc6149f5733812db35ad9990f9dea0e745c691329624fc449ff
3
+ size 469035
oscar/da_16000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_16000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5503f54a115b6ebc5a6c3b997bce5e28b2de2bab33a3516e413ada55b6dbb19d
3
+ size 479111
oscar/da_16000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_16000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36a2111ac0d5c323ea4dfcd5e424dde69155d8275a50f451d05103bd30002adf
3
+ size 477003
oscar/da_16000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_32000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:196b636d80745161e7fdfa2ecab6ad37812bccebcb1dd375c8cf2ee724021703
3
+ size 750450
oscar/da_32000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_32000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5fbb50bbb377d4366054652bc213c872960b0c095636065e99005c037987c344
3
+ size 748187
oscar/da_32000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_32000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8134162f91bd93b01f0bb4fb52f5096e87324fa933ff93678eaf61f8233b8186
3
+ size 769374
oscar/da_32000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_32000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62e17420c8be8ce11c61b73e10a55763dd752625bcecabb2b61c45db096d9134
3
+ size 767347
oscar/da_32000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_64000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07cc3dc49d4e1cf0c2a814aa0719f202e6bae8199d25c4a8ce0d66ff97913439
3
+ size 1330189
oscar/da_64000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_64000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbef2a597f0d976fc10ce73e587d5f0b93d3f332678e0d045597da1240364fc8
3
+ size 1328134
oscar/da_64000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_64000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e79796b6337d10cba75b01d372eaec14a42de0e206340d0452c906bbf0800d4f
3
+ size 1375269
oscar/da_64000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/da_64000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e60f1ca203be353c44258c9b4782017475e1e3860aa7bd60f7c0f569e1bee2a
3
+ size 1373305
oscar/da_64000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_16000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f52f7c8542fb094ec0be1ed19c1c53adc9110e2e7a94634ae58f1870163f015
3
+ size 465249
oscar/de_16000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_16000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36127ffd25b71f08abe993603ada7feb87e13a352afbdce9c6103c71852a08ef
3
+ size 462959
oscar/de_16000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_16000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d317b61ad6001a17f3abbb1bdc88f7e13f538a68e85d9ed20e67c1672dc92e16
3
+ size 472434
oscar/de_16000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_16000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f47177a89f0d0999cc9b283cf7598e72c4fe5395f0b82e5f304b54526293e224
3
+ size 470329
oscar/de_16000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_32000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa89cb185e5264b338b0a460ea3703518d61e71ad3ce67043a95aadc8431065b
3
+ size 744223
oscar/de_32000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_32000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e69c7d32b5edb7ec8f59cee17f4c281cc1a31dac4afbb8213d52f603c2cdf3cf
3
+ size 741953
oscar/de_32000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_32000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:485e82ee573fcdf6b84cdce679da83a185aafe86770f22dffddad2b9f1c481ef
3
+ size 764812
oscar/de_32000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_32000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68a1f229dedd7c4ef41b6925457ccbaad4e0f29bb74d97aee1c8ee1c8a9fb8e9
3
+ size 762875
oscar/de_32000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_64000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:187d4196adbe257712093b6db939f4a2de47a8c84553a09c0dad29e1838de132
3
+ size 1327631
oscar/de_64000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_64000_bpe.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d3df98514964d3e2126b13a9a003101f24fd108d9312e7e3467426a1550ccbf7
3
+ size 1325534
oscar/de_64000_bpe.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_64000_unigram.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:760a063b4840b36e66fba7fe8b096f32289141fa6cee3f07aa94b44a93d22947
3
+ size 1371257
oscar/de_64000_unigram.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/de_64000_unigram.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:86bd25c449754a73691d7cb1a64abaa81264bd36810ebc4ceef48cef2c6f8667
3
+ size 1369281
oscar/de_64000_unigram.sp.vocab ADDED
The diff for this file is too large to render. See raw diff
 
oscar/en_16000_bpe.100extra.sp.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa285b1aa3c0a379d8f9cbed7f9862a10733fe694d9120b29349c10652acf29c
3
+ size 458849
oscar/en_16000_bpe.100extra.sp.vocab ADDED
The diff for this file is too large to render. See raw diff