Upload all models and assets for af (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.193
+    value: 4.620
   - name: best_isotropy
     type: isotropy
-    value: 0.6686
+    value: 0.6959
   - name: vocabulary_size
     type: vocab
-    value: 426702
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AF - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,63 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.512x | 3.48 | 0.0703% | 1,403,468 |
-| **16k** | 3.805x | 3.77 | 0.0762% | 1,295,239 |
-| **32k** | 4.029x | 3.99 | 0.0807% | 1,223,476 |
-| **64k** | 4.193x 🏆 | 4.15 | 0.0840% | 1,175,389 |
+| **8k** | 3.747x | 3.75 | 0.0650% | 1,240,279 |
+| **16k** | 4.108x | 4.11 | 0.0712% | 1,131,351 |
+| **32k** | 4.402x | 4.40 | 0.0763% | 1,055,895 |
+| **64k** | 4.620x 🏆 | 4.62 | 0.0801% | 1,006,125 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Japan Nasionale Roete 461 is 'n nasionale snelweg in Japan.
-
- Verwysings 
-
-Kateg...`
+**Sample 1:** `Neede is ’n dorp in die munisipaliteit Berkelland in die provinsie Gelderland in...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁japan ▁nasionale ▁roete ▁ 4 6 1 ▁is ▁' n ... (+13 more)` | 23 |
-| 16k | `▁japan ▁nasionale ▁roete ▁ 4 6 1 ▁is ▁' n ... (+12 more)` | 22 |
-| 32k | `▁japan ▁nasionale ▁roete ▁ 4 6 1 ▁is ▁' n ... (+12 more)` | 22 |
-| 64k | `▁japan ▁nasionale ▁roete ▁ 4 6 1 ▁is ▁' n ... (+12 more)` | 22 |
+| 8k | `▁ne e de ▁is ▁’ n ▁dorp ▁in ▁die ▁munisipaliteit ... (+14 more)` | 24 |
+| 16k | `▁ne ede ▁is ▁’ n ▁dorp ▁in ▁die ▁munisipaliteit ▁berk ... (+13 more)` | 23 |
+| 32k | `▁ne ede ▁is ▁’ n ▁dorp ▁in ▁die ▁munisipaliteit ▁berk ... (+13 more)` | 23 |
+| 64k | `▁ne ede ▁is ▁’ n ▁dorp ▁in ▁die ▁munisipaliteit ▁berkelland ... (+12 more)` | 22 |
 
-**Sample 2:** `Japan Nasionale Roete 239 is 'n nasionale snelweg in Japan.
-
- Verwysings 
-
-Kateg...`
+**Sample 2:** `Japan Nasionale Roete 210 is 'n nasionale snelweg in Japan. Verwysings paaie in ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁japan ▁nasionale ▁roete ▁ 2 3 9 ▁is ▁' n ... (+13 more)` | 23 |
-| 16k | `▁japan ▁nasionale ▁roete ▁ 2 3 9 ▁is ▁' n ... (+12 more)` | 22 |
-| 32k | `▁japan ▁nasionale ▁roete ▁ 2 3 9 ▁is ▁' n ... (+12 more)` | 22 |
-| 64k | `▁japan ▁nasionale ▁roete ▁ 2 3 9 ▁is ▁' n ... (+12 more)` | 22 |
-
-**Sample 3:** `Japan Nasionale Roete 264 is 'n nasionale snelweg in Japan.
+| 8k | `▁japan ▁nasionale ▁roete ▁ 2 1 0 ▁is ▁' n ... (+9 more)` | 19 |
+| 16k | `▁japan ▁nasionale ▁roete ▁ 2 1 0 ▁is ▁' n ... (+9 more)` | 19 |
+| 32k | `▁japan ▁nasionale ▁roete ▁ 2 1 0 ▁is ▁' n ... (+9 more)` | 19 |
+| 64k | `▁japan ▁nasionale ▁roete ▁ 2 1 0 ▁is ▁' n ... (+9 more)` | 19 |
 
- Verwysings 
-
-Kateg...`
+**Sample 3:** `Ja'Net DuBois (gebore 5 Augustus – 17 Februarie was 'n Amerikaanse aktrise. Ekst...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁japan ▁nasionale ▁roete ▁ 2 6 4 ▁is ▁' n ... (+13 more)` | 23 |
-| 16k | `▁japan ▁nasionale ▁roete ▁ 2 6 4 ▁is ▁' n ... (+12 more)` | 22 |
-| 32k | `▁japan ▁nasionale ▁roete ▁ 2 6 4 ▁is ▁' n ... (+12 more)` | 22 |
-| 64k | `▁japan ▁nasionale ▁roete ▁ 2 6 4 ▁is ▁' n ... (+12 more)` | 22 |
+| 8k | `▁ja ' net ▁dub ois ▁( gebore ▁ 5 ▁augustus ... (+30 more)` | 40 |
+| 16k | `▁ja ' net ▁dub ois ▁( gebore ▁ 5 ▁augustus ... (+30 more)` | 40 |
+| 32k | `▁ja ' net ▁dub ois ▁( gebore ▁ 5 ▁augustus ... (+30 more)` | 40 |
+| 64k | `▁ja ' net ▁dubois ▁( gebore ▁ 5 ▁augustus ▁– ... (+29 more)` | 39 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.193x compression
-- **Lowest UNK Rate:** 8k with 0.0703% unknown tokens
+- **Best Compression:** 64k achieves 4.620x compression
+- **Lowest UNK Rate:** 8k with 0.0650% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,57 +129,89 @@ Kateg...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 54,654 🏆 | 15.74 | 884,766 | 16.1% | 32.6% |
-| **2-gram** | 304 🏆 | 8.25 | 15,856 | 65.2% | 98.7% |
-| **3-gram** | 293,194 | 18.16 | 2,174,878 | 7.1% | 19.5% |
-| **3-gram** | 2,671 | 11.38 | 128,050 | 26.3% | 68.2% |
-| **4-gram** | 729,597 | 19.48 | 4,032,808 | 5.3% | 15.8% |
-| **4-gram** | 15,548 | 13.92 | 732,288 | 14.1% | 38.0% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 67,018 | 16.03 | 738,183 | 13.7% | 29.1% |
+| **2-gram** | Subword | 253 🏆 | 7.98 | 13,576 | 69.5% | 99.3% |
+| **3-gram** | Word | 293,932 | 18.17 | 1,499,483 | 5.8% | 16.9% |
+| **3-gram** | Subword | 2,161 | 11.08 | 96,263 | 28.5% | 71.9% |
+| **4-gram** | Word | 555,388 | 19.08 | 2,510,434 | 6.5% | 16.6% |
+| **4-gram** | Subword | 12,658 | 13.63 | 531,540 | 15.0% | 40.0% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `van die` | 509,583 |
+| 2 | `in die` | 342,810 |
+| 3 | `is n` | 114,159 |
+| 4 | `en die` | 109,201 |
+| 5 | `is die` | 91,083 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `van suid afrika` | 26,860 |
+| 2 | `rolle in die` | 25,216 |
+| 3 | `die 20ste eeu` | 24,460 |
+| 4 | `van die 20ste` | 23,487 |
+| 5 | `eksterne skakels in` | 22,326 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `van die 20ste eeu` | 23,423 |
+| 2 | `manlike akteurs van die` | 20,397 |
+| 3 | `rolle in die rolprente` | 19,639 |
+| 4 | `van die 21ste eeu` | 15,799 |
+| 5 | `plants of the world` | 13,996 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `' n` | 602,690 |
-| 2 | `van die` | 511,614 |
-| 3 | `in die` | 336,247 |
-| 4 | `. die` | 312,198 |
-| 5 | `kategorie :` | 309,571 |
+| 1 | `e _` | 8,883,972 |
+| 2 | `n _` | 5,845,355 |
+| 3 | `i e` | 5,296,532 |
+| 4 | `e r` | 4,795,609 |
+| 5 | `_ d` | 4,496,380 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `is ' n` | 99,632 |
-| 2 | `suid - afrika` | 65,103 |
-| 3 | `kategorie : geboortes` | 42,714 |
-| 4 | `: geboortes in` | 42,714 |
-| 5 | `skakels kategorie :` | 36,080 |
+| 1 | `i e _` | 3,582,000 |
+| 2 | `_ d i` | 3,169,450 |
+| 3 | `d i e` | 3,046,581 |
+| 4 | `a n _` | 1,886,278 |
+| 5 | `e n _` | 1,538,281 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorie : geboortes in` | 42,714 |
-| 2 | `kategorie : flora van` | 35,480 |
-| 3 | `eksterne skakels kategorie :` | 34,452 |
-| 4 | `) is ' n` | 32,521 |
-| 5 | `van suid - afrika` | 27,043 |
+| 1 | `d i e _` | 2,916,346 |
+| 2 | `_ d i e` | 2,836,188 |
+| 3 | `_ v a n` | 1,357,382 |
+| 4 | `v a n _` | 1,341,795 |
+| 5 | `n _ d i` | 1,169,352 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 304
+- **Best Perplexity:** 2-gram (subword) with 253
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~38% of corpus
+- **Coverage:** Top-1000 patterns cover ~40% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -191,55 +219,86 @@ Kateg...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7872 | 1.726 | 8.26 | 1,012,367 | 21.3% |
-| **1** | 1.3494 | 2.548 | 8.73 | 6,082 | 0.0% |
-| **2** | 0.4249 | 1.342 | 2.60 | 8,362,191 | 57.5% |
-| **2** | 0.8337 | 1.782 | 5.41 | 53,087 | 16.6% |
-| **3** | 0.2059 | 1.153 | 1.52 | 21,746,451 | 79.4% |
-| **3** | 0.8222 | 1.768 | 4.50 | 287,186 | 17.8% |
-| **4** | 0.0978 🏆 | 1.070 | 1.19 | 33,034,213 | 90.2% |
-| **4** | 0.7160 🏆 | 1.643 | 3.45 | 1,293,510 | 28.4% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9426 | 1.922 | 9.97 | 884,548 | 5.7% |
+| **1** | Subword | 1.0721 | 2.102 | 6.58 | 7,654 | 0.0% |
+| **2** | Word | 0.3842 | 1.305 | 2.33 | 8,810,967 | 61.6% |
+| **2** | Subword | 0.7311 | 1.660 | 4.61 | 50,359 | 26.9% |
+| **3** | Word | 0.1707 | 1.126 | 1.40 | 20,525,798 | 82.9% |
+| **3** | Subword | 0.7061 | 1.631 | 4.02 | 231,918 | 29.4% |
+| **4** | Word | 0.0704 🏆 | 1.050 | 1.13 | 28,628,609 | 93.0% |
+| **4** | Subword | 0.6911 | 1.615 | 3.50 | 931,942 | 30.9% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `die ontwerp , babergh ( links | 297x297px | leigh het drie mense kategorie : eine`
-2. `. ( 1965 : " . die pos het op die vrye oefening 3 2 -`
-3. `, vacelet , belfast aangesluit . . die epi thēbas ; hierdie styl eskarp voor 1919`
+1. `die burger 13 augustus se fsa nommer vier soldate die uitkoms vir letterkunde in n klein`
+2. `van soest r amphoriscus cylindrus is in paradise careful he du mont dolent teen 5 6`
+3. `in die rigting van die twee broers en met die ou teeroete die liberte het die`
 
 **Context Size 2:**
 
-1. `' n wortelkelder te bêre ) is ’ n skynbare magnitude van 3 830 in 1793 het`
-2. `van die koninklike saoedi lugmag ( usaf ) het magtiging gehad om die aanstelling van chailly se`
-3. `in die tydperk tot in 1955 het motorola die elektroplateringsmetode ontwikkel , veral teenoor die st...`
+1. `van die spons behoort tot die genus geodia en tot die genus leucadendron behoort en is deur`
+2. `in die stille oseaan wat tot 4 uur later onder westerse intellektuele invloede gekom frankryk hertog...`
+3. `is n nuwe telling van 83 etse wat spesiaal vir hierdie liedjie is in die rolprente innerspace`
 
 **Context Size 3:**
 
-1. `is ' n aansteeklike siekte wat met die egiptiese heilige skrif , en regverdigheidsin daar nooit enig...`
-2. `suid - afrika internasionaal verteenwoordig en puik presteer . daar was ook geen ander kubistiese sk...`
-3. `: geboortes in 1917 kategorie : sterftes in 1932 kategorie : sterftes in 1547`
+1. `rolle in die rolprente the squaw man resurrección kongo the broken wing roaring rails en devils dice...`
+2. `van die 20ste eeu aktrises van die 21ste eeu mense aktrises van die 21ste eeu aktrises van die`
+3. `eksterne skakels in manlike akteurs van die 20ste eeu in n stormwind deur pieter kluyver wind is die`
 
 **Context Size 4:**
 
-1. `kategorie : geboortes in 1983 kategorie : engelse aktrises van die 21ste eeu kategorie : amerikaanse...`
-2. `kategorie : flora van egipte kategorie : mense in die tweede vryheidsoorlog . bron raper , peter edm...`
-3. `eksterne skakels kategorie : lewende mense kategorie : engelse manlike akteurs van die 20ste eeu kat...`
+1. `manlike akteurs van die 20ste eeu aktrises van die 20ste eeu rolprentvervaardigers in mense van die ...`
+2. `rolle in die rolprente tomorrow when the war began the weekend shift high life tidelands eksterne sk...`
+3. `plants of the world online van suid afrika plante van suid afrika gramineum`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_ho_evise_j._wom`
+2. `enstein_n_nkt_he`
+3. `igeked_dig_linid`
+
+**Context Size 2:**
+
+1. `e_nin:_sy_offie_l`
+2. `n_die_nivir_nbom_`
+3. `ie_uikaide_ver_ro`
+
+**Context Size 3:**
+
+1. `ie_van_die_bespelt`
+2. `_die_wassen_paropo`
+3. `die_van_spel_andar`
+
+**Context Size 4:**
+
+1. `die_vonnikeksadige_`
+2. `_die_branse_levisie`
+3. `_van_waar_toest,_r.`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 90.2% predictability
+- **Best Predictability:** Context-4 (word) with 93.0% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,293,510 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (931,942 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -255,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 426,702 |
-| Total Tokens | 41,728,231 |
-| Mean Frequency | 97.79 |
+| Vocabulary Size | 403,515 |
+| Total Tokens | 38,429,571 |
+| Mean Frequency | 95.24 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 6032.22 |
+| Frequency Std Dev | 6117.62 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | die | 2,854,279 |
-| 2 | van | 1,329,400 |
-| 3 | in | 1,119,312 |
-| 4 | en | 1,055,574 |
-| 5 | n | 811,721 |
-| 6 | is | 769,576 |
-| 7 | het | 649,041 |
-| 8 | wat | 345,008 |
-| 9 | kategorie | 310,570 |
-| 10 | the | 296,225 |
+| 1 | die | 2,828,931 |
+| 2 | van | 1,318,980 |
+| 3 | in | 1,109,973 |
+| 4 | en | 1,045,922 |
+| 5 | n | 802,080 |
+| 6 | is | 763,111 |
+| 7 | het | 641,876 |
+| 8 | wat | 341,748 |
+| 9 | the | 292,778 |
+| 10 | op | 289,154 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | cdmtcs | 2 |
-| 2 | researchreports | 2 |
-| 3 | akteurskategorieë | 2 |
-| 4 | mullens | 2 |
-| 5 | grafiekstruktuur | 2 |
-| 6 | roostergrafieke | 2 |
-| 7 | sokkerbekertitels | 2 |
-| 8 | chalobah | 2 |
-| 9 | sentrumverdediger | 2 |
-| 10 | guðjohnsen | 2 |
+| 1 | williamsville | 2 |
+| 2 | 1kd | 2 |
+| 3 | argiefkopie | 2 |
+| 4 | liuzhi | 2 |
+| 5 | microsat | 2 |
+| 6 | orbex | 2 |
+| 7 | afrikanertoekoms | 2 |
+| 8 | wêreldkennis | 2 |
+| 9 | gastebydraes | 2 |
+| 10 | sandkweek | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0756 |
-| R² (Goodness of Fit) | 0.994236 |
+| Zipf Coefficient | 1.0518 |
+| R² (Goodness of Fit) | 0.996010 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 41.8% |
-| Top 1,000 | 63.9% |
-| Top 5,000 | 79.6% |
-| Top 10,000 | 85.2% |
+| Top 100 | 43.7% |
+| Top 1,000 | 64.3% |
+| Top 5,000 | 79.4% |
+| Top 10,000 | 85.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9942 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 41.8% of corpus
-- **Long Tail:** 416,702 words needed for remaining 14.8% coverage
+- **Zipf Compliance:** R²=0.9960 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 43.7% of corpus
+- **Long Tail:** 393,515 words needed for remaining 15.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -325,24 +384,127 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 281,289 | 32 | 3.583 | 1.675 | 0.6651 |
-| **mono_64d** | 281,289 | 64 | 3.983 | 1.616 | 0.6686 🏆 |
-| **mono_128d** | 281,289 | 128 | 4.471 | 1.595 | 0.6515 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.6926 | 0.3664 | N/A | N/A |
+| **mono_64d** | 64 | 0.6959 🏆 | 0.3037 | N/A | N/A |
+| **mono_128d** | 128 | 0.6723 | 0.2366 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_64d with 0.6686 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 281,289 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_64d with 0.6959 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3023. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ge` | gewandel, gefloreer, getrouheidseed |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-e` | kurasse, kortikosteroïde, maatskappyname |
+| `-s` | tuttles, stakings, kenens |
+| `-er` | gefloreer, umbilorivier, koorsanger |
+| `-es` | tuttles, spectres, kladmetodes |
+| `-ng` | kruiskleding, saambring, swangerskapvergiftiging |
+| `-ie` | patagonie, photographie, kriminologie |
+| `-ing` | kruiskleding, saambring, swangerskapvergiftiging |
+| `-te` | monofisiete, skrikwekkendste, curriebekerpunte |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `pren` | 2.37x | 29 contexts | prent, prens, prend |
+| `staa` | 1.71x | 98 contexts | staat, staal, staan |
+| `ings` | 1.53x | 145 contexts | wings, rings, hings |
+| `brui` | 1.99x | 44 contexts | bruis, bruid, bruik |
+| `kend` | 1.65x | 95 contexts | kende, kendo, skend |
+| `ebru` | 2.08x | 32 contexts | gebru, hebrus, gebruk |
+| `ersk` | 1.54x | 107 contexts | perske, koersk, perski |
+| `erdi` | 1.61x | 84 contexts | verdi, ferdi, gerdi |
+| `rste` | 1.42x | 150 contexts | erste, eerste, fyrste |
+| `rdie` | 1.73x | 51 contexts | ardie, gordie, jordie |
+| `kste` | 1.54x | 71 contexts | ekster, tekste, dikste |
+| `eken` | 1.34x | 123 contexts | weken, deken, oeken |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ge` | `-e` | 63 words | geenlokusse, gebruikskode |
+| `-ge` | `-de` | 28 words | gebruikskode, geeboniseerde |
+| `-ge` | `-er` | 27 words | geigenspieler, getelegrafeer |
+| `-ge` | `-s` | 11 words | gemeentesusters, geles |
+| `-ge` | `-en` | 9 words | gefahren, gelegen |
+| `-ge` | `-te` | 6 words | geskenkte, geweldigste |
+| `-ge` | `-ie` | 5 words | getalteorie, geelglasogie |
+| `-ge` | `-es` | 4 words | geles, geowetenskaplikes |
+| `-ge` | `-ng` | 2 words | geeking, gesondheidsbevordering |
+| `-ge` | `-ing` | 1 words | geeking, gesondheidsbevordering |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| gemonteerde | **`ge-monte-er-de`** | 7.5 | `monte` |
+| bevredigende | **`bevredig-en-de`** | 6.0 | `bevredig` |
+| ouditering | **`oudit-er-ing`** | 6.0 | `oudit` |
+| kruiningen | **`kruin-ing-en`** | 6.0 | `kruin` |
+| verlorener | **`verlor-en-er`** | 6.0 | `verlor` |
+| verhardende | **`verhard-en-de`** | 6.0 | `verhard` |
+| bestuifde | **`bestuif-de`** | 4.5 | `bestuif` |
+| behoeften | **`behoeft-en`** | 4.5 | `behoeft` |
+| verminkte | **`vermink-te`** | 4.5 | `vermink` |
+| onreëlmatiger | **`onreëlmatig-er`** | 4.5 | `onreëlmatig` |
+| kollageen | **`kollage-en`** | 4.5 | `kollage` |
+| gekrummel | **`ge-krummel`** | 4.5 | `krummel` |
+| repeterende | **`repet-er-en-de`** | 4.5 | `repet` |
+| gehoorvermoë | **`ge-hoorvermoë`** | 4.5 | `hoorvermoë` |
+| eksoskelette | **`eksoskelet-te`** | 4.5 | `eksoskelet` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AF appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -350,11 +512,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.19x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (304) |
-| Markov | **Context-4** | Highest predictability (90.2%) |
+| Tokenizer | **64k BPE** | Best compression (4.62x) |
+| N-gram | **2-gram** | Lowest perplexity (253) |
+| Markov | **Context-4** | Highest predictability (93.0%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -544,7 +707,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -560,7 +724,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 05:32:18*
+*Report Date: 2026-01-03 07:17:29*

models/embeddings/monolingual/af_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1664aff332c7a349b9940cce2ba0d366d41d2cd3ceeddc6b04037c36e8b7ab30
-size 1317292760
+oid sha256:11057ead6d145fd80aa8a3489e8c8d86000071008dba47a7f9f04188e658acb2
+size 1301432503

models/embeddings/monolingual/af_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 281289
+  "vocab_size": 266109
 }

models/embeddings/monolingual/af_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c0e08a6e56419bc15cf41c317b15d8f38d74f34682f19a2349f9305f60573c26
-size 333262808
+oid sha256:933902af3f06d4dccbab2f9cda13414a48c3a2fafa734a3e6c9cc651ffed6732
+size 329060791

models/embeddings/monolingual/af_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 281289
+  "vocab_size": 266109
 }

models/embeddings/monolingual/af_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f17d4f4c541f08173c83987cd4b8539b48d2fa16f305ad30acc059e9b365735a
-size 661272792
+oid sha256:7c676e073cfbf1cb9b2e7f3e4b3e94463a17ee848c6d9df01ea4fc280e2f7ce6
+size 653184695

models/embeddings/monolingual/af_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 281289
+  "vocab_size": 266109
 }

models/subword_markov/af_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f42ef48ee495f26f7a3063da48c9ee906a4205528b9506ab33ed9fc982d9111c
-size 366862
+oid sha256:d149d8684419833d4d657f06ff43fde6f6330d2a78848a075e6c2fec57f7c4a8
+size 378608

models/subword_markov/af_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "af",
-  "unique_contexts": 6082,
-  "total_transitions": 268485278
+  "unique_contexts": 7654,
+  "total_transitions": 242297410
 }

models/subword_markov/af_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c91a87c97f14e34aeb8427bc36ed9584e434a40332a3f97910ffc78c5417d6b7
-size 2262231
+oid sha256:420f531b4f1b408ddda531f838f2d4932ecc9cbb7de16ed5589351bbace9b214
+size 1866665

models/subword_markov/af_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "af",
-  "unique_contexts": 53087,
-  "total_transitions": 268357949
+  "unique_contexts": 50359,
+  "total_transitions": 242170946
 }

models/subword_markov/af_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:994eb0d42a5e555f306278b50da3fd9f4347eccf479e465f5537de98e4429a31
-size 10525742
+oid sha256:075a8bbd37d56e56e0b0f55526f7b478207748bf68daf7f1981287409f6c5cb8
+size 8218278

models/subword_markov/af_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "af",
-  "unique_contexts": 287186,
-  "total_transitions": 268230620
+  "unique_contexts": 231918,
+  "total_transitions": 242044482
 }

models/subword_markov/af_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1cda0c6adb1b4b36a5836b96ba23dc2c9f880706859d0799461ae0a2a1d05e1a
-size 35196938
+oid sha256:e71c7cbca9385f2aabc8bff2108e5cb3876745099e692d102b8817bd6e393051
+size 26401415

models/subword_markov/af_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "af",
-  "unique_contexts": 1293510,
-  "total_transitions": 268103291
+  "unique_contexts": 931942,
+  "total_transitions": 241918018
 }

models/subword_ngram/af_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a314440c3a6118bd2558c0f5ae77b9b7567e26e683b45b28da4d7b61c5985153
-size 212926
+oid sha256:69bbe527269e41324fd2369bdb8f0798f0b62fd22d70026c15fbf678f77788e1
+size 182664

models/subword_ngram/af_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "af",
-  "unique_ngrams": 15856,
-  "total_ngrams": 268485278
+  "unique_ngrams": 13576,
+  "total_ngrams": 242297410
 }

models/subword_ngram/af_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b94502700c29b4beab32c042c6cbfb0121c688512ef843b491c7da0e3591e6a
-size 1557780
+oid sha256:8fcbc486b3961f9f14aee1492ea622129d621d205e682bab43457c4029f343ca
+size 1229801

models/subword_ngram/af_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "af",
-  "unique_ngrams": 128050,
-  "total_ngrams": 268357949
+  "unique_ngrams": 96263,
+  "total_ngrams": 242170946
 }

models/subword_ngram/af_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b7e9ec21d12eea1d385fb08e137b0390323ad38d5af55693ca5ec808d63eeb8e
-size 8462003
+oid sha256:b35e67ae40780c953eae3c44b6344369d951fcbf2c26114557257a76b11f77d5
+size 6246285

models/subword_ngram/af_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "af",
-  "unique_ngrams": 732288,
-  "total_ngrams": 268230620
+  "unique_ngrams": 531540,
+  "total_ngrams": 242044482
 }

models/tokenizer/af_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f743e4753c9efba2d99338cf7ad44c6a473a342a46e4df9671acf4191ba7afd
-size 508391
+oid sha256:2be64d6763f5e6fd310791270e3525ef8484a9db66ef0fff6e0f1476d6a96d34
+size 507153

models/tokenizer/af_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf99915b4dc311a913ef83532e1b5909f83560e280ce3c1ec65b0482070c3bdd
-size 789416
+oid sha256:8b5159876118e0a3489a99480419b844198a12a1b816ad232fed8ccae35bb818
+size 786684

models/tokenizer/af_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ae7229d1b022d86ec54a1b3e2c2739e76aa8b7c1327e3cd4f47c0530c99afff
-size 1366342
+oid sha256:4675b04acf7e973018b38e96d006eb263620e28799e83be7bdd4a27ac0994a13
+size 1361348

models/tokenizer/af_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ebbd1727f82be8bbf58a61f934983bfa3f23d2f65c59c9c6d62ab68017db7ba
-size 372400
+oid sha256:1f7dc8da1d29ab51f9c6ac5f659460f17c5222dd813dbe07b43facfc31f68728
+size 371768

models/vocabulary/af_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:abad63eeefcdc78640a7f186a0aff7fc6bad429ed352ee2a07643f9ccbe0379e
-size 6727061
+oid sha256:e2355921b7485f0850c13db4987f38a9cb0cb18c4d52369c481dbd6c58288967
+size 6384076

models/vocabulary/af_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "af",
-  "vocabulary_size": 426702,
+  "vocabulary_size": 403515,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.023913563254074188,
+    "type_token_ratio": 0.02274299800346992,
     "coverage": {
-      "top_100": 0.4125284635056328,
-      "top_1000": 0.6301073515896018,
-      "top_5000": 0.7848384789184834,
-      "top_10000": 0.8399447900573702
+      "top_100": 0.4314823858718236,
+      "top_1000": 0.6347620541014498,
+      "top_5000": 0.7838174538213594,
+      "top_10000": 0.8392913943711919
     },
-    "hapax_count": 585162,
-    "hapax_ratio": 0.5783010365029293,
-    "total_documents": 127329
+    "hapax_count": 481438,
+    "hapax_ratio": 0.5440266319228253,
+    "total_documents": 126464
   }
 }

models/word_markov/af_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9712c1db6a1f679bed53a9525fc4d37129d4817df0e1662701aba5781263ec1d
-size 74330320
+oid sha256:137d7884907248e6dab7f37ec9010eca83d41914f5a04456d97db0173992ed52
+size 77512633

models/word_markov/af_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "af",
-  "unique_contexts": 1012367,
-  "total_transitions": 52392040
+  "unique_contexts": 884548,
+  "total_transitions": 38784545
 }

models/word_markov/af_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:465d77d2f538e56b8c138515df12efeb4c4dfda8fbb0f2bd4cf5cf04ed0d532a
-size 231993381
+oid sha256:509f64b241796227fd01fdc0d6d9738c3943aabf43e5c799d193f8707fa2a669
+size 234719710

models/word_markov/af_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "af",
-  "unique_contexts": 8362191,
-  "total_transitions": 52264712
+  "unique_contexts": 8810967,
+  "total_transitions": 38658081
 }

models/word_markov/af_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1d8d8d6ec490aff768a6e7268a3a3a2ecb9e17e4e3a136bb8ba504fef57669f3
-size 430211054
+oid sha256:b7b6e1d49f05b7f0a242d75b1deca41677df4392fd3120553ba2e335ce06fee2
+size 401895512

models/word_markov/af_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "af",
-  "unique_contexts": 21746451,
-  "total_transitions": 52137387
+  "unique_contexts": 20525798,
+  "total_transitions": 38531617
 }

models/word_markov/af_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:375d27128bfbbc7da4eb573e0dc99e9765bb9855979e6b5b65f4711973a8564f
-size 585301615
+oid sha256:2b17acd652effc19ef7d02df727c29bc1d40a01cc11192d367fc60ef3d69a16d
+size 515063986

models/word_markov/af_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "af",
-  "unique_contexts": 33034213,
-  "total_transitions": 52010063
+  "unique_contexts": 28628609,
+  "total_transitions": 38405153
 }

models/word_ngram/af_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:13c0ceecb17348926c4a01efeeca2e5b09313fb6b9ebcb6509bf450f38b1768b
-size 12049266
+oid sha256:0ce0cab75f83c53a097010b2765735880a382f5bffd0e9d05fac9f3cbba823c4
+size 10363466

models/word_ngram/af_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "af",
-  "unique_ngrams": 884766,
-  "total_ngrams": 52392040
+  "unique_ngrams": 738183,
+  "total_ngrams": 38784545
 }

models/word_ngram/af_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1bea1ba20ff8534a9328f94766681494a3d52c03a132b5961ecec14dcdfa5f4e
-size 31811393
+oid sha256:ed24c80ec95bb4f7b7f3e9b41efc4116210257d7e53719a017c664294f0cb056
+size 23146048

models/word_ngram/af_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "af",
-  "unique_ngrams": 2174878,
-  "total_ngrams": 52264712
+  "unique_ngrams": 1499483,
+  "total_ngrams": 38658081
 }

models/word_ngram/af_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e84236aa01f40edbdaf74357b179a706c758462c276c3af1e0755f191f11a1f
-size 62258431
+oid sha256:00b7902ab50f3ce3549864847da14b49603e12c990921c24438e1f9e61596ab4
+size 40930272

models/word_ngram/af_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "af",
-  "unique_ngrams": 4032808,
-  "total_ngrams": 52137387
+  "unique_ngrams": 2510434,
+  "total_ngrams": 38531617
 }