Upload all models and assets for anp (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: anp
-language_name: ANP
+language_name: Angika
 language_family: indoaryan_central
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-indoaryan_central
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.779
+    value: 3.777
   - name: best_isotropy
     type: isotropy
-    value: 0.8284
+    value: 0.8282
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# ANP - Wikilangs Models
+# Angika - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **ANP** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Angika** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.293x | 3.29 | 0.1162% | 454,392 |
-| **16k** | 3.578x | 3.58 | 0.1263% | 418,207 |
-| **32k** | 3.779x 🏆 | 3.78 | 0.1334% | 395,905 |
+| **8k** | 3.298x | 3.30 | 0.1077% | 449,296 |
+| **16k** | 3.575x | 3.58 | 0.1168% | 414,503 |
+| **32k** | 3.777x 🏆 | 3.78 | 0.1234% | 392,298 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `मई ग्रेगोरी कैलंडर क 5मां महीना छेकै। इ उ सात महीना मँ सँ एक छेकै जेकरौ दिन सिनी...`
+**Sample 1:** `ई लेख खाली रंगौ के सूची लेख केरौ सूची क अँग्रेजी़ वर्णक्रम मँ रखै लेली बनलौ छै। ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
-| 16k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
-| 32k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
+| 8k | `▁ई ▁लेख ▁खाली ▁रंग ौ ▁के ▁सूची ▁लेख ▁केरौ ▁सूची ... (+15 more)` | 25 |
+| 16k | `▁ई ▁लेख ▁खाली ▁रंग ौ ▁के ▁सूची ▁लेख ▁केरौ ▁सूची ... (+13 more)` | 23 |
+| 32k | `▁ई ▁लेख ▁खाली ▁रंगौ ▁के ▁सूची ▁लेख ▁केरौ ▁सूची ▁क ... (+9 more)` | 19 |
 
-**Sample 2:** `राजा महेश ठाकुर – ई. तक मधुबनी जिला के भउर (भौर) गांव म॑ छेलै, जे मधुबनी स॑ करीब...`
+**Sample 2:** `तत्व उ छीकै जेकरा भौतिक व रासियनिक विधि द्वारा तोड़लो नय जाबे सकै छै। तत्त्व (जै...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+28 more)` | 38 |
-| 16k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+27 more)` | 37 |
-| 32k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+24 more)` | 34 |
+| 8k | `▁तत्व ▁उ ▁छीकै ▁जेकरा ▁भौतिक ▁व ▁रा स िय निक ... (+30 more)` | 40 |
+| 16k | `▁तत्व ▁उ ▁छीकै ▁जेकरा ▁भौतिक ▁व ▁रास िय निक ▁विधि ... (+27 more)` | 37 |
+| 32k | `▁तत्व ▁उ ▁छीकै ▁जेकरा ▁भौतिक ▁व ▁रासियनिक ▁विधि ▁द्वारा ▁तोड़लो ... (+22 more)` | 32 |
 
-**Sample 3:** `पति पत्नी नंदा केरऽ ई. मं॑ बनलऽ हिंदी फ़िल्म छेकै.`
+**Sample 3:** `मई ग्रेगोरी कैलंडर क 5मां महीना छेकै। इ उ सात महीना मँ सँ एक छेकै जेकरौ दिन सिनी...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
-| 16k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
-| 32k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
+| 8k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
+| 16k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
+| 32k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 3.779x compression
-- **Lowest UNK Rate:** 8k with 0.1162% unknown tokens
+- **Best Compression:** 32k achieves 3.777x compression
+- **Lowest UNK Rate:** 8k with 0.1077% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 5,052 | 12.30 | 15,168 | 20.6% | 52.3% |
-| **2-gram** | Subword | 1,738 🏆 | 10.76 | 17,876 | 38.2% | 73.8% |
-| **3-gram** | Word | 4,130 | 12.01 | 14,962 | 20.9% | 59.9% |
-| **3-gram** | Subword | 12,170 | 13.57 | 72,480 | 14.9% | 40.7% |
-| **4-gram** | Word | 6,457 | 12.66 | 28,266 | 18.2% | 56.2% |
-| **4-gram** | Subword | 41,486 | 15.34 | 205,918 | 8.4% | 27.1% |
+| **2-gram** | Word | 5,133 | 12.33 | 15,401 | 20.6% | 52.0% |
+| **2-gram** | Subword | 1,763 🏆 | 10.78 | 18,130 | 37.8% | 73.7% |
+| **3-gram** | Word | 4,136 | 12.01 | 14,976 | 21.1% | 59.7% |
+| **3-gram** | Subword | 12,510 | 13.61 | 74,071 | 14.6% | 40.2% |
+| **4-gram** | Word | 6,638 | 12.70 | 28,729 | 18.3% | 55.5% |
+| **4-gram** | Subword | 43,295 | 15.40 | 212,245 | 8.3% | 26.6% |
+| **5-gram** | Word | 4,565 | 12.16 | 20,947 | 20.4% | 62.1% |
+| **5-gram** | Subword | 74,529 | 16.19 | 271,380 | 5.9% | 20.8% |
 
 ### Top 5 N-grams by Size
 
@@ -146,19 +158,19 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `के लिए` | 2,018 |
+| 1 | `के लिए` | 1,987 |
 | 2 | `के अनुसार` | 1,711 |
-| 3 | `छै जे` | 1,623 |
-| 4 | `छै जेकरा` | 1,477 |
+| 3 | `छै जे` | 1,664 |
+| 4 | `छै जेकरा` | 1,521 |
 | 5 | `के औसत` | 1,421 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `छै जेकरा म` | 1,239 |
+| 1 | `छै जेकरा म` | 1,240 |
 | 2 | `जनगणना के अनुसार` | 1,231 |
-| 3 | `के रूप में` | 808 |
+| 3 | `के रूप में` | 796 |
 | 4 | `परिवार रहै छै` | 789 |
 | 5 | `म स्थित ऐगो` | 690 |
 
@@ -172,42 +184,62 @@ Below are sample sentences tokenized with each vocabulary size:
 | 4 | `के जनगणना के अनुसार` | 498 |
 | 5 | `गाँव छै जेकरा म` | 479 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `गाँव छै जेकरा म कुल` | 476 |
+| 2 | `छै के जनगणना के अनुसार` | 438 |
+| 3 | `0 6 आयु वर्ग के` | 436 |
+| 4 | `6 आयु वर्ग के बच्चा` | 435 |
+| 5 | `आयु वर्ग के बच्चा के` | 432 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `र _` | 44,044 |
-| 2 | `_ के` | 42,780 |
-| 3 | `के _` | 39,580 |
-| 4 | `, _` | 27,198 |
-| 5 | `। _` | 27,084 |
+| 1 | `र _` | 44,141 |
+| 2 | `_ के` | 43,544 |
+| 3 | `के _` | 39,889 |
+| 4 | `, _` | 27,806 |
+| 5 | `। _` | 27,568 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ के _` | 37,016 |
-| 2 | `_ में _` | 14,280 |
-| 3 | `_ की _` | 9,494 |
-| 4 | `_ औ र` | 9,303 |
-| 5 | `औ र _` | 9,298 |
+| 1 | `_ के _` | 37,379 |
+| 2 | `_ में _` | 14,100 |
+| 3 | `_ की _` | 9,283 |
+| 4 | `_ औ र` | 9,137 |
+| 5 | `औ र _` | 9,133 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ औ र _` | 9,269 |
-| 2 | `_ है । _` | 6,536 |
-| 3 | `_ छै । _` | 5,833 |
-| 4 | `_ ए क _` | 4,768 |
-| 5 | `र _ के _` | 3,598 |
+| 1 | `_ औ र _` | 9,104 |
+| 2 | `_ है । _` | 6,415 |
+| 3 | `_ छै । _` | 6,096 |
+| 4 | `_ ए क _` | 4,687 |
+| 5 | `_ छै , _` | 3,618 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ छै , _ जे` | 2,233 |
+| 2 | `_ भा र त _` | 2,072 |
+| 3 | `ता _ है । _` | 2,029 |
+| 4 | `_ अ नु सा र` | 2,019 |
+| 5 | `_ के _ लि ए` | 1,986 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (subword) with 1,738
+- **Best Perplexity:** 2-gram (subword) with 1,763
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~27% of corpus
+- **Coverage:** Top-1000 patterns cover ~21% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.8691 | 1.827 | 5.82 | 57,434 | 13.1% |
-| **1** | Subword | 0.9723 | 1.962 | 11.43 | 4,617 | 2.8% |
-| **2** | Word | 0.2533 | 1.192 | 1.57 | 333,590 | 74.7% |
-| **2** | Subword | 0.5474 | 1.461 | 3.83 | 52,772 | 45.3% |
-| **3** | Word | 0.0719 | 1.051 | 1.12 | 522,356 | 92.8% |
-| **3** | Subword | 0.4946 | 1.409 | 2.66 | 202,187 | 50.5% |
-| **4** | Word | 0.0215 🏆 | 1.015 | 1.03 | 583,736 | 97.9% |
-| **4** | Subword | 0.2981 | 1.230 | 1.71 | 538,126 | 70.2% |
+| **1** | Word | 0.8698 | 1.827 | 5.82 | 59,321 | 13.0% |
+| **1** | Subword | 0.9730 | 1.963 | 11.48 | 4,665 | 2.7% |
+| **2** | Word | 0.2523 | 1.191 | 1.56 | 344,866 | 74.8% |
+| **2** | Subword | 0.5491 | 1.463 | 3.85 | 53,547 | 45.1% |
+| **3** | Word | 0.0707 | 1.050 | 1.12 | 537,872 | 92.9% |
+| **3** | Subword | 0.4976 | 1.412 | 2.68 | 206,241 | 50.2% |
+| **4** | Word | 0.0212 🏆 | 1.015 | 1.03 | 599,865 | 97.9% |
+| **4** | Subword | 0.3012 | 1.232 | 1.72 | 551,827 | 69.9% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `के बीच शासन और विज्ञापन और २०० फिल्में रिलीज़ हुआ है जिसके कारण ही मैन 2`
-2. `में पाकिस्तान श्रीलंका मे पुणे शहर आरू एकरऽ द्रव्यमान संरक्षण के स्थान पऽ एकाग्र करै लेली`
-3. `है कि विक्की ग्युरेरो के बच्चा के बच्चा के 61 80 मीटर 16 राज्यो मँ संकटग्रस्त`
+1. `के सूची लेख न्यूयॉर्क 5 7 8 839 परिवार रहै के छेलै आरू सॉफ्ट लैंडिंग का`
+2. `में बर्फ़ के कुछ स्थानों पर दृष्टिपात करें पुणे शहर में आविष्कृत इक्वेटोरियम और दक्षिण जॉर्जिया`
+3. `छै जे अध्यक्ष बनान के बाल लिंग अनुपात 750 पुरुष आरु महिला छै जे संस्कृत अभिलेख`
 
 **Context Size 2:**
 
-1. `के लिए पौधों को जिन्हें फूलने से पहले उन्होंने डस्टी रोहड्स का भी समर्थन प्राप्त हो सकती`
-2. `के अनुसार मुंजथ गांव के कुल आबादी के साथ दो दुर्भाग्यपूर्ण मामलों के लिए लिख लेते थे`
-3. `छै जे बिहार राज्य मँ स्थित छै इ जिला पौराणिक काल म॑ विश्व भर में ���० १४`
+1. `के लिए उपलब्ध हैं यह या तो एक दूसरे के बाद वू गुमला वर्तमान झारखंड मँ धर्म`
+2. `के अनुसार उचगांव गांव के कुल आबादी के 4 76 छै रघरिया गाँव के औसत लिंग अनुपात`
+3. `छै जे कुल जनसंख्या के 17 98 छै जे मुख्य भूमि के लिए विला के शिखर का`
 
 **Context Size 3:**
 
-1. `छै जेकरा म कुल 22 परिवार रहै छै तुम्बापहाड़ गांव के जनसंख्या 188 छै जेकरा म पुरुष आरू`
-2. `जनगणना के अनुसार गोबिंदपुर गाँव के जनसंख्या छै जेकरा पुरुष आरु महिला छै गौरीपुर गांव के औसत लिंग`
-3. `के रूप में अरबी गोंद के साथ मिलाया जा सकता था इसी कारणवश बादशाह मुहम्‍मद बिन तुगलक ने`
+1. `छै जेकरा म कुल 122 परिवार रहै छै जनगणना के अनुसार सरही गांव के आबादी 182 छेलै जेकरा`
+2. `जनगणना के अनुसार दिघी के बाल लिंग अनुपात 695 छै जे बिहार राज्य के औसत 918 स कम`
+3. `के रूप में लाल सेना का नेतृत्व किया और बर्मिंघम अलबामा में के कुछ अहिंसक विरोधों को आयोजित`
 
 **Context Size 4:**
 
-1. `छै जेकरा म कुल 545 परिवार रहै छै के जनगणना के अनुसार मुस्तफाबाद गाँव के जनसंख्या 291 छै जे`
-2. `के औसत लिंग अनुपात 782 छै जे बिहार राज्य के औसत 918 स कम छै जनगणना के अनुसार तेतरिया`
-3. `छै जनगणना के अनुसार सहनी खेड़ा के बाल लिंग अनुपात 836 छै जे बिहार राज्य के औसत 918 स`
+1. `छै जेकरा म कुल 64 परिवार रहै छै के जनगणना के अनुसार बरियारपुर के बाल लिंग अनुपात छै जे`
+2. `के औसत लिंग अनुपात 782 छै जे बिहार राज्य के औसत 918 स कम छै जनगणना के अनुसार टकटौली`
+3. `छै जनगणना के अनुसार अमलगरिया गाँव के जनसंख्या 91 छै जेकरा म स 1 939 पुरुष आरू 1 705`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_ला_का_अनुसाय_उसनता_शि`
-2. `रचरण-हा_जे_के_शिय-_में`
-3. `क_किंगर_मशिक्षा_बांकारलिंग_`
+1. `_बादनसांख्यिकी_प्रभारत_देसदी`
+2. `रख_"_के_इति_बाल_कम्पनी`
+3. `करशान_जन_से_जुड़कर_वि`
 
 **Context Size 2:**
 
-1. `र_सम्पादक_छै_।_नीतिक_न॑_`
-2. `_के_कुल_रूप_पुर_सक्षमता_`
-3. `के_लिमिटेड_श्रेणी:_भास्कराचार_`
+1. `र_अपने_थे।वास्को_आड़े_और_`
+2. `_के_लिए_रहै_कित_दृष्टि)_के`
+3. `के_प्रारम्भिक_है_के_सार_चमत्का`
 
 **Context Size 3:**
 
-1. `_के_रूप_से_एक_छै।_जनसंख्या`
-2. `_में_रूचि_रखै_वाला_नहीं_है।_`
-3. `_की_भी_हैं_दाग_डॉक्टर_के_रूप`
+1. `_के_उच्च_पदार्थों_सँ_जुड़ली_गेले`
+2. `_में_तारे_गये_शिवनेरी_किये_जा`
+3. `_की_क्रियाक_सफल_करी_देलोगेल`
 
 **Context Size 4:**
 
-1. `_और_श्रुति_साहित्य,_दर्शन_हेतु_`
-2. `_है।_इसके_बजाय_व्याख्यान_क_ए`
-3. `_छै।_चूना_ऐगो_अंतर_छै,_अपि`
+1. `_और_अंतरिक्ष_में_था।_इसके_म`
+2. `_है।_यह_फिल्म_अभिनेता_से_इन्हों`
+3. `_छै।_नाभिकीय_शक्ति_का_रूप_मँ_`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 97.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (538,126 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (551,827 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,64 +342,64 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 26,612 |
-| Total Tokens | 692,487 |
-| Mean Frequency | 26.02 |
+| Vocabulary Size | 27,495 |
+| Total Tokens | 705,736 |
+| Mean Frequency | 25.67 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 316.81 |
+| Frequency Std Dev | 313.78 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | के | 37,114 |
-| 2 | में | 15,064 |
-| 3 | छै | 12,685 |
-| 4 | है | 12,473 |
-| 5 | की | 9,887 |
-| 6 | और | 9,313 |
-| 7 | का | 7,757 |
-| 8 | से | 7,397 |
-| 9 | को | 5,594 |
-| 10 | हैं | 5,335 |
+| 1 | के | 37,476 |
+| 2 | में | 14,866 |
+| 3 | छै | 13,486 |
+| 4 | है | 12,172 |
+| 5 | की | 9,675 |
+| 6 | और | 9,147 |
+| 7 | का | 7,600 |
+| 8 | से | 7,248 |
+| 9 | को | 5,485 |
+| 10 | हैं | 5,201 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | pmegp | 2 |
-| 2 | odop | 2 |
-| 3 | naps | 2 |
-| 4 | संवर्द्धन | 2 |
-| 5 | आईज़ | 2 |
-| 6 | रिटेल | 2 |
-| 7 | एक्सीलेंस | 2 |
-| 8 | इंस्टाग्राम | 2 |
-| 9 | कास्टिंग | 2 |
-| 10 | ईयर | 2 |
+| 1 | zeros | 2 |
+| 2 | ignored | 2 |
+| 3 | dmy | 2 |
+| 4 | mdy | 2 |
+| 5 | paren | 2 |
+| 6 | breaking | 2 |
+| 7 | inserted | 2 |
+| 8 | values | 2 |
+| 9 | separator | 2 |
+| 10 | days | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1238 |
-| R² (Goodness of Fit) | 0.994960 |
+| Zipf Coefficient | 1.1206 |
+| R² (Goodness of Fit) | 0.994934 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 40.3% |
-| Top 1,000 | 69.8% |
-| Top 5,000 | 87.2% |
-| Top 10,000 | 93.2% |
+| Top 100 | 39.9% |
+| Top 1,000 | 69.2% |
+| Top 5,000 | 86.8% |
+| Top 10,000 | 92.8% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9950 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 40.3% of corpus
-- **Long Tail:** 16,612 words needed for remaining 6.8% coverage
+- **Zipf Compliance:** R²=0.9949 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 39.9% of corpus
+- **Long Tail:** 17,495 words needed for remaining 7.2% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.8284 🏆 | 0.3485 | N/A | N/A |
-| **mono_64d** | 64 | 0.6880 | 0.2899 | N/A | N/A |
-| **mono_128d** | 128 | 0.3275 | 0.2699 | N/A | N/A |
+| **mono_32d** | 32 | 0.8282 🏆 | 0.3565 | N/A | N/A |
+| **mono_64d** | 64 | 0.7038 | 0.2985 | N/A | N/A |
+| **mono_128d** | 128 | 0.3364 | 0.2651 | N/A | N/A |
+| **aligned_32d** | 32 | 0.8282 | 0.3484 | 0.0140 | 0.1160 |
+| **aligned_64d** | 64 | 0.7038 | 0.2963 | 0.0320 | 0.1400 |
+| **aligned_128d** | 128 | 0.3364 | 0.2724 | 0.0320 | 0.1640 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8284 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.3027. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.8282 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3062. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 3.2% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.980** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -422,13 +457,11 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-प्` | प्रिंत्सीप, प्रतिअंकन, प्रभाग |
-| `-प्र` | प्रिंत्सीप, प्रतिअंकन, प्रभाग |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-ों` | साँपों, अनुभववादियों, रेलों |
+| `-ों` | कबीलों, राजकुमारियों, खातों |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -436,17 +469,16 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `tion` | 2.57x | 12 contexts | motion, action, section |
-| `atio` | 2.57x | 12 contexts | station, nations, stations |
-| `stat` | 2.59x | 6 contexts | state, states, status |
+| `tion` | 2.62x | 15 contexts | motion, action, nations |
+| `atio` | 2.64x | 12 contexts | nations, station, equation |
+| `stat` | 2.66x | 6 contexts | state, states, statea |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
 This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
 
-| Prefix | Suffix | Frequency | Examples |
-|--------|--------|-----------|----------|
-| `-प्` | `-ों` | 20 words | प्रयासों, प्रकृतिवादियों |
+*No significant affix co-occurrences detected.*
+
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -454,26 +486,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| प्रवृत्ति | **`प्र-वृत्ति`** | 4.5 | `वृत्ति` |
-| अक्षांशों | **`अक्षांश-ों`** | 4.5 | `अक्षांश` |
-| व्यवसायों | **`व्यवसाय-ों`** | 4.5 | `व्यवसाय` |
-| चक्रवातों | **`चक्रवात-ों`** | 4.5 | `चक्रवात` |
-| भागीदारों | **`भागीदार-ों`** | 4.5 | `भागीदार` |
-| तीर्थंकरों | **`तीर्थंकर-ों`** | 4.5 | `तीर्थंकर` |
-| उद्दीपकों | **`उद्दीपक-ों`** | 4.5 | `उद्दीपक` |
-| काव्यतत्वों | **`काव्यतत्व-ों`** | 4.5 | `काव्यतत्व` |
-| संग्रहालयों | **`संग्रहालय-ों`** | 4.5 | `संग्रहालय` |
-| साहित्यकारों | **`साहित्यकार-ों`** | 4.5 | `साहित्यकार` |
-| चिकित्सकों | **`चिकित्सक-ों`** | 4.5 | `चिकित्सक` |
-| उद्देश्यों | **`उद्देश्य-ों`** | 4.5 | `उद्देश्य` |
-| विश्वकोशों | **`विश्वकोश-ों`** | 4.5 | `विश्वकोश` |
-| निष्कर्षों | **`निष्कर्ष-ों`** | 4.5 | `निष्कर्ष` |
-| प्रशंसकों | **`प्र-शंसक-ों`** | 3.0 | `शंसक` |
+| महाविद्यालयों | **`महाविद्यालय-ों`** | 4.5 | `महाविद्यालय` |
+| प्रबंधकों | **`प्रबंधक-ों`** | 4.5 | `प्रबंधक` |
+| चमत्कारों | **`चमत्कार-ों`** | 4.5 | `चमत्कार` |
+| विद्वानों | **`विद्वान-ों`** | 4.5 | `विद्वान` |
+| व्याख्यानों | **`व्याख्यान-ों`** | 4.5 | `व्याख्यान` |
+| कार्टूनों | **`कार्टून-ों`** | 4.5 | `कार्टून` |
+| शास्त्रों | **`श���स्त्र-ों`** | 4.5 | `शास्त्र` |
+| कंप्यूटरों | **`कंप्यूटर-ों`** | 4.5 | `कंप्यूटर` |
+| संस्कारों | **`संस्कार-ों`** | 4.5 | `संस्कार` |
+| महासागरों | **`महासागर-ों`** | 4.5 | `महासागर` |
+| पाठ्यक्रमों | **`पाठ्यक्रम-ों`** | 4.5 | `पाठ्यक्रम` |
+| मुसलमानों | **`मुसलमान-ों`** | 4.5 | `मुसलमान` |
+| महाद्वारों | **`महाद्वार-ों`** | 4.5 | `महाद्वार` |
+| चालुक्यों | **`चालुक्य-ों`** | 4.5 | `चालुक्य` |
+| प्रकाशकों | **`प्रकाशक-ों`** | 4.5 | `प्रकाशक` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language ANP appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Angika shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -485,7 +519,7 @@ The language ANP appears to be more isolating or has a highly fixed vocabulary.
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
 | Tokenizer | **32k BPE** | Best compression (3.78x) |
-| N-gram | **2-gram** | Lowest perplexity (1,738) |
+| N-gram | **2-gram** | Lowest perplexity (1,763) |
 | Markov | **Context-4** | Highest predictability (97.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -700,4 +734,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 05:15:16*
+*Report Date: 2026-01-03 14:14:54*

models/embeddings/aligned/anp_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7af5575e958475f78e2345f53a960547b317d08c4f564f1e287dc68e616de125
+size 1036402426

models/embeddings/aligned/anp_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "anp", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/anp_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:211dd9cb35d6746640872c76eb78c86fe575a4d701ef5cf510e8d23359171c14
+size 65664

models/embeddings/aligned/anp_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "anp",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1254,
+  "vocab_size": 11815
+}

models/embeddings/aligned/anp_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62587f7000693b582c9132fe07d435b228ee6af1d47b7935f4e2b18424ab4eea
+size 259328506

models/embeddings/aligned/anp_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "anp", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/anp_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:648c377b1a1a5d1b4666b6ec914da4647deef248b9e4cdfb29f1dc4a4348720b
+size 4224

models/embeddings/aligned/anp_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "anp",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1254,
+  "vocab_size": 11815
+}

models/embeddings/aligned/anp_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d070ab7b4e6d51df3ef5192423b2444c290dcd9623fdc52ba2c66ba353a51a22
+size 518353146

models/embeddings/aligned/anp_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "anp", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/anp_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1502bda2f265ab094144f926c666d9545ee36a2e459b061c3e55b39d59ca2b6
+size 16512

models/embeddings/aligned/anp_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "anp",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1254,
+  "vocab_size": 11815
+}

models/embeddings/monolingual/anp_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a9caacde775c5d9ed45854d4282f8461327a3fb82d634b04bde4c1a7010bfb01
-size 1036094760
+oid sha256:7af5575e958475f78e2345f53a960547b317d08c4f564f1e287dc68e616de125
+size 1036402426

models/embeddings/monolingual/anp_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 11521
+  "vocab_size": 11815
 }

models/embeddings/monolingual/anp_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:550b641eec37665cd8011ee7edf15fb97b5a67b4ea66eddae766e6ab6db5fa8b
-size 259246632
+oid sha256:62587f7000693b582c9132fe07d435b228ee6af1d47b7935f4e2b18424ab4eea
+size 259328506

models/embeddings/monolingual/anp_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 11521
+  "vocab_size": 11815
 }

models/embeddings/monolingual/anp_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5d2fb6809bbbfd3fdc21a99d5055c629f05387c2aad2b8c7bd73362067d6a1b8
-size 518196008
+oid sha256:d070ab7b4e6d51df3ef5192423b2444c290dcd9623fdc52ba2c66ba353a51a22
+size 518353146

models/embeddings/monolingual/anp_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 11521
+  "vocab_size": 11815
 }

models/subword_markov/anp_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95c04e6c8a5663b3755575bf1c4f4bd7a74ee1859643e6a15404e0aefd1820ec
-size 364915
+oid sha256:0f8c7d4097372261966c6bc97fb52eb94ecf05aba1b20a249f9d296e00c88066
+size 365191

models/subword_markov/anp_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 4617,
-  "total_transitions": 2607475
+  "unique_contexts": 4665,
+  "total_transitions": 2668925
 }

models/subword_markov/anp_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:daa9cbe0600238ddf33258710db221432206594aadb8cd97e7c233736df61bc1
-size 1615892
+oid sha256:181129c2fea7a83e1ee86d4443f411e555f84964df22997884ef74fa0443b196
+size 1648766

models/subword_markov/anp_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 52772,
-  "total_transitions": 2604446
+  "unique_contexts": 53547,
+  "total_transitions": 2665844
 }

models/subword_markov/anp_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c285b9eb6f3174cff5126cc788535f54c2aac763cfabb71374694688d74bb8d7
-size 4795293
+oid sha256:ca8503e1796de5b1b63f4db40515f49a5fd0388d3ee34a09bdfe79795c374ca0
+size 4906034

models/subword_markov/anp_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 202187,
-  "total_transitions": 2601417
+  "unique_contexts": 206241,
+  "total_transitions": 2662763
 }

models/subword_markov/anp_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf8bec6d17e4fe549aa7177f537ef96f42090d3142c162b4f6ddcecc9a3ca3f9
-size 10831894
+oid sha256:e36d66386ed99fb8158cd97838a5534c8d7168ccbbec8a30652e2563e832def1
+size 11056053

models/subword_markov/anp_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 538126,
-  "total_transitions": 2598388
+  "unique_contexts": 551827,
+  "total_transitions": 2659682
 }

models/subword_ngram/anp_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:775113617eee04863418a0e3d9184050647852be0f10c589b8fe1da6543c4c55
-size 264208
+oid sha256:f2d4934c85df0e19da88307fe4c1b2c6bd7f7d7584e4e152fff2c1d790e4b622
+size 267872

models/subword_ngram/anp_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 17876,
-  "total_ngrams": 2607475
+  "unique_ngrams": 18130,
+  "total_ngrams": 2668925
 }

models/subword_ngram/anp_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d1beda611527797d1c53b1c09dc17f9ce7383569eb890ec2fb07d1f5209c0c22
-size 1044126
+oid sha256:1ee608f9e770b8aaa0c3f06ad058eb1e57096805145b07b1f13bdd793e481aa4
+size 1068374

models/subword_ngram/anp_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 72480,
-  "total_ngrams": 2604446
+  "unique_ngrams": 74071,
+  "total_ngrams": 2665844
 }

models/subword_ngram/anp_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1fe6cf4b2975ea16eac745cd73317aa126e7bd7842670a9adabb558d579c736b
-size 3083912
+oid sha256:3bae4cc49c8bd13ed2637bd3e31e241df8714d11482506bf0c1ab0ef70d2bf29
+size 3164693

models/subword_ngram/anp_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 205918,
-  "total_ngrams": 2601417
+  "unique_ngrams": 212245,
+  "total_ngrams": 2662763
 }

models/subword_ngram/anp_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0c567ba5d3c542b3799178199f32bd1bd08014f13b6b375808a83d18a7d741b
+size 4356539

models/subword_ngram/anp_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "anp",
+  "unique_ngrams": 271380,
+  "total_ngrams": 2659682
+}

models/tokenizer/anp_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dbad16cbb100bec382d11f61c37dc66eb44c51310eda04fcecebe9af7d4c358
-size 626731
+oid sha256:b4c3741b80f17d1eae30558373a32b1e6916523b0ee40e8d07c0d64d3e8f1080
+size 618098

models/tokenizer/anp_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:307cc8ede7a5261bd978c0775a7d475811f92b91caf248ac449e45d8c1a3f03d
-size 1055594
+oid sha256:811d7d1db33e325f5e70f07d4c826bf83fbc30ebff039b047a22c1b821fb235f
+size 1035857

models/tokenizer/anp_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aa6e8b6fe63bd83b4d41987afef044fc696594969b6cbbbe9dc22aee174de945
-size 430161
+oid sha256:99896dd2de3109457e12c0c017a5ab599dab704efee42adbf365433c86e2b6e2
+size 425391

models/vocabulary/anp_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:23c24e34a334b98f6f430ae4bfb9547516edad531380c24b4d8922e550ce220d
-size 502668
+oid sha256:8435ca8e03a3c4b629b3f26bb34fa4898fd59714c125734fffc27034f2525995
+size 512958

models/vocabulary/anp_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "anp",
-  "vocabulary_size": 26612,
+  "vocabulary_size": 27495,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.07945542193325766,
+    "type_token_ratio": 0.0804783609588911,
     "coverage": {
-      "top_100": 0.3854363523002036,
-      "top_1000": 0.6679237822959595,
-      "top_5000": 0.8348141768358013,
-      "top_10000": 0.8920299882905762
+      "top_100": 0.38180617731513744,
+      "top_1000": 0.6624765117231244,
+      "top_5000": 0.8305210669168468,
+      "top_10000": 0.8883205305842446
     },
-    "hapax_count": 30862,
-    "hapax_ratio": 0.5369732400737725,
-    "total_documents": 3029
+    "hapax_count": 31866,
+    "hapax_ratio": 0.5368171021377672,
+    "total_documents": 3081
   }
 }

models/word_markov/anp_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d37e0230173ff329a06a1b9fde21c583b6b6fb6e94481f23a7a68e72460175c5
-size 2920307
+oid sha256:74c751e08e10f23f5bf6fb2e82f4cd372794298d19dc29e73d0ad818a2cdb098
+size 3049321

models/word_markov/anp_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 57434,
-  "total_transitions": 720320
+  "unique_contexts": 59321,
+  "total_transitions": 734521
 }

models/word_markov/anp_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fdf85ce95679a912c641b5ffb10719ad0f16e152363efc3c3922f250107b17b8
-size 8558224
+oid sha256:c2f465a758b1a2d405c79c4ab4f4895682189bc3945752cf5cdfa7a1df55522f
+size 8916451

models/word_markov/anp_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 333590,
-  "total_transitions": 717291
+  "unique_contexts": 344866,
+  "total_transitions": 731440
 }

models/word_markov/anp_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d46521c8f50238b1c6c19de18c1d6e31bb2ae9bb5e64373564f718f502ccb21a
-size 12610274
+oid sha256:37b7a565352c35ad4e7733cf1439619b49cd1bcf9a3246a903b5df21368a82f1
+size 13044546

models/word_markov/anp_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 522356,
-  "total_transitions": 714262
+  "unique_contexts": 537872,
+  "total_transitions": 728359
 }