Add 8-bit weight-only ONNX for mobile runtimes and the decision_ai.json manifest

README.md

@@ -141,11 +141,15 @@ Objects are serialized as compact JSON with sorted keys. For `noul`, the options
 | `config.json` | ModernBERT config, plus the `dinah` block (`[OPT]` token id, question types, max length) |
 | `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` | moBERTo tokenizer (original ModernBERT vocabulary) plus the `[OPT]` token |
 | `onnx/model_int8.onnx` | int8 dynamic quantization with block-local attention, for CPU; needs `onnxruntime` (uses the `com.microsoft` MultiHeadAttention op) |
+| `onnx/model_8bit.onnx` | 8-bit weight-only quantization (MatMulNBits) with int8 embeddings, 165 MB: stays close to fp32 on older ONNX Runtime builds, such as the ones mobile apps ship (see below) |
+| `decision_ai.json` | manifest for on-device runtimes: files, SHA-256, special tokens, input and output names |
 | `dinah.py` | inference code |
 | `decision_index_engine.py` | engine for the [Decision Index kit](https://github.com/apolinario/decision-index) (`--engine decision_index_engine:DinahEngine`) |
 
 The int8 model agrees with fp32 on the top option in 94.3% of 300 typed-decisions test items, with the same accuracy (73.7% vs 73.0% on that sample). Use the PyTorch weights when you need the exact fp32 probabilities.
 
+The int8 model's accuracy depends on the ONNX Runtime version: on ORT 1.23 (the version mobile plugins ship) it agrees with fp32 on only 75.7% of the same kind of sample (69.0% accuracy). `onnx/model_8bit.onnx` agrees on 97.0% there (74.7% accuracy), at 149 ms per request on an Apple M5 CPU against 46 ms for int8. Prefer it on mobile.
+
 ## Model details
 
 - **Architecture:** ModernBERT-base (22 layers, hidden size 768, 8,192-token context), ~150M parameters, with an `[OPT]` token added to the vocabulary, a question-type embedding, an option scorer (MLP over each `[OPT]` hidden state) and a small confidence head that reads the shape of the option distribution.

decision_ai.json

@@ -0,0 +1,15 @@
+{
+  "format": "decision-ai/option-reader@1",
+  "name": "Dinah-0",
+  "license": "cc-by-nc-4.0",
+  "homepage": "https://huggingface.co/Lukitaduarte/dinah-0",
+  "files": {"model": "onnx/model_8bit.onnx", "tokenizer": "tokenizer.json"},
+  "sha256": {"onnx/model_8bit.onnx": "da9b977882dc6781e16958b9e4cd3f8ff37582fd3fd7ff41d935f49ca6f793e0", "tokenizer.json": "edeac9e7b2f170022cefd1c7b396ebca3ac9c57f2f93604844cb7fb16b1a71ca"},
+  "max_length": 8192,
+  "pad_multiple": 64,
+  "tokens": {"cls": "[CLS]", "sep": "[SEP]", "pad": "[PAD]", "option": "[OPT]"},
+  "question_types": {"noul": 0, "choice": 1, "score": 2},
+  "noul_false_default": "The statement above is false.",
+  "inputs": {"ids": "ids", "pad_mask": "pad_mask", "option_positions": "opt_positions", "option_mask": "opt_mask", "question_type": "qtype"},
+  "outputs": {"logits": "logits", "confidence_logit": "conf_logit"}
+}

onnx/model_8bit.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da9b977882dc6781e16958b9e4cd3f8ff37582fd3fd7ff41d935f49ca6f793e0
+size 164506332