kirp commited on
Commit
b447cc7
·
verified ·
1 Parent(s): dd416ab

Model card: JPT-9B JevBench public 0.853 measured with the JevBench CLI

Browse files
Files changed (2) hide show
  1. README.md +7 -4
  2. assets/jevbench.png +0 -0
README.md CHANGED
@@ -9,7 +9,7 @@ tags: [decision-model, calibration, typed-decisions, classification, multimodal]
9
  # JPT-9B
10
 
11
  [![License](https://img.shields.io/badge/license-CC_BY--NC_4.0-lightgrey)](#❯❯-license)
12
- [![JevBench v1.4 public](https://img.shields.io/badge/JevBench_v1.4_public-0.857-E8663D)](#❯-jevbench-v140)
13
  [![Decision Index 0.2.1](https://img.shields.io/badge/Decision_Index_0.2.1-46.89-E8663D)](#❯-decision-index-021)
14
  [![llm2jev](https://img.shields.io/badge/serve_with-llm2jev-blue)](https://github.com/tic-top/llm2jev)
15
 
@@ -40,8 +40,8 @@ Built on [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B): a LoRA fine-
40
 
41
  ### ❯ JevBench v1.4.0
42
 
43
- **[JevBench](https://github.com/fstandhartinger/jevbench) [2] scores general typed decisions; JPT-9B reaches 0.857
44
- public accuracy, level with Winnow-12B and just under Jev 1.13.0 — but below the smaller JPT-4B (0.879).** v1.4.0 has
45
  231 public items and 308 sealed items only the maintainer can run, so JPT-9B has no official v1.4 score yet.
46
 
47
  ![JevBench v1.4.0 public accuracy](assets/jevbench.png)
@@ -54,8 +54,8 @@ public accuracy, level with Winnow-12B and just under Jev 1.13.0 — but below t
54
  | [JPT-35B-A3B](https://huggingface.co/kirp/jpt-35b-a3b) (ours) | 35B-A3B | 0.892 | pending | pending |
55
  | [JPT-4B](https://huggingface.co/kirp/jpt-4b) (ours) | 4B | 0.879 | pending | pending |
56
  | Jev 1.13.0 (TypeSafe AI, API) | closed | 0.866 | 0.367 | 63.3 |
57
- | **JPT-9B** | 9B | **0.857** | pending | pending |
58
  | Winnow-12B Q8 | 12B | 0.857 | 0.331 | 55.6 |
 
59
  | JevK5 v0.2.0 | 27B | 0.853 | 0.331 | 62.0 |
60
  | decider-35b-a3b | 35B-A3B | 0.831 | 0.315 | 41.2 |
61
  | Hopper | — | 0.823 | 0.341 | 59.4 |
@@ -66,6 +66,9 @@ public accuracy, level with Winnow-12B and just under Jev 1.13.0 — but below t
66
  | kev 4B (research preview) | 4B | 0.662 | 0.224 | 36.1 |
67
 
68
  Source: other rows from `results/v1.4/jevbench-v1.4-results.json` at jevbench commit `2fa63fa` (2026-09-23).
 
 
 
69
  ¹ Not in the v1.4 results; its own card's number on the 231 public items. A same-prompt Qwen3.5-9B zero-shot run on
70
  the public items has not been done yet.
71
 
 
9
  # JPT-9B
10
 
11
  [![License](https://img.shields.io/badge/license-CC_BY--NC_4.0-lightgrey)](#❯❯-license)
12
+ [![JevBench v1.4 public](https://img.shields.io/badge/JevBench_v1.4_public-0.853-E8663D)](#❯-jevbench-v140)
13
  [![Decision Index 0.2.1](https://img.shields.io/badge/Decision_Index_0.2.1-46.89-E8663D)](#❯-decision-index-021)
14
  [![llm2jev](https://img.shields.io/badge/serve_with-llm2jev-blue)](https://github.com/tic-top/llm2jev)
15
 
 
40
 
41
  ### ❯ JevBench v1.4.0
42
 
43
+ **[JevBench](https://github.com/fstandhartinger/jevbench) [2] scores general typed decisions; JPT-9B reaches 0.853
44
+ public accuracy, level with JevK5 and just under Winnow-12B and Jev 1.13.0 — but below the smaller JPT-4B (0.879).** v1.4.0 has
45
  231 public items and 308 sealed items only the maintainer can run, so JPT-9B has no official v1.4 score yet.
46
 
47
  ![JevBench v1.4.0 public accuracy](assets/jevbench.png)
 
54
  | [JPT-35B-A3B](https://huggingface.co/kirp/jpt-35b-a3b) (ours) | 35B-A3B | 0.892 | pending | pending |
55
  | [JPT-4B](https://huggingface.co/kirp/jpt-4b) (ours) | 4B | 0.879 | pending | pending |
56
  | Jev 1.13.0 (TypeSafe AI, API) | closed | 0.866 | 0.367 | 63.3 |
 
57
  | Winnow-12B Q8 | 12B | 0.857 | 0.331 | 55.6 |
58
+ | **JPT-9B** | 9B | **0.853** | pending | pending |
59
  | JevK5 v0.2.0 | 27B | 0.853 | 0.331 | 62.0 |
60
  | decider-35b-a3b | 35B-A3B | 0.831 | 0.315 | 41.2 |
61
  | Hopper | — | 0.823 | 0.341 | 59.4 |
 
66
  | kev 4B (research preview) | 4B | 0.662 | 0.224 | 36.1 |
67
 
68
  Source: other rows from `results/v1.4/jevbench-v1.4-results.json` at jevbench commit `2fa63fa` (2026-09-23).
69
+ Ours measured with JevBench's CLI at `2fa63fa` (SGLang 0.5.18, `kirp/jpt-9b@7114b0c`): 197/231, easy 48 · standard 68 ·
70
+ hard 81. Our own harness gets 198: on `original-intent-02-0` two options tie exactly (0.4864 each) and the two tools
71
+ break the tie differently.
72
  ¹ Not in the v1.4 results; its own card's number on the 231 public items. A same-prompt Qwen3.5-9B zero-shot run on
73
  the public items has not been done yet.
74
 
assets/jevbench.png CHANGED