Instructions to use eaddario/Nanbeige4.3-33B-JQK with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use eaddario/Nanbeige4.3-33B-JQK with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16
# Run inference directly in the terminal:
llama cli -hf eaddario/Nanbeige4.3-33B-JQK:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16
# Run inference directly in the terminal:
llama cli -hf eaddario/Nanbeige4.3-33B-JQK:F16

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16
# Run inference directly in the terminal:
./build/bin/llama cli -hf eaddario/Nanbeige4.3-33B-JQK:F16

Use Docker

docker model run hf.co/eaddario/Nanbeige4.3-33B-JQK:F16

Llama new
LM Studio
Jan

vLLM

How to use eaddario/Nanbeige4.3-33B-JQK with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "eaddario/Nanbeige4.3-33B-JQK"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eaddario/Nanbeige4.3-33B-JQK",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/eaddario/Nanbeige4.3-33B-JQK:F16

Ollama
How to use eaddario/Nanbeige4.3-33B-JQK with Ollama:
```
ollama run hf.co/eaddario/Nanbeige4.3-33B-JQK:F16
```
Unsloth Desktop

How to use eaddario/Nanbeige4.3-33B-JQK with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "eaddario/Nanbeige4.3-33B-JQK:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use eaddario/Nanbeige4.3-33B-JQK with Docker Model Runner:
```
docker model run hf.co/eaddario/Nanbeige4.3-33B-JQK:F16
```

Lemonade

How to use eaddario/Nanbeige4.3-33B-JQK with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull eaddario/Nanbeige4.3-33B-JQK:F16

Run and chat with the model

lemonade run user.Nanbeige4.3-33B-JQK-F16

List all available models

lemonade list

Hermes Agent

How to use eaddario/Nanbeige4.3-33B-JQK with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default eaddario/Nanbeige4.3-33B-JQK:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use eaddario/Nanbeige4.3-33B-JQK with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/Nanbeige4.3-33B-JQK:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "eaddario/Nanbeige4.3-33B-JQK:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Experimental `JQ*K` global target bits‑per‑weight quantization of Nanbeige/Nanbeige4.3-33B

Using non-standard (forked) LLaMA C++ for quantization. The JQ1_K, JQ2_K, JQ3_K, JQ4_K, JQ5_K, JQ6_K & JQ8_0 models will NOT work with standard LLaMA C++.

Using JQ*_K has shown to improve KLD between 10% to 50% compared to standard Q*_K types. More information is available in eaddario/JQ2_K-JQ6_K-Test-Models and PR #30715.

Original model: Nanbeige/Nanbeige4.3-33B

From the original model creators:

1. Introduction

In Nanbeige4.3, we take a foundational step toward pushing the boundaries of lightweight agents. Our strategy centers on building a high-capacity teacher agent first, which will subsequently anchor the distillation of advanced agentic capabilities into smaller models.

To this end, we first introduce Nanbeige4.3-33B, designed as the flagship teacher model for the Nanbeige4.3 series. Delivering state-of-the-art agentic performance among open models in the ~30B parameter class, Nanbeige4.3-33B excels at complex coding and professional cowork tasks. It not only offers a powerful, cost-effective option for immediate real-world deployment, but also lays the necessary foundation to power our upcoming small model distillation pipeline (e.g., for Nanbeige4.3-3B).

⚠️ PLEASE READ THIS BEFORE USING THESE EXPERIMENTAL VERSIONS! ⚠️

An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. There are many approaches to accomplish this, including architecture simplification and knowledge distillation, but my focus has been primarily on quantization and pruning.

The method to produce these experimental versions involves using a custom version of llama-quantize to automatically select the lowest error quantization recipe that achieves a global target bits‑per‑weight (bpw) and uses a new quant family JQ*_K.

There are two a pull requests (#15550 and #30715) to merge these changes back into the core llama.cpp project. This may or may not ever happen so, until then, the modified versions will be available on GitHub.

For testing and comparison, I use models produced by Bartowski (see credits below) and Unsloth (Daniel and Michael Han do some really interesting stuff!) but when they don't provide versions of the required model, tests and comparisons are against standard quantization obtained by simply running llama-quantize with no further optimizations.

All experimental versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration. In llama.cpp, an imatrix is a calibration file derived from running representative text through the model and collecting activation statistics. It is used to weight quantization error so that error in more “important” directions (as estimated from activations) is penalized more heavily.

The process to generate these models is roughly as follows:

Convert the original model's safetensors to GGUF F16
Estimate the Perplexity score for the F16 model (baseline) using the wikitext-2-raw-v1 dataset, and save the logits
Generate an imatrix from the most appropriate calibration dataset
Quantize the baseline model targeting a bpw average (e.g. llama-quantize --target-bpw 4.5678 --state-file --imatrix imatrix.gguf baseline-model-F16.gguf)
Calculate Perplexity, KL Divergence, ARC (Easy+Challenge), GPQA-Diamond, HellaSwag, MMLU-Redux, Truthful QA and WinoGrande scores for each quantized model

Misconceptions about BF16 to F16 Conversion

A common concern when converting BFloat16 (BF16) models to Float16 (F16) is the potential for accuracy loss. Specifically:

Weight Clipping (Overflow): Clipping, or overflow, is often feared but only occurs if a model's weights exceed the range of ±65,503. This is a relatively rare issue in practice.
Subnormal Zeroing (Underflow): A more frequent occurrence is underflow, where weights smaller than approximately 5.96x10⁻⁸ are converted to zero.

Crucially, when the F16 model is subsequently used for quantization, the resulting degradation in metrics like Perplexity (PPL) or Kullback–Leibler Divergence (KLD) is minimal. Any variations are typically restricted to the hundreds or thousandths decimal places compared to the BF16 model.

However, considering that weight clipping presents a more substantial risk to model integrity, every BF16 base model undergoes validation prior to the conversion process. Consequently, no models hosted in this repository exhibit performance degradation due to overflow clipping.

While BF16 offers precision benefits, performance remains a key factor.

Conversion Speed: Tests, such as timing convert_hf_to_gguf.py, show a notable performance difference, with conversion to BF16 being 15–30% slower than to F16.
Inference Speed: A less pronounced but still present difference (3–6%) is observed during inference. Although native BF support has been introduced by many chip manufacturers, the slower performance may stem from the entire software and hardware stack (firmware, libraries, etc.) not being fully optimized yet.

The choice to prioritize F16 over BF16 is driven by a focus on maximizing performance in specific deployment environments. My primary objective is not large-scale quantization production, a domain where others like Bartowski and Unsloth excel at, but rather optimizing inference performance for resource-constrained environments. Since BF16 support is not yet widespread in areas like mobile, edge, and embedded devices, using F16 ensures broader compatibility and easier optimization for these use cases.

Advantages and disadvantages of the global target bits‑per‑weight quantization process

Advantages

Target arbitrary size models
- When specifying --target-bpw 4.5678 for instance, the algorithm will produce a model (nearly) exactly of that size, which is very useful for maximizing VRAM usage. In a system with 24GB VRAM and a 70B model, standard quants might produce a 16.8GB file (too small, quality left on table) or a 24.1GB file (won't fit). This approach can generate a 23.85GB file to utilize the hardware fully.
Data-driven mixed precision often can improve quality at fixed size
- Instead of using hardcoded heuristics (e.g. make attn_v Q5_K for a 70B model), that may be sub‑optimal for a given architecture or size, the quantization mix is determined by the actual error sensitivity of the specific model's weights. This, in practice, often yields a better quality/size trade-off, especially in aggressive quantization scenarios (1.5 to 3.5 bpw), or for unusual architectures.
- Please note: llama.cpp’s heuristics have been tuned across many models and are highly optimized; although the target bpw method produces better quality often (>75% based on tests with 130 models from 11 different families), it can also lose in surprising cases.
Allows better like-for-like comparisons between models and families
- Standard llama.cpp quantization uses hardcoded rules like: "use Q4_K_M, except bump some tensors up/down, except fall back if incompatible, except keep some tensors unquantized..." and for that reason, two different models quantized with the same Q4_K_M type can end up with very different bpw (e.g. 4.75 and 4.30).
- All things being equal, the performance of a model is usually proportional to its overall bpw size; models with a higher bpw tend to perform better than lower bpw models. Since model A has simply been given more bits, it will typically perform better (lower perplexity, better eval scores, etc.) even if the underlying quantization method is identical. That makes comparing the performance not a controlled experiment, because the comparison is between models with different effective compression ratios.
- --target-bpw tries to address that by making the experiment more controlled: each model gets quantized to land on (approximately) the same global byte budget, so that the models' performance differences are more attributable to architecture/training differences, quantization error behaviour at the same compression ratio, optimizer’s allocation decisions, etc.

Disadvantages

Quantization process is significantly slower than standard
- This approach can take 5x-10x longer as it quantizes a sample of most tensors into 15 different formats, dequantizes them back to floats, computes error diffs, and selects the best size/error option that fits the global bpw budget.
- However, the --state-file option will save/use the above-mentioned computations so that future quantizations, for the same model, can be generated at normal speed. It also allows to interrupt the computation process and resume it at a later time.
The optimization target is only a proxy for the model's performance quality
- The process minimizes a per-tensor estimated error computed from sampled rows, not actual perplexity or divergence of output distributions (a future version may address this). Since errors interact nonlinearly across layers, there are no guarantees it will select the best possible quantization recipe subject to the bpw size constraint.
An imatrix with activations data is required for best results
- Activation data is required to compute the bias factor (i.e. the systematic error projected onto activation directions). If the imatrix file does not contain activation data, the --target-bpw option will refuse to run.

Models

Bits per weight, size, perplexity and KL Divergence scores

Model	BPW	Size (GB)	μPPL	𝜌PPL	μKLD	Same Top-P
Nanbeige4.3-33B-F16	16.0004	61.69	23.574336 ±0.232765	100%	N/A	N/A
Nanbeige4.3-33B-IQ1_M	1.9997	7.71	34.430471 ±0.332981	81.55%	1.230741 ±0.004763	54.973 ±0.129
Nanbeige4.3-33B-JQ1_K	1.9997	7.71	39.087353 ±0.396894	81.07%	1.273260 ±0.005082	54.431 ±0.129
Nanbeige4.3-33B-Q2_K	3.0564	11.78	25.456473 ±0.250897	94.76%	0.416241 ±0.002438	74.182 ±0.113
Nanbeige4.3-33B-JQ2_K	3.0564	11.78	23.799018 ±0.227186	95.01%	0.389655 ±0.002269	74.851 ±0.112
Nanbeige4.3-33B-Q3_K	3.9270	15.14	23.811927 ±0.234108	97.74%	0.172589 ±0.001582	83.538 ±0.096
Nanbeige4.3-33B-JQ3_K	3.9270	15.14	24.334109 ±0.242035	98.00%	0.149090 ±0.001450	84.656 ±0.093
Nanbeige4.3-33B-Q4_K	4.8425	18.67	23.971208 ±0.239591	98.95%	0.070468 ±0.001012	89.859 ±0.078
Nanbeige4.3-33B-JQ4_K	4.8425	18.67	23.993358 ±0.239506	99.12%	0.056265 ±0.000852	90.761 ±0.075
Nanbeige4.3-33B-Q5_K	5.6767	21.89	23.426267 ±0.233495	99.12%	0.055529 ±0.000965	91.532 ±0.072
Nanbeige4.3-33B-JQ5_K	5.6766	21.89	23.734256 ±0.236930	99.37%	0.033910 ±0.000721	92.991 ±0.066
Nanbeige4.3-33B-Q6_K	6.5627	25.30	23.861180 ±0.239024	99.56%	0.018520 ±0.000559	94.989 ±0.056
Nanbeige4.3-33B-JQ6_K	6.5629	25.30	23.466374 ±0.233750	99.53%	0.020032 ±0.000576	94.936 ±0.057
Nanbeige4.3-33B-Q8_0	8.4995	32.77	23.728107 ±0.237398	99.68%	0.010387 ±0.000430	96.370 ±0.048
Nanbeige4.3-33B-JQ8_0	8.5001	32.77	23.734085 ±0.237426	99.67%	0.010880 ±0.000430	96.072 ±0.050

ARC, GPQA-Diamond, HellaSwag, MMLU-Redux, Truthful QA, and WinoGrande scores

Scores generated using llama-perplexity with 750 tasks per test where possible, and a context size of 1024 tokens.

For the test data used in the generation of these scores, follow the appropriate links: ARC Challenge, Truthful QA, GPQA-Diamond, HellaSwag, MMLU-Redux, WinoGrande

Model	ARC Challenge	GPQA-Diamond	HellaSwag	MMLU-Redux	Truthful QA	WinoGrande	Avg Score
Nanbeige4.3-33B-IQ1_M	55.8667 ±1.8143	30.8081 ±3.2895	56.67	55.7333 ±1.8149	30.6667 ±1.6849	59.7333 ±1.7920	48.25
Nanbeige4.3-33B-JQ1_K	54.9333 ±1.8180	30.3030 ±3.2743	57.33	52.6667 ±1.8244	30.4000 ±1.6807	61.4667 ±1.7783	47.85
Nanbeige4.3-33B-Q2_K	67.4667 ±1.7119	24.7475 ±3.0746	71.73	73.4667 ±1.6132	33.8667 ±1.7292	67.8667 ±1.7063	56.52
Nanbeige4.3-33B-JQ2_K	65.8667 ±1.7325	27.7778 ±3.1912	71.60	69.6000 ±1.6807	34.6667 ±1.7389	67.3333 ±1.7137	56.14
Nanbeige4.3-33B-Q3_K	68.1333 ±1.7026	27.2727 ±3.1731	72.93	73.7333 ±1.6080	32.9333 ±1.7172	68.1333 ±1.7026	57.19
Nanbeige4.3-33B-JQ3_K	68.4000 ±1.6988	28.2828 ±3.2088	73.60	74.2667 ±1.5974	33.2000 ±1.7207	68.4000 ±1.6988	57.69
Nanbeige4.3-33B-Q4_K	67.8667 ±1.7063	23.7374 ±3.0314	74.00	74.9333 ±1.5836	33.6000 ±1.7259	67.6000 ±1.7100	56.96
Nanbeige4.3-33B-JQ4_K	68.6667 ±1.6949	25.7576 ±3.1156	73.73	77.2000 ±1.5330	34.1333 ±1.7325	67.6000 ±1.7100	57.85
Nanbeige4.3-33B-Q5_K	68.6667 ±1.6949	22.7273 ±2.9858	74.13	76.6667 ±1.5454	33.7333 ±1.7276	67.6000 ±1.7100	57.25
Nanbeige4.3-33B-JQ5_K	67.0667 ±1.7172	23.7374 ±3.0314	73.73	76.1333 ±1.5576	34.4000 ±1.7358	69.3333 ±1.6849	57.40
Nanbeige4.3-33B-Q6_K	67.6000 ±1.7100	24.2424 ±3.0533	74.40	75.3333 ±1.5751	32.9333 ±1.7172	68.4000 ±1.6988	57.15
Nanbeige4.3-33B-JQ6_K	67.0667 ±1.7172	24.2424 ±3.0533	74.53	76.9333 ±1.5392	34.0000 ±1.7309	68.9333 ±1.6909	57.62
Nanbeige4.3-33B-Q8_0	67.2000 ±1.7155	25.2525 ±3.0954	74.13	76.2667 ±1.5546	33.4667 ±1.7242	69.0667 ±1.6889	57.56
Nanbeige4.3-33B-JQ8_0	66.8000 ±1.7207	24.7475 ±3.0746	74.27	76.4000 ±1.5515	33.3333 ±1.7225	68.4000 ±1.6988	57.32

Tokens per second benchmarks

Scores generated using llama-bench. Standard (llama-quantize with no optimization) Q4_K_M quantization included for comparison.

model	size	params	backend	threads	test	t/s
Nanbeige4.3-33B-IQ1_M	7.71 GiB	33.12 B	MTL,BLAS	12	pp512	188.53 ± 15.38
Nanbeige4.3-33B-IQ1_M	7.71 GiB	33.12 B	MTL,BLAS	12	tg128	23.97 ± 0.32
Nanbeige4.3-33B-IQ1_M	7.71 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	38.34 ± 1.94
Nanbeige4.3-33B-Q2_K	11.78 GiB	33.12 B	MTL,BLAS	12	pp512	199.00 ± 5.44
Nanbeige4.3-33B-Q2_K	11.78 GiB	33.12 B	MTL,BLAS	12	tg128	21.08 ± 0.41
Nanbeige4.3-33B-Q2_K	11.78 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	35.38 ± 1.03
Nanbeige4.3-33B-Q3_K	15.14 GiB	33.12 B	MTL,BLAS	12	pp512	202.32 ± 8.78
Nanbeige4.3-33B-Q3_K	15.14 GiB	33.12 B	MTL,BLAS	12	tg128	20.40 ± 0.31
Nanbeige4.3-33B-Q3_K	15.14 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	36.53 ± 0.61
Nanbeige4.3-33B-Q4_K	18.67 GiB	33.12 B	MTL,BLAS	12	pp512	187.04 ± 10.81
Nanbeige4.3-33B-Q4_K	18.67 GiB	33.12 B	MTL,BLAS	12	tg128	17.56 ± 0.02
Nanbeige4.3-33B-Q4_K	18.67 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	31.22 ± 0.26
Nanbeige4.3-33B-Q5_K	21.89 GiB	33.12 B	MTL,BLAS	12	pp512	183.19 ± 6.98
Nanbeige4.3-33B-Q5_K	21.89 GiB	33.12 B	MTL,BLAS	12	tg128	15.67 ± 0.20
Nanbeige4.3-33B-Q5_K	21.89 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	28.25 ± 0.19
Nanbeige4.3-33B-Q6_K	25.30 GiB	33.12 B	MTL,BLAS	12	pp512	195.83 ± 6.82
Nanbeige4.3-33B-Q6_K	25.30 GiB	33.12 B	MTL,BLAS	12	tg128	16.85 ± 0.47
Nanbeige4.3-33B-Q6_K	25.30 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	30.56 ± 0.07
Nanbeige4.3-33B-Q8_0	32.77 GiB	33.12 B	MTL,BLAS	12	pp512	211.53 ± 7.50
Nanbeige4.3-33B-Q8_0	32.77 GiB	33.12 B	MTL,BLAS	12	tg128	14.54 ± 0.07
Nanbeige4.3-33B-Q8_0	32.77 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	26.59 ± 0.14
Nanbeige4.3-33B-JQ1_K	7.71 GiB	33.12 B	MTL,BLAS	12	pp512	194.45 ± 12.31
Nanbeige4.3-33B-JQ1_K	7.71 GiB	33.12 B	MTL,BLAS	12	tg128	24.52 ± 0.29
Nanbeige4.3-33B-JQ1_K	7.71 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	36.81 ± 0.78
Nanbeige4.3-33B-JQ2_K	11.78 GiB	33.12 B	MTL,BLAS	12	pp512	169.07 ± 15.74
Nanbeige4.3-33B-JQ2_K	11.78 GiB	33.12 B	MTL,BLAS	12	tg128	20.24 ± 1.91
Nanbeige4.3-33B-JQ2_K	11.78 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	33.75 ± 1.01
Nanbeige4.3-33B-JQ3_K	15.14 GiB	33.12 B	MTL,BLAS	12	pp512	181.60 ± 6.27
Nanbeige4.3-33B-JQ3_K	15.14 GiB	33.12 B	MTL,BLAS	12	tg128	18.30 ± 0.13
Nanbeige4.3-33B-JQ3_K	15.14 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	31.98 ± 1.17
Nanbeige4.3-33B-JQ4_K	18.67 GiB	33.12 B	MTL,BLAS	12	pp512	154.05 ± 0.45
Nanbeige4.3-33B-JQ4_K	18.67 GiB	33.12 B	MTL,BLAS	12	tg128	18.41 ± 0.38
Nanbeige4.3-33B-JQ4_K	18.67 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	31.45 ± 0.47
Nanbeige4.3-33B-JQ5_K	21.89 GiB	33.12 B	MTL,BLAS	12	pp512	145.46 ± 0.48
Nanbeige4.3-33B-JQ5_K	21.89 GiB	33.12 B	MTL,BLAS	12	tg128	16.07 ± 1.37
Nanbeige4.3-33B-JQ5_K	21.89 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	26.94 ± 0.52
Nanbeige4.3-33B-JQ6_K	25.30 GiB	33.12 B	MTL,BLAS	12	pp512	144.59 ± 0.48
Nanbeige4.3-33B-JQ6_K	25.30 GiB	33.12 B	MTL,BLAS	12	tg128	13.85 ± 0.55
Nanbeige4.3-33B-JQ6_K	25.30 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	24.91 ± 0.59
Nanbeige4.3-33B-JQ7_K	32.77 GiB	33.12 B	MTL,BLAS	12	pp512	203.61 ± 8.28
Nanbeige4.3-33B-JQ7_K	32.77 GiB	33.12 B	MTL,BLAS	12	tg128	14.11 ± 0.17
Nanbeige4.3-33B-JQ7_K	32.77 GiB	33.12 B	MTL,BLAS	12	pp1024+tg1024	26.01 ± 0.18

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the original model the better, thus the closest to 0 the better.

AI2 Reasoning Challenge (ARC): a benchmark to evaluate the ability of AI models to answer complex science questions that require logical reasoning beyond pattern matching.

GPQA-Diamond: a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.

HellaSwag: the Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations (bit of a mouthful!) is a benchmark designed to test commonsense natural language inference. It requires the model to predict the most likely ending of a sentence.

MMLU: the Massive Multitask Language Understanding evaluates LLMs’ general knowledge and problem-solving abilities across 57 subjects, including elementary mathematics, US history, computer science, and law.

Truthful QA: evaluates how well LLMs generate truthful responses to questions. It identifies whether AI models can avoid generating false or misleading information, particularly in areas where human knowledge is prone to misconceptions.

Winogrande: based on the Winograd Schema Challenge, is a natural language understanding task requiring models to resolve ambiguities in sentences involving pronoun references.

Credits

LLaMa C++ has a large and vibrant community of contributors (~1,600 last time I checked) that actively maintain and extend its functionality, adding new models and architectures almost as fast as they appear. Considering the breakneck speed at which the AI/ML field is advancing, this alone is a remarkable feat!

While I'm grateful to all contributors, I want to recognise three in particular:

Colin Kealty (Bartowski), for the many contributions and for being one of the best sources of high quality quantized models available on Hugging Face
Georgi Gerganov for his amazing work with llama.cpp and the ggml/gguf libraries
Iwan Kawrakow for being one of the key authors behind the many quantization algorithms and the imatrix functionality.

Downloads last month: 1,364

GGUF

Model size

33B params

Architecture

llama

Hardware compatibility

1-bit

2-bit

6-bit

8-bit

16-bit

View +2 variants

Model tree for eaddario/Nanbeige4.3-33B-JQK

Base model

Nanbeige/Nanbeige4.3-33B-Base

Finetuned

Nanbeige/Nanbeige4.3-33B

Quantized

(102)

this model

Dataset used to train eaddario/Nanbeige4.3-33B-JQK

Paper for eaddario/Nanbeige4.3-33B-JQK

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Paper • 2311.12022 • Published Nov 20, 2023 • 35

Experimental JQ*K global target bits‑per‑weight quantization of Nanbeige/Nanbeige4.3-33B

1. Introduction

⚠️ PLEASE READ THIS BEFORE USING THESE EXPERIMENTAL VERSIONS! ⚠️

Misconceptions about BF16 to F16 Conversion

Advantages and disadvantages of the global target bits‑per‑weight quantization process

Advantages

Disadvantages

Models

Bits per weight, size, perplexity and KL Divergence scores

ARC, GPQA-Diamond, HellaSwag, MMLU-Redux, Truthful QA, and WinoGrande scores

Tokens per second benchmarks

Metrics used

Credits

Model tree for eaddario/Nanbeige4.3-33B-JQK

Dataset used to train eaddario/Nanbeige4.3-33B-JQK

Paper for eaddario/Nanbeige4.3-33B-JQK

Experimental `JQ*K` global target bits‑per‑weight quantization of Nanbeige/Nanbeige4.3-33B