Instructions to use CreitinGameplays/Mistral-Nemo-12B-R1-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CreitinGameplays/Mistral-Nemo-12B-R1-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CreitinGameplays/Mistral-Nemo-12B-R1-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CreitinGameplays/Mistral-Nemo-12B-R1-v0.1")
model = AutoModelForCausalLM.from_pretrained("CreitinGameplays/Mistral-Nemo-12B-R1-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CreitinGameplays/Mistral-Nemo-12B-R1-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CreitinGameplays/Mistral-Nemo-12B-R1-v0.1

SGLang

How to use CreitinGameplays/Mistral-Nemo-12B-R1-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CreitinGameplays/Mistral-Nemo-12B-R1-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use CreitinGameplays/Mistral-Nemo-12B-R1-v0.1 with Docker Model Runner:
```
docker model run hf.co/CreitinGameplays/Mistral-Nemo-12B-R1-v0.1
```

Manual Activation of Thinking/Output block

#25

by DavidAU - opened Mar 29, 2025

Discussion

DavidAU

Mar 29, 2025

Excellent job on the model... been looking for this!!!

Manual activation:
I found that using "chatml" template (manually selected) and this system prompt:

You are an AI focused on providing systematic, well-reasoned responses. Response Structure: - Format: {reasoning}{answer} - Process: Think first, then answer.

Activates the thinking "block" and output generation.

However, I see at the source repo the "tokenizer... json" was updated 4 days ago, so this might fix the issue with the "jinja template" / "block" reasoning activation.

Going to download source and locally quant...

CreitinGameplays

Owner Mar 29, 2025

•

edited Mar 29, 2025

Oh thanks for your comment! Could you provide me in details an example on how did you do that?

Also, didn't the model get stuck in an infinite response loop? And what parameters you set?

DavidAU

Mar 30, 2025

Hey;

Seems some of the copy/paste did not come thru (put extra spaces in the "think" tags):

SYSTEM PROMPT:

You are an AI focused on providing systematic, well-reasoned responses. Response Structure: - Format: < think >{reasoning}</ think >{answer} - Process: Think first, then answer.

USAGE:
Lmstudio ; developer mode -> entered "system prompt", set "chat template" to "chatml"

TEMPS: .6 ;
Found temps .1 ish best for solving ; but got loops sometimes ; whereas temps over 1 reduction in loops (both "thinking" and "output" loops).
These temps seems to be mistral specific, as other "thinking mistrals" work/solve best at very low temps.

(other params; Rep pen 1.1 , TopK 40 , topP .95, minP .05; Rep pen range: 64-128 (helps keep reasoning on track / quality of output)

Looping issues (output, and maybe thinking) can be filtered out using parameters like rep pen range, rep pen and/or DRY settings.

NOTE: Without system prompt; "thinking" works, followed by output... but not in a "thinking" block.

CreitinGameplays

Owner Apr 18, 2025

•

edited Apr 18, 2025

Hi @DavidAU , thank you for responding my last comment! I made a new finetune on this model, on a better reasoning dataset (I believe), could you take a look on it? Its CreitinGameplays/Mistral-Nemo-12B-R1-v0.2
Edit: I did some tests on the new finetuned model (on Kaggle) and i think the looping issues was kinda fixed (at least it didn't happen to me yet, but seems way much better than v0.1 imo)

DavidAU

Apr 18, 2025

@CreitinGameplays
Excellent , I will download (source) / try it out.
If I may ask - did you target specific layers during tuning or entire model overall?

Thanks - and great work!

CreitinGameplays

Owner Apr 18, 2025

Yes I finetuned the entire model 😉.
Have a great day!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment