Instructions to use typeof/mistral-3.3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/mistral-3.3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/mistral-3.3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/mistral-3.3B")
model = AutoModelForCausalLM.from_pretrained("typeof/mistral-3.3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/mistral-3.3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/mistral-3.3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/mistral-3.3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/mistral-3.3B

SGLang

How to use typeof/mistral-3.3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/mistral-3.3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/mistral-3.3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/mistral-3.3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/mistral-3.3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/mistral-3.3B with Docker Model Runner:
```
docker model run hf.co/typeof/mistral-3.3B
```

What's this model?

by KnutJaegersberg - opened Oct 28, 2023

Discussion

KnutJaegersberg

Oct 28, 2023

Is it chopped only or further fine tuned and ready to be used?

mrfakename

Nov 10, 2023

Any update @typeof ?

typeof

Owner Nov 13, 2023

@KnutJaegersberg @mrfakename apologies for the delay... mistakenly made this public 🫠
It is a part of an ongoing research project in model distillation...
Just updated to most recent improvements - feel free to try it out, share, and let us know what you think!

Hopefully will have more details to share soon!

mrfakename

Nov 13, 2023

•

edited Nov 14, 2023

Hi @typeof , thanks for making this model! I see the license is CC-BY instead of Apache. Are you planning to change this, or do you intend to keep it as CC-BY?

KnutJaegersberg

Nov 14, 2023

I felt like attempting to fine tune it, but I cant use the model that came out of it. some error on sentencepiece side.

imi2

Nov 19, 2023

It has a high perplexity,
here is minima: [1]5.6161,[2]6.6278,[3]7.0043,
here is mistral-3.3b: [1]79.7248,[2]131.6063,[3]126.4081,
here is marx-3.3b: [1]7.2258,[2]8.3805,[3]9.0478,

output text:

./main -m ~/Storage/typeof_mistral-3.3B/mistral-3.3B-Q4_K_M.gguf -p "I have"

I have been working on my own project lately, but I still need to do some research and work on them as soon as possible. I think we are ready for these new game features for their next game in the series. [end of text]

KnutJaegersberg changed discussion status to closed Dec 6, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment