Instructions to use jtatman/sciphi-micro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jtatman/sciphi-micro with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jtatman/sciphi-micro")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jtatman/sciphi-micro")
model = AutoModelForCausalLM.from_pretrained("jtatman/sciphi-micro")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use jtatman/sciphi-micro with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jtatman/sciphi-micro"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jtatman/sciphi-micro",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jtatman/sciphi-micro

SGLang

How to use jtatman/sciphi-micro with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jtatman/sciphi-micro" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jtatman/sciphi-micro",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jtatman/sciphi-micro" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jtatman/sciphi-micro",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jtatman/sciphi-micro with Docker Model Runner:
```
docker model run hf.co/jtatman/sciphi-micro
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model Card for Model ID

This is a model with altered parameters from a mergekit slice of SciPhi/SciPhi-Self-RAG-Mistral-7B-32k.

Model Details

Model Description

This model is an experimental model using minimal slices to gather core model properties that can be further trained.

The parameters have been reduced to just under 96 million. This is an experiment to see how far slicing can be taken while retaining original weight associations.

As such, he base model is a nonsense producer, and won't return much useful. However, a suprising portion of the original sciphi model has been retained as far as gradients go.

The model will be used for layer analysis and trained on a close approximation of the sciphi datasets using trainable parameters to see what original weights might be usable.

This process will be ongoing to see if rank stabilized tuning can save and enhance the original model information through recognizing original weight associations in the preserved layers, even after model resizing.

There is a twin (parent) project with a less siginificant size reduction (600 million params) that is being used for training analysis here: jtatman/sciphi-mini-600m

Downloads last month: 10

Safetensors

Model size

96M params

Tensor type

F32

Model tree for jtatman/sciphi-micro

Quantizations

1 model