🚨 Now you can run Starcoder- 2 models locally on your Mac M1 Pro Apple Silicon with 16GB memory! 🧑🏽‍💻 ⚡️✨
text
🚨 Now you can run Starcoder- 2 models locally on your Mac M1 Pro Apple Silicon with 16GB memory! 🧑🏽‍💻 ⚡️✨
new_line
new_line
Below is the UX with Twinny extension using 
text
Below is the UX with Twinny extension using 
https://huggingface.co/bigcode/starcoder2-3b
bigcode/starcoder2-3b
model
resource
https://huggingface.co/bigcode/starcoder2-3b
 for FIM and 
text
 for FIM and 
https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf
codellama/CodeLlama-7b-Instruct-hf
model
resource
https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf
 for chat. Dev tools is showing the prompt being sent to ollama server.
text
 for chat. Dev tools is showing the prompt being sent to ollama server.
new_line
new_line
Starcoder-2 is now supported in 
text
Starcoder-2 is now supported in 
llama.cpp
`llama.cpp`
inline_code
text
https://github.com/ggerganov/llama.cpp/pull/5795
https://github.com/ggerganov/llama.cpp/pull/5795
link
!
text
!
new_line
cd llama.cpp
python convert-hf-to-gguf.py ../starcoder2-3b/ --outfile models/starcoder2-3b.gguf --outtype "f16"
./quantize models/starcoder2-3b.gguf models/starcoder2-3b-Q4_K_M.gguf Q4_K_M
```
cd llama.cpp
python convert-hf-to-gguf.py ../starcoder2-3b/ --outfile models/starcoder2-3b.gguf --outtype "f16"
./quantize models/starcoder2-3b.gguf models/starcoder2-3b-Q4_K_M.gguf Q4_K_M
```
code_fence
text
new_line
For more details, please go through the following tweet thread: 
text
For more details, please go through the following tweet thread: 
https://x.com/sourab_m/status/1764583139798823235?s=20
https://x.com/sourab_m/status/1764583139798823235?s=20
link
🚨 Now you can run Starcoder- 2 models locally on your Mac M1 Pro Apple Silicon with 16GB memory! 🧑🏽‍💻 ⚡️✨

Below is the UX with Twinny extension using https://huggingface.co/bigcode/starcoder2-3b for FIM and https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf for chat. Dev tools is showing the prompt being sent to ollama server.

Starcoder-2 is now supported in `llama.cpp`  https://github.com/ggerganov/llama.cpp/pull/5795!
```
cd llama.cpp
python convert-hf-to-gguf.py ../starcoder2-3b/ --outfile models/starcoder2-3b.gguf --outtype "f16"
./quantize models/starcoder2-3b.gguf models/starcoder2-3b-Q4_K_M.gguf Q4_K_M
``` 
For more details, please go through the following tweet thread: https://x.com/sourab_m/status/1764583139798823235?s=20
🤯
osanseviero lvwerra clem Kukedlc notune jshuadvd codito rjmacarthy
❤️
clem Kukedlc Yoben BrigitteTousi rjmacarthy
🤝
clem