Question: mobile RAG performance?
#43
by 3morixd - opened
We use embedding models for mobile RAG — indexing personal documents on phone.
Question: what's the retrieval quality after 4-bit quantization? Does it degrade more for multilingual text?
Our tests: <2% quality loss at 4-bit for English, but we need more data for other languages.
- Dispatch AI (FZE), Sharjah UAE
Hi @3morixd
Since quality loss depends heavily on your specific use case, I recommend trying Matryoshka Representation. It's our official feature designed to help you balance speed and accuracy perfectly.