Are there any current methods to speed up inference?
#5
by zhouchongqin - opened
At present, the model doesn't support vllm. Are there any current methods to speed up inference?
Thanks for your interest. However, GVE does not support any acceleration framework. You can still use Transformers and try parallel inference on multiple GPUs.