How to use other local models about wikichat HOT 2 CLOSED

Hi!

Right now, we support models like LLaMA via the Together API. You can set it up by following the instructions at https://github.com/stanford-oval/WikiChat#set-up-the-llm-endpoint.

Alternatively, you can use HuggingFace's TGI to host a local model. For that, you can use their Docker image by running a command like docker run --gpus all --shm-size 1g -p <port>:80 -v ./workdir:/data ghcr.io/huggingface/text-generation-inference:1.3.4 --model-id <model-name-from-the-huggingface-hub> --hostname 0.0.0.0 --num-shard <number-of-gpus> where <port> is the same port you set in https://github.com/stanford-oval/WikiChat/blob/main/llm_config.yaml#L43.

However, using a model like LLaMA-2-13B as is leads to poor results, as it is not good at following our few-shot prompts. So you might have a better shot with higher-quality models like Mistral/Mixtral, or instruction-tuned models.

We are planning to release small distilled (i.e. fine-tuned specifically for WikiChat) models soon, which you can host on a single GPU yourself, using TGI with the above command. These models perform much much better raw LLMs like LLaMA-2, so please stay tuned!

from wikichat.

Update: I've added the distilled models I mentioned. You can give them a try!

from wikichat.