Run a local AI from Django with Llama.cpp

Clone the repository and install the dependencies:

git clone https://github.com/emencia/django-local-ai
cd django-local-ai
make install

In case of an error Failed to build llama-cpp-python try this and run again:

source .venv/bin/activate
export SETUPTOOLS_USE_DISTUTILS=stdlib
pip install --upgrade --force-reinstall setuptools

Get the websockets server:

make installws

To make this work you need a Llama.cpp ggml compatible language model. For example we will use GPT4All-13B-snoozy from this repository:

cd some/dir/where/to/put/your/models
wget https://huggingface.co/TheBloke/GPT4All-13B-snoozy-GGML/resolve/main/GPT4All-13B-snoozy.ggmlv3.q5_0.bin

Change your settings accordingly by updating the MODEL_PATH setting in main/settings.py. Use an absolute path.

First, append this to your Django settings (./main/settings.py):

CENTRIFUGO_HOST = "http://localhost"
CENTRIFUGO_PORT = 8427
CENTRIFUGO_HMAC_KEY = "b4265250-3672-4ed9-abe5-b17ca67d0104"
CENTRIFUGO_API_KEY = "21dae466-e63e-41d1-8f75-3aa94b41c893"
SITE_NAME = "django-local-ai"

and don't forget to finish the setup:

.venv/bin/python manage.py initws --settings=main.settings.local

Now, you're ready to enjoy your local LLM !

Run the http server:

make run

Run the task queue that will handle language model calls:

make lm

Run the websockets server:

make ws

Create a superuser:

make superuser

Open the frontend at localhost:8000 and login