A simple demo ChatBot application that provides both an LLM and a RAG (Retrieval-Augmented Generation) based Chatbot to interact with a locally running LLM (Large Language Model).
|
Warning
|
Currently this is relatively unoptimized with certain inefficiencies regarding working with embeddings. This will be speeded up in future releases. |
With relevant embeddings the RAG Chatbot will answer queries on Red Hat OpenShift AI and Red Hat Open Data Science (ODS) that the pure LLM-based Chatbot will struggle with.
Clone repository
1) Create a new directory (e.g. ai-rag)
2) Clone the repository inside the new directory
$ mkdir ai-rag
$ cd ai-rag
$ git clone https://github.com/rhpds/ai-chatbots.gitRun LLM Locally
1) Install the LLM runtime Ollama from here or follow the instructions in the installation guide.
2) Start Ollama running on your local machine either via the Ollama App you downloaded or via the command line ollama serve.
3) In a separate terminal window, launch the the mistral LLM
$ ollama run mistral|
Note
|
The first time this will download the 'mistral' LLM and may take a few minutes. mistral is approximately 4GB in size.
|
There are two methods to run the ChatBot application:
-
Run in a container (easier method)
-
Run using Python
|
Note
|
Older or low-memory machines may have performance issues running the application. Ideally a relatively recent laptop, or machine, with at least 32GB of RAM locally. |
This is the recommended way to run the ChatBot application.
podman run --rm \
--name ragbot \
-e OLLAMA_BASE_URL="http://host.docker.internal:11434" \
-p 7001:7001 \
-p 7002:7002 \
-p 7003:7003 \
tonykay/ai-rag-chatbot:0.6.6-
Browse to your Chatbot of Choice - or both
| Landing Page | URL |
|---|---|
LLM Chatbot (no RAG) |
|
RAG Chatbot |
|
Both Chatbots side-by-side |
This method requires Python 3.11 or higher and assumes you will use a virtual environment, or venv, called venv-chatbot to install the dependencies.
-
Create a Python Virtual Environment (venv)
venvfor the Python dependenciespython -m venv venv-chatbot
NoteThis will create a new directory called venv-chatbotin your current working directory.gitwill ignore this directory by default as it is already added to your.gitignore. -
Activate your new
venv:source venv-chatbot/bin/activateNoteYour prompt should now show (venv-chatbot)to indicate the virtual environment is active. However, this may vary depending on your shell. -
Install the dependencies using the requirements file. First:
python -m pip install -r requirements.txt
Sample Output (Your output may differ and include a message about upgrading pip)<TRUNCATED> asn1-0.5.1 pyasn1-modules-0.3.0 pydantic-2.6.3 pydantic-core-2.16.3 pyjwt-2.8.0 pymupdf-1.23.26 pypika-0.48.9 pyproject_hooks-1.0.0 python-dateutil-2.9.0.post0 python-dotenv-1.0.1 python-engineio-4.9.0 python-graphql-client-0.4.3 python-multipart-0.0.6 python-socketio-5.11.1 regex-2023.12.25 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 safetensors-0.4.2 scikit-learn-1.4.1.post1 scipy-1.12.0 sentence_transformers-2.5.1 simple-websocket-1.0.0 six-1.16.0 sniffio-1.3.1 starlette-0.32.0.post1 sympy-1.12 syncer-2.0.3 tenacity-8.2.3 threadpoolctl-3.3.0 tiktoken-0.6.0 tokenizers-0.15.2 tomli-2.0.1 torch-2.2.1 tqdm-4.66.2 transformers-4.38.2 typer-0.9.0 typing-extensions-4.10.0 typing-inspect-0.9.0 uptrace-1.22.0 urllib3-2.2.1 uvicorn-0.25.0 uvloop-0.19.0 watchfiles-0.20.0 websocket-client-1.7.0 websockets-12.0 wrapt-1.16.0 wsproto-1.2.0 yarl-1.9.4 zipp-3.17.0
-
Run the application:
./entrypoint.sh
Sample Output<TRUNCATED> 2024-03-25 06:48:52 - Use pytorch device_name: mps 2024-03-25 06:48:54 - Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information. File path is /Users/tok/Dropbox/PARAL/Projects/redhat-rag-demo/ai-chatbots/chatbot-rag/rag-persist/record_manager_cache.sql Indexing stats: {'num_added': 334, 'num_updated': 0, 'num_skipped': 0, 'num_deleted': 0} 2024-03-25 06:49:01 - Your app is available at http://localhost:7002 huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) ::1 - - [25/Mar/2024 06:49:51] "GET grv/ HTTP/1.1" 200 - -
Browse to
http://localhost:7003to access the application. The right hand iFrame will show the RAG Chatbot and the left hand iFrame will show the LLM Chatbot.
|
Note
|
The LLM Chatbot is available at http://localhost:7001 and the RAG Chatbot is available at http://localhost:7002 if you prefer to use them separately. And that RAG Chatbot can take a few moments to start.
|
The application is a simple web interface that allows you to ask questions about Red Hat OpenShift AI and Red Hat Open Data Science (ODS). The application will use the locally running LLM and RAG to answer your questions.
|
Note
|
The first time you ask a question it may take a few seconds to respond as the application will need to generate embeddings for the question and the documents in the database. |
-
chainlit
-
LangChain
-
chromadb - Chroma Database Vector Store to store and retrieve document chunks
-
Hugging Face
sentance_transformers- embeddings * * *
Podman and Docker differ,
Docker cmd:
Podman cmd:
Add --uidmap 1000:0:1 --uidmap 0:1:1000
podman run --uidmap 1000:0:1 --uidmap 0:1:1000 --rm --name ragnar -e OLLAMA_BASE_URL="http://host.docker.internal:11434" -v $(pwd):/home/user/app -p 7861:7860 tonykay/ai-rag-chatbot:0.1.0*
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent