Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!

Built with LangChain, GPT4All, LlamaCpp, Chroma and SentenceTransformers.

The Base Project contains much of the functionality, but struggles to run on even a moderately powerful Laptop (16GB Memory). For many use cases confidentiality of information is key - documents cannot be passed to ChatGPT online, or the cloud. The following changes were made to allow a proof of concept ingesting your documents securely on your local computer.

1. Containerisation not required to run the Project, but makes it at lot easier to setup and move between machines (e.g. avoids Python Library version conflicts). The project now contains a .devcontainer file the VSCode editor can use to spin up the appropriate Dev container on your machine. To use you will need Docker Desktop running and VSCode will prompt you if you wish to open the project in a container.
2. Updated requirements.txt as some needed Python libraries were missing. Referenced this file in containers (e.g. VSCode will auto install needed libraries if you use that tool).
3. Batch ingestation - since sharding the documents (for later search) is very memory intensive. ingest.py has been modified to only process X documents per run (value set in max_number_of_parts_per_run in this script). Script will skip any previously processed documents.
4. Offline questions - previous Web based interface impressive but very slow (approx 3 minutes per question) which is hard to Demo. Modifed the privateGPT.py script to include a list of questions at the end that get asked automatically and capture to a logfile.
5. Recording and playback - New script readerGPT.py plays back the log file at a resonable speed as if the questions were be asked / answered in a reasonable timeframe. Useful for Demos.
6. Graph (visualize.py) - The document store contains valuable information, even without the ChatGPT layer (e.g. "Given this document, find me the 5 closest documents"). Not (yet) as optimised for laptops, many laptops may stuggle to visualise larger datasets
7. Comments added to scripts - mainly to help me understand the ingestion / Q&A process - but hopefully useful for other people

In order to set your environment up to run the code here, first install all requirements:

Download the LLM model and place it in a directory of your choice:

• LLM: default to ggml-gpt4all-j-v1.3-groovy.bin. If you prefer a different GPT4All-J compatible model, just download it and reference it in your .env file.

Copy the example.env template into .env

cp example.env .env

pip3 install -r requirements.txt

and edit the variables appropriately in the .env file.

MODEL_TYPE: supports LlamaCpp or GPT4All
PERSIST_DIRECTORY: is the folder you want your vectorstore in
MODEL_PATH: Path to your GPT4All or LlamaCpp supported LLM
MODEL_N_CTX: Maximum token limit for the LLM model
MODEL_N_BATCH: Number of tokens in the prompt that are fed into the model at a time. Optimal value differs a lot depending on the model (8 works well for GPT4All, and 1024 is better for LlamaCpp)
EMBEDDINGS_MODEL_NAME: SentenceTransformers embeddings model name (see https://www.sbert.net/docs/pretrained_models.html)
TARGET_SOURCE_CHUNKS: The amount of chunks (sources) that will be used to answer a question

Note: because of the way langchain loads the SentenceTransformers embeddings, the first time you run the script it will require internet connection to download the embeddings model itself.

This repo uses a state of the union transcript as an example. Replace this with your own documents in the source_documents and be sure to add to the .gitignore file.

Put any and all your files into the source_documents directory

The supported extensions are:

• .csv: CSV,
• .docx: Word Document,
• .doc: Word Document,
• .enex: EverNote,
• .eml: Email,
• .epub: EPub,
• .html: HTML File,
• .md: Markdown,
• .msg: Outlook Message,
• .odt: Open Document Text,
• .pdf: Portable Document Format (PDF),
• .pptx : PowerPoint Document,
• .ppt : PowerPoint Document,
• .txt: Text file (UTF-8),

Run the following command to ingest all the data.

python ingest.py

Output should look like this:

Creating new vectorstore
Loading documents from source_documents
Loading new documents: 100%|██████████████████████| 1/1 [00:01<00:00,  1.73s/it]
Loaded 1 new documents from source_documents
Split into 90 chunks of text (max. 500 tokens each)
Creating embeddings. May take some minutes...
Using embedded DuckDB with persistence: data will be stored in: db
Ingestion complete! You can now run privateGPT.py to query your documents

It will create a db folder containing the local vectorstore. Will take 20-30 seconds per document, depending on the size of the document. You can ingest as many documents as you want, and all will be accumulated in the local embeddings database. If you want to start from an empty database, delete the db folder.

Note: during the ingest process no data leaves your local environment. You could ingest without an internet connection, except for the first time you run the ingest script, when the embeddings model is downloaded.

Note (Modification): The number of document (chunks) processed during each run is limited to allow for reduced memory. You may need to run the ingest script several times for all your documents to be ingested. Previously ingested documents are ignored during subsequent runs. Search for max_number_of_parts_per_run in the ingest.py script to change this limit.

In order to ask a question, edit the list of questions at the of privateGPT.py. Then run the script

python privateGPT.py

And watch as the for the script asks and captures the answers to your questions. You'll need to wait 1 to 3 minutes per question(depending on your machine) while the LLM model consumes the prompt and prepares the answer. Once done, it will print the answer and the 4 sources it used as context from your documents.

Once you have asked some questions, you no doubt will want to share with colleagues. To replay the session (at a much faster speed), using the logs and with coloured output, run the following output.

python readerGPT.py

Note: you could turn off your internet connection, and the script inference would still work. No data gets out of your local environment.

Selecting the right local models and the power of LangChain you can run the entire pipeline locally, without any data leaving your environment, and with reasonable performance.

• ingest.py uses LangChain tools to parse the document and create embeddings locally using HuggingFaceEmbeddings (SentenceTransformers). It then stores the result in a local vector database using Chroma vector store.
• privateGPT.py uses a local LLM based on GPT4All-J or LlamaCpp to understand questions and create answers. The context for the answers is extracted from the local vector store using a similarity search to locate the right piece of context from the docs.
• GPT4All-J wrapper was introduced in LangChain 0.0.162.

To use this software, you must have Python 3.10 or later installed. Earlier versions of Python will not compile.

If you encounter an error while building a wheel during the pip install process, you may need to install a C++ compiler on your computer.

To install a C++ compiler on Windows 10/11, follow these steps:

1. Install Visual Studio 2022.
2. Make sure the following components are selected:
   • Universal Windows Platform development
   • C++ CMake tools for Windows
3. Download the MinGW installer from the MinGW website.
4. Run the installer and select the gcc component.

When running a Mac with Intel hardware (not M1), you may run into clang: error: the clang compiler does not support '-march=native' during pip install.

If so set your archflags during pip install. eg: ARCHFLAGS="-arch x86_64" pip3 install -r requirements.txt

This is a test project to validate the feasibility of a fully private solution for question answering using LLMs and Vector embeddings. It is not production ready, and it is not meant to be used in production. The models selection is not optimized for performance, but for privacy; but it is possible to use different models and vectorstores to improve performance.