This a repo to create a QA Retrieval Bot using both Open LLM as well as OpenAI in Databricks.
To get started please change the configuration notebook in util/notebook-config.py.
Please look at the documentation Below on configuration for specific LLM's
Please change the util/notebook-config.py to point to the location where your files are stored.
config['loc'] = <"Location where the files are stored">
Currently the code supports the following version
- Open AI
- MPT-30b Chat version
- Llama-2-13b HF-chat version
- Llama-2-70-b HF-chat version with 8-bit Quantization
The following code is tested on ML DBR GPU 13.2 Runtime
The Code to run open LLMS has been tested on the below single node cluster configurations:
- AWS : g5-12xlarge [4 A10's]
- Azure : NC24Ads_A100_v4 [1 A100]
The TGI pipeline has support to run on older Generation GPU's like the V100's but has not been tested extensively
There are two ways to convert the PDF to TXT
- Using Azure Form Recognizer To use form recognizer you need to add
config['use_azure_formrecognizer'] = True
and add the URL and KEY from the Azure portal
config['formendpoint']
config['formkey']
- Using Langchain PDF converter set
config['use_azure_formrecognizer'] = False
You can use any cluster to run the OpenAI model and need to set the following configs
set model_id t0 openai
config['model_id'] = 'openai'
os.environ['OPENAI_API_KEY'] = '<your open AI API Key>'
Note : when using OpenAI you do not need to run the 02_load_model Notebook and Azure open AI keys would need extra configurations to be added
To use LLAMA-2 models, you need to agree to the terms and condition of HuggingFace and provide an API key to download the models Refer to these steps to download the key : https://huggingface.co/docs/api-inference/quicktour#get-your-api-token and set the below parameters
config['model_id'] = 'meta-llama/Llama-2-XXb-chat-hf'
config['HUGGING_FACE_HUB_TOKEN'] = '<your HF AI API Key>'
Note : to need to keep 02_load_model Notebook running to have the API running
you need to set the below config
config['model_id'] = 'mosaicml/mpt-30b-chat'
Note : to need to keep 02_load_model Notebook running to have the API running
The current notebook use the following the embedding models
- For openAI : text-embedding-ada-002
- All open LLM's : intfloat/e5-large-v2
The open LLM embedding can be changed by over-riding the Dictionary in utils/notebook-config.py