This is a simple Streamlit application that allows users to input a prompt and submit it to a local runningn LLM endpoint. The application then displays the result from the endpoint on the same page. The endpoint is a llama.cpp server that is running on localhost.
-
Make sure you have Python installed on your machine.
-
Install the required libraries by running:
pip install streamlit requests
-
Save the provided Python script (e.g.,
llm_chat_st_app.py
) on your local machine.
llama.cpp should be installed and running in server mode on localhost http://0.0.0.0:8080
https://github.com/ggerganov/llama.cpp.git
./llama.cpp/server -c 4096 --host 0.0.0.0 -t 16 --mlock -m ./my_models/TheBloke/LLama2-7B-chat-GGUF/llama-2-7b-chat.Q4_K_M.gguf
Open a terminal and navigate to the directory containing the script. Run the following command to start the Streamlit app:
streamlit run llm_chat_st_app.py
This will launch a local development server, and you can access the application in your web browser.
- Enter a prompt in the text input field.
- Click the "Submit" button or press "Enter" to send the prompt to the specified endpoint.
- The result from the endpoint will be displayed on the same page.
Note: The application uses Streamlit's form functionality to capture the "Enter" key press.
- Streamlit
- Requests
Orhan Cavus