A frontend for large language models like ๐จ Koala or ๐ฆ Vicuna running on CPU with llama.cpp, using the API server library provided by llama-cpp-python. NOTE: I had to discontinue this project because its maintenance takes more time than I can and want to invest. Feel free to fork :)
This looks great, but I seem to have an issue with importing the llama-cpp api server.
File "~/Eucalyptus-Chat/api-server.py", line 22, in
uvicorn.run(server.app, host=args.host, port=args.port)
AttributeError: module 'llama_cpp.server' has no attribute 'app'
I've tried referring to app.py in llama-cpp-python/llama_cpp/server directly and then the api-server starts but throws a module error. not sure if this is a bug or something strange on my system.