[Feature Request] Conversational Search (RAG) with a local LLM about typesense HOT 10 OPEN

I misspoke earlier. Turns out that we actually added support for vLLM through which you can run several local LLMs. Just haven't documented it yet.

Will post a link here once we update the docs.

from typesense.

I misspoke earlier. Turns out that we actually added support for vLLM through which you can run several local LLMs. Just haven't documented it yet.

Will post a link here once we update the docs.

Thats huge. If you manage to make this easily accessible it could be quite the hype.
Looking forward to try it!

from typesense.

It takes only care of the R part of RAG but yes, custom models and using GPU are supported.
And you also have the option to generate the embeddings yourself and store them.

Check out all the subtopics of this part of the documentation:
https://typesense.org/docs/26.0/api/vector-search.html#index-embeddings

from typesense.

@piccaso Typesense does support the "AG" part of RAG, by integrating with ChatGPT / Cloduflare APIs: https://typesense.org/docs/26.0/api/conversational-search-rag.html

@elliot-sawyer We don't yet have a way to integrate with local LLMs. But I'll leave this open as a feature request.

from typesense.

May I know which local LLMs you're looking for?

from typesense.

I don't have a particular one in mind yet - would any of the Typesense models on HuggingFace be appropriate? I'll have an NVIDIA A100 available in a couple of months to do some Typesense work with, but only on the stipulation that I use a locally downloaded LLM (no network or API keys).

from typesense.

@jasonbosco It's nice to know that this is being done. I already started testing this myself, without documentation (heh) and ran into an authorization problem. Now api_key is not used by vLLM at all, could you add this? In case when LLM inference on different server it's is important to have authentication.

from typesense.

CC: @ozanarmagan

from typesense.

I misspoke earlier. Turns out that we actually added support for vLLM through which you can run several local LLMs. Just haven't documented it yet.

Will post a link here once we update the docs.

It would be great if we could use any OpenAI compatible APIs like Ollama LLama.cpp VLLM and etc, i think that would save everybody a lot of headache, we just need a way to set the openai api url in the creation of the model.

from typesense.

You can now customize the openai api base url: https://typesense.org/docs/26.0/api/vector-search.html#using-openai-compatible-apis

from typesense.