feat: add streaming and websocket support for all chain types about lanarky HOT 13 CLOSED

Amazing project!

My use case would be an LLM chatbot, which requires state. FastAPI supports WebSockets which might be a good solution here, or is there another approach possible?

from lanarky.

You mean by keeping track of history in the client? And thus sending it again at every request?

yup yup. for example: https://github.com/mckaywrigley/chatbot-ui.

I'll have a look if I am able to. Maybe you can point out where in your code it would fit best?

Yes sure. I think the responses module seems best for this: https://github.com/ajndkr/fastapi-async-langchain/tree/main/fastapi_async_langchain/responses. You can do the following:

• create base websocket response
• create base callback class for websocket inside callbacks module
• create websocket response for LLMChain (I can take over the rest)

from lanarky.

i added 'LLMChain' in Base Models in /home/outscale/miniconda3/envs/react-qback/lib/python3.9/site-packages/lanarky/routing/utils.py and it worked

['LLMChain', 'ConversationChain', 'AgentExecutor', 'RetrievalQAWithSourcesChain', 'ConversationalRetrievalChain']

from lanarky.

hi! i think it depends on how you are building the frontend but I do like the idea of adding support for websockets.

It's already implemented in https://github.com/hwchase17/chat-langchain but it's better if the implementation can be added to this package as well. I will create a feature request for this.

Would you like to contribute towards it?

from lanarky.

hi! i think it depends on how you are building the frontend

You mean by keeping track of history in the client? And thus sending it again at every request?

but I do like the idea of adding support for websockets.

It's already implemented in https://github.com/hwchase17/chat-langchain but it's better if the implementation can be added to this package as well. I will create a feature request for this.

Cool!

Would you like to contribute towards it?

I'll have a look if I am able to. Maybe you can point out where in your code it would fit best?

from lanarky.

@ajndkr Sorry, but I don't have enough time for it. If I was a bit more familiar with FastAPI I'd do it, but I have only worked with Flask for APIs.

from lanarky.

no problem! i will look into this. will let you know once it's ready.

from lanarky.

Great, thanks man!

from lanarky.

@ajndkr can I contact you somehow? I'm struggling with something similar to your project, maybe you can help me out.

from lanarky.

@pors you can find me on twitter! https://twitter.com/ajndkr

from lanarky.

Using LLMChain doesn't work.

File "/home/outscale/miniconda3/envs/react-qback/lib/python3.9/site-packages/lanarky/routing/utils.py", line 88, in create_request_from_langchain_dependency
    raise TypeError(
TypeError: Error! Creating a Request model for 'LLMChain' is not currently supported by 'LangchainRouter.add_langchain_api_route()'.
Available chain types: ['ConversationChain', 'AgentExecutor', 'RetrievalQAWithSourcesChain', 'ConversationalRetrievalChain']

To use a custom chain type, you must define your own FastAPI endpoint.

I'm looking into a way to use a chat system but without using the memory since the history of messages is stored inside a database and there are different users using the chatbot.
So i can't use ConversationChain. I'm trying with LLMChain instead but it looks like it is not supported

from lanarky.

good catch! @AmineDjeghri

from lanarky.

closing this issue as I currently have no future plans to add support for all chain and agent types. I will add support iteratively based on demand/popularity.

from lanarky.