Support for Langchain's acall method about trulens HOT 7 CLOSED

Right I was facing some issues when I tried doing pip install -e ., so I instead removed the package, and added my cloned trulens repo to my PYTHONPATH. May have been the cause of the issue, I didn't use npm run dev, and only changed it to False as I was facing some error when trying to view the Evaluations page on the dashboard.

I think it was pretty smooth sailing using it, and probably wouldn't have been an issue if I pip install the package the right way.

Thanks for checking! Saw your profile, I'm from Singapore as well btw, good to meet you!

from trulens.

Merged today here, and coming in the release this week (0.8.0)!

Feel free to run from main to get this update before then as follows:

cd trulens/trulens_eval
pip install -e .

from trulens.

Hey, thanks so much for the help with building this!

Found a couple issues while testing it out.

Just as a note, I had changed the _RELEASE variable to False for the streamlit dashboard to work.

A strange finding is that, currently the async streaming only works with TruLens when placing the AsyncIteratorCallbackHandler in the LLM's constructor, but does not work when placed in the LLMChain, or directly in the request call function of the LLMChain, have tried out both ways below, which either raise an error or output nothing. I unfortunately require the use case of the callback in the request call function, when I am calling the chain, as I can potentially get interrupts with my chains, which I currently handle by passing new asynciteratorcallbackhandlers for each call.

chain = LLMChain(llm=llm, prompt=prompt, callbacks=[callback])

f_res_record = asyncio.create_task( #the usecase i would like 
        tc.acall_with_record(
            inputs=dict(question=message),
            callbacks=[callback],
        )
    )

Additionally, another error I've noted is when I try to integrate memory into my LLMChain, I receive the following API Error. I have created a gist where the error can be reproduced.

https://gist.github.com/anunayajoshi/bef85fa2941a8e81cc61732182daaded

The error I receive is as follows -

File "D:\Coding\trulens\trulens_eval\trulens_eval\provider_apis.py", line 263, in post
    raise RuntimeError(error)
RuntimeError: You need to specify either `text` or `text_target`.

I hope I've provided sufficient information for the issues faced, let me know how I can help!

from trulens.

Hi. Thanks for the feedback. Can you take a look at #340 to check if this allows your use case of callback to acall_with_record?

from trulens.

Hi @anunayajoshi 👋

Thanks again for filing this issue. I wanted to check with you regarding:

Just as a note, I had changed the _RELEASE variable to False for the streamlit dashboard to work.

Just to confirm, you followed @joshreini1's suggestion above and installed trulens, but found that you had to change _RELEASE in init.py to false to get the dashboard to work? Were you running npm run dev while doing this? What version of trulens were you on before pulling the code and testing out the changes?

Thanks in advance for your reply! In addition, this PR should bundle in the built react components in the repo from now on, so we likely won't need the npm step in the and hopefully makes it easier to test the latest code straight from the repository.

from trulens.

That's interesting... not sure why that is happening, but will look into it nonetheless. Thanks for providing the details, and glad to hear that you had an overall great experience. :)

woohoo! 🇸🇬 great to meet you too (also your temperature taking bot project 👀 )

from trulens.

Closing this one out for the acall fix.

from trulens.