Guidance on supporting TGI-based LLM about datadreamer HOT 3 CLOSED

Hi, thanks for looking at integrating this. Before looking at this further, we do actually have TGI support I believe.

See the HFAPIEndpoint class: https://datadreamer.dev/docs/latest/datadreamer.llms.html#datadreamer.llms.HFAPIEndpoint

You can see that it uses InferenceClient from huggingface_hub under the hood:

which is compatible with TGI according to this page: https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/consuming_tgi#inference-client

Does this solve your need?

from datadreamer.

Perfect, yep, I can explain those, the answers are a little complicated.

1. Even though you are specifying lazy=False, your Prompt step is indeed running lazily due to the :memory: option (in that code path it actually requires a folder to write results.) When a step is run lazily, internally we call the iterator backing the lazy result just to get the first element (a "peek" operation) in order to check what it's type is, since we can't detect the type with an iterator. So the extra calls are coming from these "peek" operations. You generally don't need to be concerned with those though for your implementation though as an end user and they don't add any significant overhead for two reasons: 1) these peeks only read the first element, so if your lazy iterator returns 1 million rows, only the first element is read, so the overhead doesn't scale with the number of rows 2) results from the LLM are cached.

I can work on trying to see if I can remove that constraint so it doesn't run lazily in the future though. I will take this up as an enhancement.

2. No examples at this time other than this page, but I think this page should have everything you need: https://datadreamer.dev/docs/latest/pages/advanced_usage/creating_a_new_datadreamer_.../step.html

and feel free to ask any questions if you run into any trouble.

from datadreamer.

@AjayP13 Thanks! This helps with my issue to use TGI. Two follow-up questions -

1. I am planning to implement some custom steps for my use-case so trying to understand what was the issue with my test script. Why do I see get_generated_texts called printed 6 times
2. I am interested in implementing a SuperStep to generate multi-turn synthetic data. Do you have any examples I can refer to?

from datadreamer.