Implement tokenization for Ollama models in refactoring-tokenization branch about scrapegraph-ai HOT 6 CLOSED

which version are you using?

from scrapegraph-ai.

This is on the refactoring-tokenization branch. :)

It looks like Ollama does not actually provide a tokenization endpoint yet, and (perhaps because of this) the Langchain implementation of get_num_tokens is incomplete, see: ollama/ollama#1716

I think we should leave this issue open until Ollama adds a tokenization endpoint - hopefully shouldn't be too long as there is an open PR for it already.

In the meantime I will put in a PR to update the branch so that it uses the Langchain implementation, as an interim solution, which hopefully will be improved once Ollama is improved.

I did come across an alternative which is to use the hugging face package to download the tokenizer directly, however you need to know the huggingface model ID and pass a huggingface API key in order to use it (for gated models, inc Llama 3.1) and I suspect we don't want to add that complexity here if a better solution is not too far away.

from scrapegraph-ai.

I've added the tokenization code for Ollama and Mistral. I noticed that the new chunking code in pre/beta is calling the tokenizers for every word (which for Mistral takes a few seconds per word, so on long webpages it can run for a very long time) so I have switched it back to using semchunk which only seems to make a few calls to the tokenizers. Let me know if another approach is preferred.

Also for Ollama token counting to be fully correct, we are dependent on Ollama implementing token counting endpoints as per the comment above. However hopefully once that is done, langchain will implement it within their existing API and therefore no more changes will be needed in Scrapegraph.

from scrapegraph-ai.

🎉 This issue has been resolved in version 1.19.0-beta.7 🎉

The release is available on:

• v1.19.0-beta.7
• GitHub release

Your semantic-release bot 📦🚀

from scrapegraph-ai.

hi,
please update to the new beta

from scrapegraph-ai.

🎉 This issue has been resolved in version 1.20.0-beta.1 🎉

The release is available on:

• v1.20.0-beta.1
• GitHub release

Your semantic-release bot 📦🚀

from scrapegraph-ai.