How to add extra tokens in tiktoken? about tiktoken HOT 5 CLOSED

You can create your own Encoding object by passing in a dict from token-bytes to the integer token value.
See this:

from tiktoken.

If you want to have your own encoding registered so that tiktoken.get_encoding(...) works, you can create a new submodule under the tiktoken_ext namespace. This ability to extend tiktoken's registry is currently undocumented. See

from tiktoken.

Thank you so much for the swift reply! I'll have a try.

from tiktoken.

Sorry for "hijacking", this isssue, but feel it is related enough, instead of opening a separate issue.

When looking at https://huggingface.co/docs/tokenizers/api/added-tokens, what would be the best way, to (as much as possible) stay in-sync with the original tokenizer configs), when adding special tokens in tiktoken?

As to match, rstrip, lstrip and normalized, for example. Understandable if not all is possible (yet) in tiktoken, but any further recommendations, as to what is possible currently and how, move current GPT2 based encoders, with added special tokens, to tiktoken, would be appreciated!

from tiktoken.

I've added more documentation for this over here: https://github.com/openai/tiktoken#extending-tiktoken

from tiktoken.