Why the tokenizer is slower than tiktoken? about tokenizers HOT 5 OPEN

Hey, could you share a reproducer?
Some things are related to the fact that we keep track of the offset and a lot of information, which tiktoken does not.
But we could only do this when ask and improve speed potentially.

from tokenizers.

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from tokenizers.

It's high in my priority to do benchmarks and improve our code if needed!

from tokenizers.

For HF, we use

from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
text = "xxx"
start = time.time()
encoded_input = tokenizer.encode(truncated_text)
end = time.time()

For tiktoken, we just initialize the tokenizer by tiktoken, all the other are the same

tokenizer = tiktoken.encoding_for_model("gpt-2")

please let me know if you need any other information

from tokenizers.

You are using GPT2Tokenizer which is the slow one. Use GPT2TokenizerFast 😅

from tokenizers.