Llama-3 offset-mapping needs fixing about tokenizers HOT 5 OPEN

@ArthurZucker Is it possible to fix this in tokenizers ?

from tokenizers.

Yep, you are right, I'll dive a bit to see why we have this!

from tokenizers.

Awesome thank you!

from tokenizers.

@ArthurZucker Is there a workaround in the meantime?

from tokenizers.

I guess a related problem is this: it adds the bos token several times at the beginning:

prompt = "Hello world"
x = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=True, add_generation_prompt=True)
y = tokenizer(tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True))["input_ids"]
x == y

x != y.

Looking at the code of apply_chat_template, tokenize=True sets add_special_tokens=False. It is not clear to me why.

from tokenizers.