HuggingFace Model Differences about genre HOT 2 CLOSED

Hi

I've now found the cause of a number of the other remaining difference (whitespace and subsequent offset differences). This is due to the tokenizer in Hugging Face having a default behaviour which is not required to keep it the same as FS. When calling it to decode the token ids to a string you can set the following flag "clean_up_tokenization_spaces" to False, again this can be used in the examples directly but is also needed in the appropriate utils methods for HF.

Also to ensure identical results for longer text the max_length should be set to 202 not 200 as i mentioned above as FS treats the eos as extra and there seems to be something else as 201 was not sufficient. With all this i've run it over several thousands of texts (truncating inputs to 255 characters) and got identical outputs from FS and HF.

Tony

from genre.

Fixed! 🙂

from genre.