Tokenizer is not behaving as expected on special tokens (doesn't recognize `pad` and `eos` tokens) about seqio HOT 1 CLOSED

Hi @armancohan, this is working as expected. eos, pad, etc. are special ids (with corresponding labels for human readability) and are used after encoding to denote special meaning, e.g. denotes that a given piece of text was not found in the vocab, so if "armancohan" isn't in the vocab, running decode(encode("armancohan")) will give you ''. But, these labels themselves may or may not be in the vocab, e.g. encode("") may map to 87 denoting that it wasn't found in the vocab, or to one or more vocab ids if it belongs to the vocab. Similarly, 1 is appended at the end of ids to denote end of sequence and has a label "" to make it human readable, but actually encoding "" won't (and shouldn't) return the eos id.

from seqio.