Possible ByteVocabulary Bug about seqio HOT 2 OPEN

@nconstant-google this does look like a bug

from seqio.

@nconstant-google this does look like a bug

This can be reproduced by running a pretrained byt5 model (from https://github.com/google-research/byt5) using the following command (I'm using GPU, I believe the error would persist with TPUs):

export BUCKET=~/data/t5/byt5-data/
export TASK=byt5_ner_multilingual
export MODEL_DIR="${BUCKET}/${TASK}"

python -m t5.models.mesh_transformer_main
--module_import="byt5.tasks"
--model_dir="${MODEL_DIR}"
--gin_file="${MODEL_DIR}/operative_config.gin"
--t5_tfds_data_dir="${BUCKET}/t5-tfds"
--gin_file="eval.gin"
--gin_file="beam_search.gin"
--gin_param="utils.run.mesh_shape = 'model:2,batch:1'"
--gin_param="utils.run.mesh_devices = ['gpu:0','gpu:1']"
--gin_param="run.dataset_split = 'validation'"
--gin_param="MIXTURE_NAME = 'byt5_ner_eval.ja'" \

from seqio.