albert tokenization about bert-for-tf2 HOT 2 CLOSED

@mshlis - thank you for your message! Which albert code exactly are you referring to?
Please note, while the google's bert pre-trained models use a vocab_size of 30522 and a custom tokenizer, the albert pre-trained models from tfhub use a vocab_size of 30000 and sentencepice as tokenizer.

So, when using some of the pre-trained albert models, the model input should be prepared/tokenized differently.

It seems, I currently have no example of how tokenization should be best done for albert, so I'll have to add one! Thank you for noticing this!

from bert-for-tf2.

I have recently replaced the bert/tokenization.py with bert/tokenization/bert_tokenization.py and bert/tokenization/albert_tokenization.py and will soon add some sample code in the README.

from bert-for-tf2.