Custom tokenizer layer about bert-for-tf2 HOT 5 CLOSED

hope that helps:

pip install tensorflow-text

and then try something along those lines:

import tensorflow_text as text

tokenizer = text.BertTokenizer(os.path.join(ckpt_dir, 'vocab.txt'))
tok_ids = tokenizer.tokenize(["hello, cruel world!", "abcccccccd"]).merge_dims(-2,-1).to_tensor(shape=(2, max_seq_len))

from bert-for-tf2.

@ptamas88 Did you succeed to make it works ? I have the same question

from bert-for-tf2.

yes, usually the tokenizer is not part of the graph. For this you'll need a tokenizer that has a TF implementation, like sentencepiece when using albert. For BERT you might try the tf.text BertTokenizer (https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/BertTokenizer.md) - I haven't used it myself, but it should work.

from bert-for-tf2.

@ptamas88 Did you succeed to make it works ? I have the same question

haven't tried since, but i will check out the solution @kpe mentioned

from bert-for-tf2.

it didn't work， still throw OperatorNotAllowedInGraphError

from bert-for-tf2.