the added special tokens seem to convert to ids uncorrectly with my chinese datasets about pure HOT 6 CLOSED

Yes, our ALBERT experiments are based on add_new_tokens. Did you try just to use tokenizer.tokenize()?

from pure.

Thanks for your reply!
I used tokenizer.tokenize() to find that the added_tokens.json did not work, it still only use the origin vocab.txt. What I really want is to keep the added token, rather than tokenized with wordPiece. Thus, I wondered whether re-pretraining is necessary so that I can use both origin and added tokens to apply the convert_tokens_to_ids() function in your experiment?

Could you please tell me the detailed steps of ALBERT experiments kindly, since I find the vocab_size was modified from 30000 to 30046 in rel-alb-ctx0.zip.

from pure.

Hi, I just want to check back and see whether you have resolved the issue.
I don't think you need to re-pretrain the model (we didn't in our ALBERT experiments). You may check this function to see how we augment the vocabulary.

from pure.

Hi, I also had this problem after trying to train a model with Bert base. Could you please answer me the following questions:

1. In the paper you also use base Bert to get the results of typed text markers. However, if I understand you correctly, Albert was used?

2. To add to that, in the code the "add_new_tokens" parameter is set to false per default and only to true for Albert. Why is that? If I understand the code correctly, only if this is set to true, the model will use the typed markers with entity labels and not the unknown token. Shouldn't this be true per default?

3. Can I just set the add_new_tokens parameter to true when I want to train base Bert (or any other model than albert) with it?

from pure.

@hero-png Thanks for your questions!

1. We tried both BERT and ALBERT as the base model in our experiments with typed markers.
2. For BERT experiments, we directly used the unused tokens in BERT vocabulary ([unused1], [unused2]) as the markers (check this line). For ALBERT experiments, we had to use add_new_tokens because there aren't pre-defined unused tokens in ALBERT vocabulary.
3. Yes, you can set add_new_tokens to be true when training BERT.

from pure.

@a3616001 Thank you very much for your quick reply!

from pure.