meelfy / pytorch_pretrained_bert Goto Github PK

When I call from_pretrained, the code runs into this error:

File "/home/gregory.werner/.local/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 197, in from_pretrained
tokenizer = cls(resolved_vocab_file, *inputs, **kwargs)
TypeError: init() got an unexpected keyword argument 'max_len'

for line 169, I faced RunTimeError 'pow not implemented for Half' training a model. For all pow functions are used under torch as torch.pow() except this one. Is there a probability to miss this one ?

Hi,
I use pytorch_pretrained_BERT/examples/python run_lm_finetuning.py to fit the model with monolingual set of sentences. I use bert multilingual cased model.

Once the model is fine-tuned, I get the loss for given sentences with the following code:

def get_score(sentence, model):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    model.eval()
    predictions=model(tensor_input)
    loss_fct = torch.nn.CrossEntropyLoss()
    loss = loss_fct(predictions.squeeze(),tensor_input.squeeze()).data 
    return math.exp(loss)

sentence = "ﺶﻋﺮﺴﺗﺎﻧ؛ ﺩ پښﺕﻭ ﺶﻋﺭپﻮﻬﻧې ﻥﻭی پړﺍﻭ - ﺕﺎﻧﺩ"
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
stats=torch.load('pytorch_model.bin')
bertMaskedLM = BertForMaskedLM.from_pretrained('bert-base-multilingual-cased', state_dict=stats)

print(get_score(sentence, bertMaskedLM))

78637.05198167797

bertMaskedLM_orig = BertForMaskedLM.from_pretrained('bert-base-multilingual-cased')
print(get_score(sentence, bertMaskedLM_orig))

7.919475431571431

The strange thing is that the fine-tuned model returns much higher loss scores, even if the evaluated sentence appeared in monolingual training data.

Is something I am doing wrong? I just want to check how well the given sentence fits into LM.

Regards and thanks in advance

hi ，nice work
When i use this package, BertModel.from_pretrained('bert-base-uncased'), it's always download when program restart, Can I just download it once？