AllenNLP biased towards BERT about allennlp HOT 12 OPEN

Ok, thanks!
I'll try testing something like this and will report back.

from allennlp.

Hey @pvcastro, a couple questions:

1. In all experiments (BERT-AllenNLP, RoBERTa-AllenNLP, BERT-transformers, RoBERTa-transformers) were you using the same optimizer?
2. When you used transformers directly (for BERT-transformers and RoBERTa-transformers) was that a CRF model as well, or was that just using the (Ro|B)ertaForSequenceClassification models?

from allennlp.

Hi @epwalsh , thanks for the feedback!

1. Yes, I was using the huggingface_adamw optimizer.
2. No, it wasn't an adaptation with CRF, I used the straight run_ner script script from the hf's examples. But I believe the CRF layer would only improve results, as they usually do with bert models.

from allennlp.

Gotcha! Oh yes, I meant BertForTokenClassification, not BertForSequenceClassification 🤦

So I think the most likely source for a bug would be in the PretrainedTransformerMismatched(Embedder|TokenIndexer). And any differences between BERT and RoBERTa would probably have to do with tokenization. See, for example:

from allennlp.

I was assuming that just running some unit tests from the AllenNLP repository, to confirm that these embedders/tokenizers are producing tokens with the same special tokens as RoBERTa architecture would be enough to discard these. I ran some tests using RoBERTa and confirmed that it's not relying on CLS. Was this too superficial to reach any conclusions?

from allennlp.

I'm not sure. I mean, I thought we did have pretty good test coverage there, but I know for a fact that's one of the most brittle pieces of code in the whole library. It would break all of the time with new releases of transformers. So that's my best guess.

from allennlp.

Do you think it makes sense for me to run additional tests for the embedder comparing embeddings produced by a raw RobertaModel and the actual PretrainedTransformerMismatchedEmbedder? To try to see if they are somehow getting "corrupted" in the framework.

from allennlp.

I guess I would start by looking very closely at the exact tokens that are being used for each word by the PretrainedTransformerMismatchedEmbedder. Maybe pick out a couple test instances to check where the performance gap between the BERT and RoBERTa predictions is largest.

from allennlp.

This issue is being closed due to lack of activity. If you think it still needs to be addressed, please comment on this thread 👇

from allennlp.

This issue is being closed due to lack of activity. If you think it still needs to be addressed, please comment on this thread 👇

from allennlp.

Sorry, I'll try to get back to this next week, haven't had the time yet 😞

from allennlp.

No rush, I thought adding the "question" label would stop @github-actions bot from closing this, but I guess not.

from allennlp.