Not Being able to reproduce Bert results about pytorch-widedeep HOT 5 CLOSED

Hey @TheLegendAli thanks for opening the issue, I will look into it asap 🙂

from pytorch-widedeep.

I think I know what might happen (bear in mind I looked in my phone and is 6:30am here in the London). I think you are passing pred_dim = 2 while this is a binary classification problem and has to be equal to 1, look here: https://github.com/jrzaurin/pytorch-widedeep/blob/master/pytorch_widedeep/models/wide_deep.py#L86

To be honest we should be better/more explicit regarding this parameter, since I see how it can cause confusion.

Anyway, let me know if this was the issue and thanks for opening it, I will add some warning in the next release.

from pytorch-widedeep.

Hi thanks for the feedback, that definitely did improve the results. However, there is still a wide difference in performance. I think I might be doing something else wrong. And I hope you had some coffee in the morning

from pytorch-widedeep.

okay then, I'll check

from pytorch-widedeep.

Hey @TheLegendAli , so, after a quick look my main comment would be that you need to make sure you are comparing the same thing, since at the moment this is not the case.

For example, in one case you are using bert-base-uncased with a certain tokenizer set up, while in another case you are using distilbert-base-uncased. This should not really matter much, but is worth pointing out.

Perhaps a more relevant aspect is the fact that in one case you are not freezing the model weights, resulting in you finetunning the model to the data. However, when using the library you are completely freezing the weights (by setting up freeze_bert=True).

Let's do something, since you are in the slack channel let's close the issue and move the conversation there :)

Thanks for opening it!

from pytorch-widedeep.