A few questions about FreeLB and dropout about freelb HOT 2 CLOSED

Hey, thanks for trying out!

We did implement the modified dropout in the code for both fairseq and Huggingface Transformers. Most of the modification are on the modeling part. For an easier reference, see the lines involving the variable dp_mask in huggingface-transformers/src/transformers/modeling_albert.py.

For a comparison of the results with and without such a modification, please refer to Table 4 in the paper.

from freelb.

Wow. Thanks for your prompt reply! I get it.
I use the Vanilla transformers framework and feel confused about the "dp_mask" parameter. It does play a role in your revised ”Albert“ version, which is used to control the dropout mask used in the forwarding step. Thanks. Maybe We need a balance between performance and code modification (especially for Vanilla transformers framework, e.g. for other Bert-variant models, do the same modification as your "Albert") when using the same dropout suggestion.

from freelb.