adapter-BERT: loader reports missing weights about bert-for-tf2 HOT 2 CLOSED

@tom-schoener - yes, that's expected - there are no adapter weights in the original pre-trained bert checkpoints (i.e. those from google-research/bert).
More concerning is however the "trainable params: 0" line in the summary. To fix this, please, put the

l_bert.apply_adapter_freeze()

only once the model has been build (i.e. after the model.build(), which would instantiate the properly sized weights).

And as a side note - usually adapter_size does not have to be too big. Depending on the task as small as 4 or 8 could be sufficient, and sometimes even without an adapter, freezing all of bert and tweaking only the layer_normalization layers could work surprisingly well.

from bert-for-tf2.

Thank you for the quick response. The problem with the number of trainable weights being 0 was just a copy-paste error for the code example.
I tried your suggested adapter_size of 4 and it works quite well for my task. Also, freezing all layers and only tweaking the normalization layers sounds interesting. I am going to try that as well.

I really like your BERT implementation for TF Keras - keep up the good work!

from bert-for-tf2.