bert-for-tf2 for Sentiment Analysis about bert-for-tf2 HOT 4 CLOSED

@muhammadfahid51 - have you took a look at the examples, i.e. https://github.com/kpe/bert-for-tf2/blob/master/examples/tpu_movie_reviews.ipynb

I'm not sure whether doing something fancy on top of BERT when fine-tuning brings much (compared to just classifying on the [CLS] activations). I believe it is rather important to try not over-fitting and find a good learning rate schedule.

from bert-for-tf2.

@muhammadfahid51 - what do you mean by "model is training" - does the loss drop over time? What hyper parameters do you use? What is your kernel size in the CNN and how many CNN layers do you use? And how large is your dataset?
I don't see why your model above should not work if using proper hyper-parameters.

from bert-for-tf2.

@kpe Hey,
Training loss does not drop over time and neither the validation loss. About parameters I have changed them alot. Like tried different number of layers, filters, regularization parameters etc but to no avail.
But I think the problem was something else. My data is in Urdu Language and it has got 10k samples of 3 classes with 50(Positive), 30(Negative) and 20(Neutral) percent distribution. I tried the same model on imdb data and it worked well.
But one tweak that remains is,
How does bert handles unknown words. Does it assign all of them to '[UNK]' and then learns them while training or is there some other technique ?

from bert-for-tf2.

@muhammadfahid51 - I believe the BERT models were pre-trained on English only. So until there is a multilingual pre-trained model you might have to pre-train from scratch yourself. To check how the tokenization works with your language, you could tokenize and convert the tokens to ids, and then back.
Currently bert-for-tf2 does not have an example/tools for pre-training from scratch. When pre-training from scratch it might be easier using the workflow from the original google-research/bert. Once you have the pre-trained weights, you could fine-tune with bert-for-tf2.

from bert-for-tf2.