joshvarty / googleqalabeling Goto Github PK

From: https://www.kaggle.com/c/google-quest-challenge/discussion/118612

https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification
https://www.kaggle.com/c/gendered-pronoun-resolution
https://www.kaggle.com/c/tensorflow2-question-answering

Jigsaw looks more similar to this , here are some top solutions from Jigsaw

1st Place solution:
https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280#latest-619135

4th place solution :
https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/101927#latest-628546

8th place solution :
https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/100961#latest-593873

From: https://www.kaggle.com/c/google-quest-challenge/discussion/118736

Here are the list of few amazing kernels from past NLP competitions that will help you learn about pre-processing and text-cleaning.

How to: Preprocessing when using embeddings
Improve your Score with some Text Preprocessing
Improve your Score with Text Preprocessing -V2
How To: Preprocessing for GloVe Part1: EDA

My understanding is that RAdam is probably better for NLP stuff. Let's give it a try here.

This is done while setting up our optimizer:

            no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
            optimizer_grouped_parameters = [
                {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.8},
                {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
            ]

What is the reason for this?

We should take an existing language model and train it on all StackExchange data before using it in our task.

It's available here: https://archive.org/download/stackexchange

We have about 48,000 unique words from the StackExchange Q&A sites. We're creating embeddings from Crawl300 which has over 2 million words with their corresponding embeddings.

Somehow we're getting over 25% of words as UNKNOWN.

This seems super high, we should investigate which words are unknown and see if there is an issue we need to fix.

Implementation:

class SpatialDropout(nn.Dropout2d):
    def forward(self, x):
        x = x.unsqueeze(2)  # (N, T, 1, K)
        x = x.permute(0, 3, 2, 1)  # (N, K, 1, T)
        x = super(SpatialDropout, self).forward(x)  # (N, K, 1, T), some features are masked
        x = x.permute(0, 3, 2, 1)  # (N, T, 1, K)
        x = x.squeeze(2)  # (N, T, K

I think we're supposed to randomly dropout entire channels NOT entire words. I think this implementation ends up dropping entire words.

We should check both of these statements ^

How often can a full question fit into 512 tokens? Any benefit to increasing the size?

By default our current approach uses sequences of length 512 that are fed to the BERT transformer. Is this a good length? How many sequences are less than this, how many are more?

Our EDA tried to explore this, but we weren't tokenizing the same way.

Test Set: 29.83% of sentences are 512 or more tokens
Train Set: 26.93% of sentences are 512 or more tokens

There's a library called nlpaug we might want to use.

See: https://www.kaggle.com/c/google-quest-challenge/discussion/123269

Are there other similar Kaggle competitions?

From: https://forums.fast.ai/t/mixup-data-augmentation/22764/4

Have you tried mixup with anything else but classification?

Yes, I’ve tried it in NLP, mixing the outputs of the embedding layers and it has given good results. Hoping to have time to experiment with this more and write a paper about it when the development of fastai_v1 slows down a bit 😉 I think it would also be helpful in tabular data (again mixing the embeddings fro categorical variables), not sure about object detection since I don’t see how you mixup the targets (which is critical in making mixup work properly).

5 Fold BERT Model 0.298820, 0.298820, 0.293037, 0.314699, 0.308084 LB: 0.312

What do the intermediate shapes look like? Would it make sense to stack multiple linear layers at the output like we do in some vision tasks?