Train a small Electra from scratch and fine tune it with GLUE(SST-2) and with ImDb data. After getting the result, compare it with Bert.

• Train Electra from Scratch
• Fine Tune the trained Electra_small on GLUE tasks for sequence classification
• Fine Tune the pretrained Electra_base on imdb unsupervised dataset
• Fine Tune the post pretrained Electra on Imdb sentiment classification dataset
• Ablation Experiment of Bert

    $ git clone https://github.com/LiZongyue/Electra_small.git
    $ cd ~/Electra_small
    $ pip install .

Train_from_Scratch
Before running the pre-training example, you should get a file that contains text on which the language model will be trained. A good example of such text is the WikiText-103 dataset. Download WikiText.raw for the example.

fine-tune_GLUE
The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.

Accuracy of SST-2 classification : 96.87%

For more detail about the dataset and how to run the example, click here

fine-tune_pretrained_electra
This example could also be called as post pre-training Electra. The Large Movie Review Dataset is for fine-tuning the pretrained Electra-Base. In this task, 50k data samples for unsupervised learning will be used. The data directory is ~/aclImdb/train/unsup/ once the dataset is downloaded through this link.

fine-tune_Imdb
In this example, data supplier is as same as Fine Tune the pretrained Electra_base on imdb unsupervised dataset. The data directory is ~/aclImdb/train/pos/, ~/aclImdb/train/neg/, ~/aclImdb/test/pos/ and ~/aclImdb/test/neg/ once the dataset is downloaded through this link. Use Data Generator to generate the wohle dataset which could be used directly by the script

ablation-Bert
In this example, there are two scripts running on Bert. First uses the unsupvised dataset of ImDb to fine tune a pre-trained base Bert, second uses the sentiment classification dataset of ImDb to do the Ablation experiment to compare with Electra.