Git Product home page Git Product logo

curriculum_learning's Introduction

On Curriculum Learning for Commonsense Reasoning

PyTorch code for the NAACL 2022 paper "On Curriculum Learning for Commonsense Reasoning"

Requirements:

This code has been tested on torch==1.9.0 and transformers==4.3.2. Other required packages are bayes_opt and tqdm.

Prepare Repository:

Download the datasets used in the paper from the following locations: SocialIQA CosmosQA CODAH ProtoQA WinoGrande GLUE

Save to the ./data/ folder. Our split of the HellaSWAG-2K dataset is available at the ./data/hellaswag/ folder.

Training RoBERTa Models for Results in Table 1 in paper:

The training is performed in two stages. Example commands are shown for the CosmosQA dataset. Scripts for other datasets are available in the repository.

  • Stage 1: In this stage, the teacher models are trained via finetuning of pretrained RoBERTa-Large models using the standard method of randomly sampled training mini-batches. This also serves as the baseline RoBERTa model in our experiments. We provide scripts to find the best hyperparameters for the teacher model using bayesian optimization. To find the best RoBERTa model for CosmosQA, run\
cd roberta
python grid_search_hyperparams_roberta_cosmosqa.py

The default location for the model checkpoints is ./baselines/. For SocialIQA, CODAH and WinoGrande, we use the best hyperparameters reported in existing literature.

  • Ranking of training samples: In this preprocessing step before Stage 2, the predictions from teacher models are used to rank the training samples in the order of difficulty. To get predictions for CosmosQA, run
bash cosmosqa.sh eval_valid
python process_logits.py
  • Stage 3: In this stage, the student models are trained via curriculum learning using the ranked dataset from the previous step. We use bayesian optimization to find the best parameters for the pacing function; scripts for performing the optimization are available for all datasets in the repository. To find the best pacing function for CosmosQA, run:
python search_pacing_function_params_cosmosqa.py

Citation:

@inproceedings{maharana2022oncurriculum,
  title={On Curriculum Learning for Commonsense Reasoning},
  author={Maharana, Adyasha and Bansal, Mohit},
  booktitle={NAACL},
  year={2022}
}

curriculum_learning's People

Contributors

adymaharana avatar ajdesh2000 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

ajdesh2000 1171-jpg occasionallynlp

curriculum_learning's Issues

Problems replicating

Hey,

I am trying to reproduce your results on the CODAH-dataset and there are some open ends in the code.

The following files are missing / never get generated:

  1. 'train_ranked_by_qap.tsv',. There is the train_with_scores.csv, but those examples are not ordered, which is necessary for the curriculum to work, if I am not mistaken?
  2. checkpoint-best_logits.txt and checkpoint-best_train_logits.txt. Do we manually have to select and rename them?

Other things that cause errors:

  1. Building the dataset the code expects 'segment-ids' to be in the features, but they are not (and they are also not necessary for roberta)

Other clarifications:

  1. The convert_examples_to_features function has a curriculum argument which adds qap-scores to the features, but the argument is not used in any of the scripts
  2. checkpoint-100_xx; checkpoint-200_xx aso.. correspond to to different stages of the first epoch and checkpoint-epoch-1_xx aso to everything that comes afterwards?
  3. For the curriculum learning condition, only the final performances are saved in codah_qap_cl.jsonl for every hyperparameter-set from the bayesian optimisation process? The corresponding baseline performance would be in ./baselines/codah-roberta-large/fold_4/is_test_false_eval_results.txt.?

Thanks a lot for your help!

AttributeError for gridsearch-script

Hey,

Using the recommended version of transformers (4.3.2), I get an AttributeError for gridsearch-script (roberta/grid_search_hyperparams_roberta_cosmosqa.py):

AttributeError: type object 'BertConfig' has no attribute 'pretrained_config_archive_map'

I think this should be resolvable by downgrading the transformers package (?)

Best,
Lucas

KeyError: 'segment_ids'

Hi, I'm getting KeyError: 'segment_ids' when running python grid_search_hyperparams_roberta_cosmosqa.py. Looks like some lines were commented out here:

curriculum_learning/roberta/utils_multiple_choice.py

Line 79 in e5edf7f

# {"input_ids": input_ids, "input_mask": input_mask, "segment_ids": segment_ids}

Could you help me with the changes that need to be made to get it to work again?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.