The accompanying code to produce the results in . Compares various LLMs in their ability to do zero- and few-shot text classification.

• Installation
• Usage
• Results

After cloning this repo and installing Poetry, run poetry install, then, poe accelerate (because poetry struggles with torch)

To download a model run: python utils/download.py hugging-face-model

Reproduce the experiment results by running the two bash files with bash pretrain_comparison.sh and bash two_stage. This will produce a consolidated file of the perforamnce metrics by run in data/results.csv.

To do an individual pass of data through an LLM, use the following command:

run -f path/to/data -t task -m hugging-face-model

Optionally, you can pass the flags:

• examples / -e: Whether to use zero or few shot learning.
• new_tokens \ -n: The number of tokens for the model to generate.

New tasks can be added using the format in utils/prompt.py. Model config is housed in utils/model.py and can be customized there.

To train a self-supervised model on many examples, run:

train -f path/to/data -t task

Optionally, you can pass the flags:

• -confident / -c: Enables Confident Learning which filters out likely mislabeled data

evaluate

All models run on a single Nvidia A100