In this assignment, you are asked to complete the code for finetuning a distilled BERT model for the named entity recognition (NER) task on a small subset of CoNLL 2003, a famous and widely-used dataset for NER. The training framework and auxiliary functions are already provided for you. What you need to do is complete the code within

# --- TODO: start of your code ---

# --- TODO: end of your code ---

so that the model can run properly. Changing the code outside the TODO block is not necessary.

The functions to be completed are

• src.dataset.dataset.Dataset.encode: for data pro-processing. It encodes the input tokens into the BERT token format
• src.dataset.collate.DataCollator.__call__: for batch processing. It collates the instances in each sampled batch so that they meet the BERT model input format.
• src.train.Trainer.training_step: for model finetuning. The part to be completed concerns the weight updating and loss tracking.
• src.train.Trainer.get_loss: for loss calculation.
• src.train.Trainer.evaluate: for model evaluation. The part to be completed concerns model inference.

For detailed description and requirment, please check the annotation before each TODO block.

The code is built on Python 3.10 and Hugging Face Transformers Library with customized data processor and trainer. Other package requirements are listed in requirements.txt. You are suggested to run the code in an isolated virtual conda environment. Suppose you have already install conda in your device, you can create a new environment and activate it with

conda create -n 310 python=3.10
conda activate 310

Then, you can install the required packages with

pip install -r requirements.txt

Alternatively, you can also use other Python version manager or virtual environments such as pyenv or docker to you prefer.

If you are using a Unix-like system such as Linux or MacOS, you can run the code through the provided run.sh file. You first need to edit run.sh to complete your name and GTID, and then run

./run.sh [GPU ID]

for example,

./run.sh 0

if you want to GPU-0 to accelerate your training. If you leave GPU ID blank, the model will be trained on CPU.

For MacOS running on M* chips, running bash ./run.sh will automatically take advantage of mps accelaration. You can disable this behavior by adding --no_mps argument into the Python call in the sh file. This feature is deprecated as mps sometimes returns incorrect results.

Alternately, you can also run the code with

[CUDA_VISIBLE_DEVICES=...] python run.py --name <your name> --gtid <your GTID> [other arguments...]

# for example, python run.py --name "George Burdell" --gtid 123456789 --lr 1e5 --batch_size 4096 --n_epochs 4096

If your code runs successfully, you will see a record.log file in your log folder suppose you keep the --log_path argument as default. The log file should track your training status, reporting the training loss, and the validation performance for each training epoch, and the final test performance.

For this assignment, you should submit a ner.<GivenName>.<FamilyName>.<GTID>.zip (e.g. ner.George.Burdell.123456789.zip) file containing ./src/ and ./log/ folders and all their contents. Do not include ./data/, .gitignore, LICENSE or other files or folders. If your using Unix-like systems, you can run

zip -r ner.<GivenName>.<FamilyName>.<GTID>.zip log/ src/

The final zip file should not be larger than 50KB.

With default hyper-parameters, each training epoch takes roughly 30s to run on Mac M1 CPU, 18s on mps, and 2s on Nvidia A5000/RTX4090 GPU.