This repository is dedicated to the 2023 ADL final project (Group 7), collaboratively developed by Peng-Ting Kuo (R10322029) and Yu-Hung Sun (R10343017).
The Graph folder contains all of the graphs we used in the report and presentation.
run.sh and plot.sh is for one-step replication.

Path/to/Your/Dataset

!bash run.sh data/test.json data/test_answer.json

Please note that our model was trained on Google Colab, which may lead to slight variations when run on local machines.

Use !python3 train.py for training

!python3 train.py \
    --model_name_or_path ValkyriaLenneth/longformer_zh \
    --tokenizer_name_or_path tokenizer_name_or_path \
    --train_file data/train.json \
    --validation_file data/validation.json \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --max_length 512 \
    --learning_rate 2e-4 \
    --num_label 4 \
    --gradient_accumulation_steps 4 \
    --num_train_epochs 10 \
    --output_dir Your/Model/Path

Upon completion of the training, the best model will be saved to a specified path in the given Google Drive path.

Use !python3 predict.py in jupyter notebook or bash run.sh for generating prediction result.

!python3 predict.py \
    --predict_file data/test.json \
    --reference_file data/test_answer.json

For visualizing data distribution and training logs used in our report and presentations, use !python3 plot.py in a Jupyter Notebook or employ the provided bash script.

!bash plot.sh Your/Data/Path Your/Record/Path Output/Path

"We experimented with various pre-trained models and large language models (LLMs) during the training process. The following are the outcomes of these experiments:

To gain a detailed understanding of our methodology, we invite you to explore our Jupyter notebook files. These notebooks provide a comprehensive guide to the steps and processes we followed:

1. Longformer Pre-Trained Model Experiment: For insights into our work with the Longformer pre-trained model, refer to the notebook ADL_Final_Project_Longformer.ipynb, authored by Peng-Ting Kuo. This notebook offers an in-depth look at our experiment and findings.
2. Taiwan-LLAMA Related Experiment: If you are keen on learning about our experiment related to Taiwan-LLAMA, please view the notebook 4class_TWLLM.ipynb, crafted by Yu-Hung Sun. It provides a thorough exploration of our approach and results in this report.

We have hosted a demo interface on HuggingFace. You can experiment with this model at HuggingFace Spaces.
To launch this interface, run app.py.