This repository contains the evaluation code for Sailor, a suite of open language models for South-East Asia.

Sailor is developed by the Sea AI Lab and Singapore University of Technology and Design.

Sailor is a suite of Open Language Models tailored for South-East Asia (SEA), focusing on languages such as 🇮🇩Indonesian, 🇹🇭Thai, 🇻🇳Vietnamese, 🇲🇾Malay, and 🇱🇦Lao. Developed with careful data curation, Sailor models are designed to understand and generate text across diverse linguistic landscapes of SEA region. Built from Qwen 1.5, Sailor encompasses models of varying sizes, spanning from 0.5B to 7B versions for different requirements. Benchmarking results demonstrate Sailor's proficiency in tasks such as question answering, commonsense reasoning, reading comprehension and etc. in SEA languages.

• Continually pretrained on 200 Billion to 400 Billion tokens over 7 languages, including Indonesian, Thai, Vietnamese, Malay, Lao, English and Chinese.
• Various model sizes (0.5B, 1.8B, 4B and 7B) to support different requirements.
• Strong performance on SEA benchmarks such as XQuAD, TydiQA, XCOPA, Belebele and M3Exam.
• No restrict on the research and the commercial use, but should comply with the Qwen 1.5 license.

To learn more details, please access the technical report.

You can find all the Sailor models in our Huggingface home page here:

• Sailor-0.5B
• Sailor-1.8B
• Sailor-4B
• Sailor-7B
• Sailor-0.5B-Chat
• Sailor-1.8B-Chat
• Sailor-4B-Chat
• Sailor-7B-Chat

Here are the results of the evaluation of the models on question answering tasks. The evaluation results are presented in the form of tables, where the first column is the model name, and the reset columns are the performance on Thai (th), Indonesian (id), and Vietnamese (vi) languages, respectively. The results of Sailor models are highlighted in bold. You can find the full evaluation results on the different tasks and our evaluation code to reproduce the results in the eval directory.

We use OpenCompass to evaluate the models. To install the required packages, run the following command under this folder:

# setup opencompass environment
conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
pip install pythainlp langid
mkdir data

To build the evaluation script, run the following command under this folder:

cp -r eval/configs/* opencompass/configs/
cp -r eval/data/* opencompass/data/
cp -r eval/datasets/* opencompass/opencompass/datasets/
cp eval/icl_sailor_evaluator.py opencompass/opencompass/openicl/icl_evaluator/
cp eval/sailor_text_postprocessors.py opencompass/opencompass/utils/
echo "from .icl_sailor_evaluator import AnsEvaluator, TextGenEvaluator  # noqa" >> "opencompass/opencompass/openicl/icl_evaluator/__init__.py"
echo "from .sailor_text_postprocessors import *  # noqa" >> "opencompass/opencompass/utils/__init__.py"
echo "from .xquad import *  # noqa: F401, F403" >> "opencompass/opencompass/datasets/__init__.py"
echo "from .tydiqa_id import *  # noqa: F401, F403" >> "opencompass/opencompass/datasets/__init__.py"
echo "from .xcopa_sea import *  # noqa: F401, F403" >> "opencompass/opencompass/datasets/__init__.py"
echo "from .m3exam import *  # noqa: F401, F403" >> "opencompass/opencompass/datasets/__init__.py"
echo "from .belebele import *  # noqa: F401, F403" >> "opencompass/opencompass/datasets/__init__.py"
cp eval/eval_sailor.py opencompass/configs/

To run the evaluation, run the following command under this folder:

cd opencompass
python run.py configs/eval_sailor.py -w outputs/sailor --num-gpus 1 --max-num-workers 64

You can also modify the script to evaluate other models like Qwen1.5, Llama, Mistral, etc.

We provide a simple demo to chat with Sailor-7B-Chat . You can also develop it using the provided demo code.

If you use this repository or sailor models, please cite

@misc{dou2024sailor,
      title={Sailor: Open Language Models for South-East Asia}, 
      author={Longxu Dou and Qian Liu and Guangtao Zeng and Jia Guo and Jiahui Zhou and Wei Lu and Min Lin},
      year={2024},
      eprint={2404.03608},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

If you have any questions, please raise an issue in our Github or contact us at [email protected] and [email protected].