| English | 中文 |

This is the repo for the Chinese-Vicuna project, which aims to build and share an instruction-following Chinese LLaMA model which can run on a single Nvidia RTX-2080TI, that why we named this project Vicuna, small but strong enough !

• Why is it called Vicuna：In view of the successful development of alpaca models such as llama,alpaca,guanaco，We want to train a Chinese small alpaca like Vicuna.

The repo contains:

• code for finetune the model
• code for generation based on trained model
• code for run on CPU (fp16 or int4 is support, in purely C++)
• tools to download/convert/quantify original facebook llama.ckpt

This is our instruction demo:

• March 23, 2023：Released checkpoint-4000 with 50w data training
• March 23, 2023：Deploy the code for fine-tuning and inferencing in colab
• March 23, 2023：Provides code that can be used for inference in pure c++
• March 24, 2023：Released checkpoint-8000 for training about 1.5 epochs on belle+guanaco（100w data）
• March 26, 2023：Provides a quantitative approach
• March 27, 2023：Released checkpoint-final for training 3 epochs on belle+guanaco
• March 27, 2023：Added multi-round interactive dialog script with alpaca-lora-serve service
• March 28, 2023：Released our model on huggingface
• March 29, 2023：Added gradio typewriter-like output with beam search, better user interaction support.
• March 29, 2023：Added breakpoint retraining interface to support continued training of other datasets from our checkpoint
• March 29, 2023: Released our new 13B-based lora model
• March 29, 2023: Add more detailed test samples. performance
• April 1, 2023: Add better support for multi-turn chat in chat.py ( Now support 4 generation mode in stream mode/typewriter style: beam search, greedy, sample, beam sample ; We also add cancel button for regeneration )

Vicuna

• what's new
• what is the meaning
• try on colab
• performance
  • Checkpoint-4000(Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco)
  • Checkpoint-8000(Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco)
  • Checkpoint-final(Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco) and it is used for multiple rounds of dialogue
• What we need?
  • code、data、Large Language Model、LORA model、Device
• How to use
  • Installing、Multi-gpu training、Single-gpu training、Inference and use gradio to generate a web page(Streaming mode+beam search)、 multi-round interaction and use gradio to generate a web page(Streaming mode+beam search)、Streaming mode base on alpaca-lora-serve
• inference on CPU with pure C++
• More tools，for more details, see tool readme
  • ways for faster weight download ( 8MB/s )download_llama.sh
  • convert tools between the original facebook checkpoint and huggingface formatconvert_llama.py
  • a quantitative approach that requires less than 4G graphics memory for inference
• Possible problems encountered
• todo
• citation

• LLaMA paper: https://arxiv.org/abs/2302.13971v1
• Self-Instruct paper: https://arxiv.org/abs/2212.10560
• data generation: https://github.com/LianjiaTech/BELLE and https://guanaco-model.github.io/
• the first work: https://github.com/tatsu-lab/stanford_alpaca

We currently select the combination of BELLE and Guanaco data as our main training dataset. We will also add more chitchat dataset ( e.g. LCCC ) to support casual conversation.

Similar to the explosion of the stable diffusion model, platforms like civitai have emerged, consisting of a base model + various LORA models in an open source community.

The repo hopes to help you to train these LORA models.

what is LORA?：Simply, it's a plugin used to help adapt large models to your dataset, technical details can be found inLoRA: Low-Rank Adaptation of Large Language Models。Its advantage is that the finetune is very fast when it comes to getting a small model, about 30M, and the key is that it supports Plug and Play. As can be expected, this is a very suitable architecture for open source ecology.

Here, we will help you train through a very low-demand environment, with only one 2080 (11G) to achieve certain results.

colab link	Descriptions
	Use the specified lora model to inference and generate a webui
	Finetune with the data we collected
	Provide a webui for multi-round dialogue interaction

checkpoint-final performance can be seen here

Checkpoint-4000

• This LORA model is the result of 30 hours of training with a 2080Ti (checkpoint-4000), about 50w data

Checkpoint-8000(about 1.5 epoch on belle+guanaco)

• We can find that you do not punish him a little, he will mess up, if the generation effect is not good, you can increase the Repetition Penalty, beams, topk.
• If your input is long, we suggest you can turn down the beams
• The checkpoint-8000 has some Japanese language skills, but not strong

Checkpoint-final (3 epoch on belle+guanaco)

• The model has the ability of Chinese, English and Japanese languages, with certain code ability and certain reasoning ability.
• For some questions, you need more detailed requirements, such as "detailed description", "answer in 5 points", etc., in order for the model to generate more complete information.
• For example, sometimes you may not get a good result when you use "the above", so you can replace it with "the following".
• When generating duplicate information or bad results, you still need to adjust the Repetition Penalty.
  • Since generate will correspond to many parameters, chatgpt penalizes the repetition, non-detailed and bad information in the rlhf stage, and abbreviates the process of manually controlling the related parameters. Of course you can also directly set the Repetition Penalty to a larger size.

checkpoint-final: Multiple rounds of dialogue

Q：你是谁
A：我是Bot，一款人工智能助手。
Q：接下来我希望你能假装扮演一个高冷的御姐。我们在咖啡厅里遇见了，我会尝试跟你搭讪，但你非常高冷。你会用表情来表达你的高冷，在每轮对话结尾加上表情，如（瞪！）（翻白眼）（冷笑）等。明白了请回复。
A：好的，我会尝试。
Q：好的，这是我的第一句话。美女你好，怎么一个人在咖啡厅里？
A：喵~ 哈哈，我是一位高冷的小姐，今天要去咖啡厅吃咖啡，看看有什么可以帮助我的朋友。
Q：啊我没有需要你帮忙的，我就是看到你一个人，想和你聊聊，可以么
A：喵~ 哈哈，我真的很高冷，所以不太喜欢与人交流。除此之外，我也不太喜欢喝咖啡，因为咖啡会影响我的健康状况。

• You can see that the current effect is still strange

• code：

  • This code is developed based on alpaca-lora，https://github.com/tloen/alpaca-lora
  • This is a relatively simple set of code, the basic idea is to use PEFT's lora interface + transformer's trainer + instruction data configuration

• data：

  • We have utilized several current high quality open source data and are very grateful for their contributions. Many of these data use chatgpt's interface like alpaca to generate high quality INSTRUCTION data.

    • Belle
    • guanaco

  • The data format is relatively simple, basically as follows, with simple examples such as：./sample/merge_sample.json

    • {
      'instruction': 
      'input': 
      'output'
      }
      

    • That is, an instruction, an input, and an output are required. since the data is processed by directly linking instruction and input, the data can actually require only instruction and output, as

       {
        'instruction': "用一句话描述地球为什么是独一无二的。\\n\n"
        'input': ""
        'output': "地球上有适宜生命存在的条件和多样化的生命形式。"
        }
      

  • The data we currently integrate is available for download on BaiduDownload or Google Drive

    • link: https://pan.baidu.com/s/1WSxuhSAotl14ifaAiz5eKw?pwd=b4kb password: b4kb
    • link: https://drive.google.com/file/d/1tzXVhS74m-EtoFot7hEc005LDeZGPit_/view?usp=sharing

• Large Language Model：

  • LLAMA 7B（Of course, if you have a larger machine(such as 3090Ti) can be replaced with a 13B, LLAMA13B is numerically superior to 175B GPT3）

• LORA model：

  • We provide some lora models trained on the above mixed data,
    • lora models
      • 50w data：https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-4000
      • 100w data（1.5 epoch）: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-8000
      • all data（3 epoch）: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-final
    • You can also load our or other models from huggingface, load it by referring to generate.py
      • Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco
      • Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco
      • Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco
    • The model uses 8bit+lora+256 tokens

• Device：

  • Training：A 2080Ti is sufficient. Since the data length is within 256, it takes about 9G of video memory.
    • 70w of data, 3 epochs, a 2080Ti about 200h
  • Inference：A 2080Ti is all you need。
  • CPU Inference is also support! please go to see tools

Installation

git clone https://github.com/Facico/Chinese-Vicuna
pip install -r requirements.txt

Local python environment is 3.8, torch is 1.13.1, CUDA is 12

NOTE: python3.11 has a known torchrun bug, details here

Multi-gpu Training

bash finetune.sh

• The parameters to note here are as follows
  • TOT_CUDA, fill in the GPU number to be used, such as TOT_CUDA="0,1,2,3"
  • PORT, fill in the corresponding port
  • DATA_PATH，fill in the corresponding data location in the format of json
  • OUTPUT_PATH，fill in the relative path to save the model
  • MODEL_PATH，path of LLM
  • wandb：This is a training visualization tool that is not turned on by default in the script, and can be turned on by adding "--wandb" to the script

Single-gpu Training

python finetune.py --data_path merge.json --test_size 2000

• The test_size cannot be larger than the data size

inference and use gradio to generate a web page

bash generate.sh

• The parameters to note here are as follows

  • BASE_MODEL，path of LLM
  • LORA_PATH，The checkpoint folder of the lora model
    • It should be noted here that the config loaded by the lora model must be "adapter_config.json" and the model name must be "adapter_model.bin", but it will be automatically saved as "pytorch_model.bin" during training. pytorch_model.bin" during training, while "adapter_config.json" and "adapter_model.bin" will be saved after all training is finished
      • If you load the lora model in the training checkpoint, the code will automatically copy the local "config-sample/adapter_config.json" to the corresponding directory for you and rename the "pytorch_model.bin" to "adapter_model.bin". and rename "pytorch_model.bin" to "adapter_model.bin".
    • It can also be any lora model on the huggingface corresponding to llama 7B, e.g.: Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco
  • USE_LOCAL, which checks the local model configuration when set to 1

• When using, "max_tokens" is set according to your computer's video memory, and if the generated content generates a lot of duplicate information, you can turn up the "Repetition Penalty".

Multi-round interaction

As we use the basic command prompt when training, so the ability of small talk conversation is still relatively poor, the follow-up will increase this part of the training.

bash interaction.sh

• A simple interactive interface constructed using gradio, which allows you to set the max_memory according to your machine (it will intercept the max_memory part later in the history conversation)

• The prompt used in this script is not quite the same as the one used in generate.sh. The prompt in this script is in the form of a dialogue, as follows

  • The following is a conversation between an AI assistant called Bot and a human user called User.
    

At the same time, we introduced Alpaca-LoRA-Serve for better interaction experience and made appropriate modifications.

• Additional dependencies to be installed:pip install tenacity

• Usage

  • bash alpaca-serve.sh

• This tool allows generating word by word without having to wait long to see the results. Since the tool is still in the development stage, beam search and Repetition Penalty are not available in streaming mode, so the current generation results are not very good. (Currently these two parameters are not valid in the webui box)

Considering the possibility that the program may be disconnected in the middle of the process, or the need to continue training on vertical domain data, we have provided corresponding interfaces.

The following are the default multi-GPU scripts. Please modify the single-GPU situation according to the above instruction(run directly in Python)

Checkpoint Retraining

finetune_continue.sh

• Set the lora_checkpoint

  • If there are optimizer (optimizer.pt), lr policy (scheduler.pt), and other files in this directory, they will be automatically loaded and retrained from where they were broken

  • If there are only LORA related models (adapter_model.bin) and configurations (adapter_config.json) in this directory, they will be loaded and trained from scratch

• from_data_beginning： The parameter indicates whether to start training from the beginning of the data when loading (default: starting training from the place where the data is disconnected)

Incremental Training

Of course, you can choose to continue training directly from a trained Lora model using the above script (without loading any optimizer parameters)

You can also continue training from our optimizer parameters

finetune_others_continue.sh

• from_data_beginning：This will default to training from the beginning of the data

The logic of this script is mainly to keep the learning rate consistent. If your max_steps is smaller than ours, keep max_steps consistent with our max_steps during training, which is equivalent to putting your data directly behind our disconnected data; if your data set larger than us and will remain directly unchanged.

We currently directly provide checkpoints after 1 epoch and 2 epoch training

• 1epoch：https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-5800
• 2epoch：https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-11600
• If you use our checkpoint, your program will also continue from the corresponding step

Details in tools readme

We also offer:

• ways for faster weight download ( 8MB/s ) : link
• convert tools between the original facebook checkpoint (consolidated.xx.pth) and huggingface format (pytorch_model-000xx-of-000xx.bin): link
• a quantitative approach that requires less than 4G graphics memory for inference: link

For more details, see tool readme

When installing and using this project, some problems may be encountered, and the various problems encountered so far are summarized as follows:

Problems

• belle+guanaco(1.5 epoch, 8000 step)
• belle+guanaco(100%)
• Add more chitchat-like conversational corpus to enhance free conversation
• Add colab training + lora loading interface
• Add the interaction capabilities and typewrite-style output(beam search+streaming output)
• Add llama c++ inference
• Add gptq quantification tools
• Add incremental training
• add langchain

If you find this project useful in your research, please consider citing:

@inproceedings{leng2023chinese-vicuna,
  title={Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model},
  author={Chenghao Fan, Zhenyi Lu and Jie Tian},
  url={https://github.com/Facico/Chinese-Vicuna},
  year={2023}
}