bohdanpetryshyn / code-llama-fim-fine-tuning Goto Github PK

I try to use your hyperparameters to verify your paper, bug the result is differernt from yours. I use these hyperparameters:

--model_name_or_path "..."
--dataset_file_name "..."
--splits "train"
--max_seq_len 4096
--max_steps 1000
--save_steps 100
--eval_steps 100
--logging_steps 1
--log_level "info"
--logging_strategy "steps"
--evaluation_strategy "steps"
--save_strategy "steps"
--push_to_hub False
--bf16 False
--fp16 True
--learning_rate 2e-4
--lr_scheduler_type "cosine"
--weight_decay 0.1
--warmup_ratio 0.05
--max_grad_norm 1.0
--output_dir "fine-tune-19-54"
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 8
--gradient_checkpointing True
--use_reentrant True
--dataset_text_field "content"
--test_size 0.1
--fim_rate 0.9
--fim_spm_rate 0.5
--use_peft_lora True
--lora_r 32
--lora_alpha 64
--lora_dropout 0.1
--lora_target_modules "all-linear"
--use_4bit_quantization True
--use_nested_quant True
--bnb_4bit_compute_dtype "bfloat16"
--use_flash_attn False

If you don't mind, can you share the hyperparameters used to train the original Code-Llama?

   self.eot_token_id = tokenizer.encode(
        tokenizer.eot_token, add_special_tokens=False
    )[0]

File "/home/hsai/works/code-llama-fim-fine-tuning/dataset.py", line 45, in init
tokenizer.eot_token, add_special_tokens=False
AttributeError: 'LlamaTokenizerFast' object has no attribute 'eot_token'. Did you mean: 'eos_token'?