CodeDoumentation Inference and Evaluation about codebert HOT 3 CLOSED

When running the script run.py in code2nl
and using the model dowloaded from the drive link

I am getting this error:

Missing key(s) in state_dict: "bias", "encoder.embeddings.word_embeddings.weight", "encoder.embeddings.position_embeddings.weight", "encoder.embeddings.token_type_embeddings.weight", "encoder.embeddings.LayerNorm.weight", "encoder.embeddings.LayerNorm.bias", "encoder.encoder.layer.0.attention.self.query.weight", "encoder.encoder.layer.0.attention.self.query.bias", "encoder.encoder.layer.0.attention.self.key.weight", "encoder.encoder.layer.0.attention.self.key.bias", "encoder.encoder.layer.0.attention.self.value.weight", "encoder.encoder.layer.0.attention.self.value.bias", "encoder.encoder.layer.0.attention.output.dense.weight", "encoder.encoder.layer.0.attention.output.dense.bias", "encoder.encoder.layer.0.attention.output.LayerNorm.weight", "encoder.encoder.layer.0.attention.output.LayerNorm.bias", "encoder.encoder.layer.0.intermediate.dense.weight", "encoder.encoder.layer.0.intermediate.dense.bias", "encoder.encoder.layer.0.output.dense.weight", "encoder.encoder.layer.0.output.dense.bias", "encoder.encoder.layer.0.output.LayerNorm.weight", "encoder.encoder.layer.0.output.LayerNorm.bias", "encoder.encoder.layer.1.attention.self.query.weight", "encoder.encoder.layer.1.attention.self.query.bias", "encoder.encoder.layer.1.attention.self.key.weight", "encoder.encoder.layer.1.attention.self.key.bias", "encoder.encoder.layer.1.attention.self.value.weight", "encoder.encoder.layer.1.attention.self.value.bias", "encoder.encoder.layer.1.attention.output.dense.weight", "encoder.encoder.layer.1.attention.output.dense.bias", "encoder.encoder.layer.1.attention.output.LayerNorm.weight", "encoder.encoder.layer.1.attention.output.LayerNorm.bias", "encoder.encoder.layer.1.intermediate.dense.weight", "encoder.encoder.layer.1.intermediate.dense.bias", "encoder.encoder.layer.1.output.dense.weight", "encoder.encoder.layer.1.output.dense.bias", "encoder.encoder.layer.1.output.LayerNorm.weight", "encoder.encoder.layer.1.output.LayerNorm.bias", "encoder.encoder.layer.2.attention.self.query.weight", "encoder.encoder.layer.2.attention.self.query.bias", "encoder.encoder.layer.2.attention.self.key.weight", "encoder.encoder.layer.2.attention.self.key.bias", "encoder.encoder.layer.2.attention.self.value.weight", "encoder.encoder.layer.2.attention.self.value.bias", "encoder.encoder.layer.2.attention.output.dense.weight", "encoder.encoder.layer.2.attention.output.dense.bias", "encoder.encoder.layer.2.attention.output.LayerNorm.weight", "encoder.encoder.layer.2.attention.output.LayerNorm.bias", "encoder.encoder.layer.2.intermediate.dense.weight", "encoder.encoder.layer.2.intermediate.dense.bias", "encoder.encoder.layer.2.output.dense.weight", "encoder.encoder.layer.2.output.dense.bias", "encoder.encoder.layer.2.output.LayerNorm.weight", "encoder.encoder.layer.2.output.LayerNorm.bias", "encoder.encoder.layer.3.attention.self.query.weight", "encoder.encoder.layer.3.attention.self.query.bias", "encoder.encoder.layer.3.attention.self.key.weight", "encoder.encoder.layer.3.attention.self.key.bias", "encoder.encoder.layer.3.attention.self.value.weight", "encoder.encoder.layer.3.attention.self.value.bias", "encoder.encoder.layer.3.attention.output.dense.weight", "encoder.encoder.layer.3.attention.output.dense.bias", "encoder.encoder.layer.3.attention.output.LayerNorm.weight", "encoder.encoder.layer.3.attention.output.LayerNorm.bias", "encoder.encoder.layer.3.intermediate.dense.weight", "encoder.encoder.layer.3.intermediate.dense.bias", "encoder.encoder.layer.3.output.dense.weight", "encoder.encoder.layer.3.output.dense.bias", "encoder.encoder.layer.3.output.LayerNorm.weight", "encoder.encoder.layer.3.output.LayerNorm.bias", "encoder.encoder.layer.4.attention.self.query.weight", "encoder.encoder.layer.4.attention.self.query.bias", "encoder.encoder.layer.4.attention.self.key.weight", "encoder.encoder.layer.4.attention.self.key.bias", "encoder.encoder.layer.4.attention.self.value.weight", "encoder.encoder.layer.4.attention.self.value.bias", "encoder.encoder.layer.4.attention.output.dense.weight", "encoder.encoder.layer.4.attention.output.dense.bias", "encoder.encoder.layer.4.attention.output.LayerNorm.weight", "encoder.encoder.layer.4.attention.output.LayerNorm.bias", "encoder.encoder.layer.4.intermediate.dense.weight", "encoder.encoder.layer.4.intermediate.dense.bias", "encoder.encoder.layer.4.output.dense.weight",

Besides, I suggest you go to our new repo about this task, which is more efficient and needs less GPU resource.
https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text

from codebert.

When running the script run.py in code2nl
and using the model dowloaded from the drive link

I am getting this error:

Missing key(s) in state_dict: "bias", "encoder.embeddings.word_embeddings.weight", "encoder.embeddings.position_embeddings.weight", "encoder.embeddings.token_type_embeddings.weight", "encoder.embeddings.LayerNorm.weight", "encoder.embeddings.LayerNorm.bias", "encoder.encoder.layer.0.attention.self.query.weight", "encoder.encoder.layer.0.attention.self.query.bias", "encoder.encoder.layer.0.attention.self.key.weight", "encoder.encoder.layer.0.attention.self.key.bias", "encoder.encoder.layer.0.attention.self.value.weight", "encoder.encoder.layer.0.attention.self.value.bias", "encoder.encoder.layer.0.attention.output.dense.weight", "encoder.encoder.layer.0.attention.output.dense.bias", "encoder.encoder.layer.0.attention.output.LayerNorm.weight", "encoder.encoder.layer.0.attention.output.LayerNorm.bias", "encoder.encoder.layer.0.intermediate.dense.weight", "encoder.encoder.layer.0.intermediate.dense.bias", "encoder.encoder.layer.0.output.dense.weight", "encoder.encoder.layer.0.output.dense.bias", "encoder.encoder.layer.0.output.LayerNorm.weight", "encoder.encoder.layer.0.output.LayerNorm.bias", "encoder.encoder.layer.1.attention.self.query.weight", "encoder.encoder.layer.1.attention.self.query.bias", "encoder.encoder.layer.1.attention.self.key.weight", "encoder.encoder.layer.1.attention.self.key.bias", "encoder.encoder.layer.1.attention.self.value.weight", "encoder.encoder.layer.1.attention.self.value.bias", "encoder.encoder.layer.1.attention.output.dense.weight", "encoder.encoder.layer.1.attention.output.dense.bias", "encoder.encoder.layer.1.attention.output.LayerNorm.weight", "encoder.encoder.layer.1.attention.output.LayerNorm.bias", "encoder.encoder.layer.1.intermediate.dense.weight", "encoder.encoder.layer.1.intermediate.dense.bias", "encoder.encoder.layer.1.output.dense.weight", "encoder.encoder.layer.1.output.dense.bias", "encoder.encoder.layer.1.output.LayerNorm.weight", "encoder.encoder.layer.1.output.LayerNorm.bias", "encoder.encoder.layer.2.attention.self.query.weight", "encoder.encoder.layer.2.attention.self.query.bias", "encoder.encoder.layer.2.attention.self.key.weight", "encoder.encoder.layer.2.attention.self.key.bias", "encoder.encoder.layer.2.attention.self.value.weight", "encoder.encoder.layer.2.attention.self.value.bias", "encoder.encoder.layer.2.attention.output.dense.weight", "encoder.encoder.layer.2.attention.output.dense.bias", "encoder.encoder.layer.2.attention.output.LayerNorm.weight", "encoder.encoder.layer.2.attention.output.LayerNorm.bias", "encoder.encoder.layer.2.intermediate.dense.weight", "encoder.encoder.layer.2.intermediate.dense.bias", "encoder.encoder.layer.2.output.dense.weight", "encoder.encoder.layer.2.output.dense.bias", "encoder.encoder.layer.2.output.LayerNorm.weight", "encoder.encoder.layer.2.output.LayerNorm.bias", "encoder.encoder.layer.3.attention.self.query.weight", "encoder.encoder.layer.3.attention.self.query.bias", "encoder.encoder.layer.3.attention.self.key.weight", "encoder.encoder.layer.3.attention.self.key.bias", "encoder.encoder.layer.3.attention.self.value.weight", "encoder.encoder.layer.3.attention.self.value.bias", "encoder.encoder.layer.3.attention.output.dense.weight", "encoder.encoder.layer.3.attention.output.dense.bias", "encoder.encoder.layer.3.attention.output.LayerNorm.weight", "encoder.encoder.layer.3.attention.output.LayerNorm.bias", "encoder.encoder.layer.3.intermediate.dense.weight", "encoder.encoder.layer.3.intermediate.dense.bias", "encoder.encoder.layer.3.output.dense.weight", "encoder.encoder.layer.3.output.dense.bias", "encoder.encoder.layer.3.output.LayerNorm.weight", "encoder.encoder.layer.3.output.LayerNorm.bias", "encoder.encoder.layer.4.attention.self.query.weight", "encoder.encoder.layer.4.attention.self.query.bias", "encoder.encoder.layer.4.attention.self.key.weight", "encoder.encoder.layer.4.attention.self.key.bias", "encoder.encoder.layer.4.attention.self.value.weight", "encoder.encoder.layer.4.attention.self.value.bias", "encoder.encoder.layer.4.attention.output.dense.weight", "encoder.encoder.layer.4.attention.output.dense.bias", "encoder.encoder.layer.4.attention.output.LayerNorm.weight", "encoder.encoder.layer.4.attention.output.LayerNorm.bias", "encoder.encoder.layer.4.intermediate.dense.weight", "encoder.encoder.layer.4.intermediate.dense.bias", "encoder.encoder.layer.4.output.dense.weight",

Hi, you don't need to download the model from the drive link. Just running as the following command, the pre-trained model will be downloaded from huggingface:

lang=php #programming language
lr=5e-5
batch_size=64
beam_size=10
source_length=256
target_length=128
data_dir=../data/code2nl/CodeSearchNet
output_dir=model/$lang
train_file=$data_dir/$lang/train.jsonl
dev_file=$data_dir/$lang/valid.jsonl
eval_steps=1000 #400 for ruby, 600 for javascript, 1000 for others
train_steps=50000 #20000 for ruby, 30000 for javascript, 50000 for others
pretrained_model=microsoft/codebert-base #Roberta: roberta-base

python run.py --do_train --do_eval --model_type roberta --model_name_or_path $pretrained_model --train_filename $train_file --dev_filename $dev_file --output_dir $output_dir --max_source_length $source_length --max_target_length $target_length --beam_size $beam_size --train_batch_size $batch_size --eval_batch_size $batch_size --learning_rate $lr --train_steps $train_steps --eval_steps $eval_steps

from codebert.

Thanks a lot ! will try the repo you mentioned.

from codebert.