tokenizer = transformers.AutoTokenizer.from_pretrained("Salesforce/codegen-350M-multi")
model = transformers.CodeGenForCausalLM.from_pretrained(pretrain_dir,config=os.path.join(pretrain_dir,"config.json"))   

Traceback (most recent call last):
  File "/home/User/code-models/get_retrained_model_distribution.py", line 54, in <module>
    tokenizer, retrained_model = model_utils.load_retrained_model(f"output/finetune_codegen/20230316-{args.ds_type}-Epoch30/final_checkpoint-{args.ds_type}-1",model_name)
  File "/home/User/code-models/utils/model_utils.py", line 68, in load_retrained_model
    model = transformers.CodeGenForCausalLM.from_pretrained(pretrain_dir,config=os.path.join(pretrain_dir,"config.json"), local_files_only=True)    
  File "/home/User/.conda/envs/transformers/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2379, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/User/.conda/envs/transformers/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2695, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for CodeGenForCausalLM:
        size mismatch for transformer.h.0.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.0.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.0.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.0.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.1.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.1.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.1.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.1.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.2.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.2.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.2.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.2.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.3.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.3.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.3.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.3.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.4.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.4.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.4.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.4.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.5.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.5.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.5.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.5.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.6.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.6.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.6.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.6.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.7.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.7.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.7.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.7.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.8.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.8.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.8.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.8.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.9.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.9.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.9.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.9.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.10.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.10.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.10.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.10.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.11.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.11.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.11.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.11.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.12.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.12.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.12.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.12.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.13.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.13.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.13.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.13.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.14.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.14.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.14.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.14.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.15.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.15.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.15.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.15.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.16.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.16.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.16.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.16.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.17.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.17.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.17.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.17.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.18.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.18.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.18.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.18.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for transformer.h.19.attn.qkv_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([3072, 1024]).
        size mismatch for transformer.h.19.attn.out_proj.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
        size mismatch for transformer.h.19.mlp.fc_in.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
        size mismatch for transformer.h.19.mlp.fc_out.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
        size mismatch for lm_head.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([51200, 1024]).
        size mismatch for lm_head.bias: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([51200]).
        You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.

Model params_num: 7064217600, params_size: 28256.87MB
Model params_num_total: 7064217600, params_size_total: 28256.87MB
{'params_num': 7064217600, 'params_size': 28256870400, 'params_num_total': 7064217600, 'params_size_total': 28256870400}

The multi-lingual dataset BIGQUERY is a subset of Google’s publicly available BigQuery dataset,
which consists of code (under open-source license) in multiple programming languages.