Hi, I am trying to replicate the results of PLBART for the code refi

Resolved with this PR (<a class="issue-link js-issue-link" data-error-text="Failed to

Thank you for pointing this out. It is a bug as we can see <a href="https://github.com

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Missing "java" token in Hugging Face Tokenizer about plbart HOT 3 CLOSED

wasiahmad commented on September 4, 2024

Missing "java" token in Hugging Face Tokenizer

from plbart.

Comments (3)

wasiahmad commented on September 4, 2024 2

Resolved with this PR (huggingface/transformers#19980).

from plbart.

wasiahmad commented on September 4, 2024 1

Thank you for pointing this out. It is a bug as we can see here, instead FAIRSEQ_LANGUAGE_CODES should be defined as:

FAIRSEQ_LANGUAGE_CODES = {
    "base": ["__java__", "__python__", "__en_XX__"],
    "multi": ["__java__", "__python__", "__en_XX__", "__javascript__", "__php__", "__ruby__", "__go__"],
}

Otherwise, the regular token java in the vocab will be treated as a special token. Now in the following:

print(tokenizer.decode(model_inputs['input_ids'][0], skip_special_tokens=True, clean_up_tokenization_spaces=False))

Since you do skip_special_tokens=True, the java token is removed.

To verify if tokenization works correct, we can do:

from transformers import PLBartTokenizer
code = "public void METHOD_1 ( TYPE_1 VAR_1 ) throws java.lang.Exception { super . METHOD_1 ( VAR_1 ) ; METHOD_2 ( VAR_1 ) ; }"
tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-base", language_codes="base")
print(tokenizer.tokenize(code))

which outputs:

['▁public', '▁void', '▁METHOD', '_1', '▁(', '▁TYPE', '_1', '▁VAR', '_1', '▁)', '▁throws', 'java', '▁.', 'lang', '.', 'Exception', '▁{', '▁super', '▁.', '▁METHOD', '_1', '▁(', '▁VAR', '_1', '▁)', '▁;', '▁METHOD', '_2', '▁(', '▁VAR', '_1', '▁)', '▁;', '▁}']

And I think the tokenization is fine.

from plbart.

wasiahmad commented on September 4, 2024 1

@gchhablani Can you help resolving the bug? The FAIRSEQ_LANGUAGE_CODES should be defined as:

FAIRSEQ_LANGUAGE_CODES = {
    "base": ["__java__", "__python__", "__en_XX__"],
    "multi": ["__java__", "__python__", "__en_XX__", "__javascript__", "__php__", "__ruby__", "__go__"],
}

from plbart.

Recommend Projects