How to use new BPE codes and vocab to generate parallel data about codegen HOT 5 OPEN

Hi

1. You just need to pass your .code and .vocab files to preprocess.py using the fastbpe_vocab_path and fastbpe_code_path parameters. It will reload your BPE instead of computing a new one.
2. test.cpp_sa-java_sa.cpp_sa.pth contains the cpp functions for cpp-java translation and test.cpp_sa-java_sa.java_sa.pth contains the java functions.
3. We used the BPE codes and vocab learned from the monolingual dataset (used for MLM) when computing the monolingual function dataset using the parameters I mentioned in 1.

from codegen.

Thanks @brozi for the reply.
Few more clarifications questions.

1. In case we don't have unit tests, then in AE and BT training do we require to create 'train.cpp_sa-java_sa.java_sa.pth', val.cpp_sa-java_sa.java_sa.pth, and test.cpp_sa-java_sa.java_sa.pth files ? In case what will be the value of --bt_steps parameter ?
2. If we have some parallel train set then just adding mt_steps will train the model on parallel train set ? Also what kind of loss function is used in mt_steps and bt_steps?
3. What is use of line 539 in evaluator.py

from codegen.

Hi @dineshkh,

1. You still need to keep the same bt_steps parameters no matter what your test/valid set is. If you don't have unit tests, you can set --eval_computation to false and your models will be evaluated with metrics such as BLEU, token accuracy, and perplexity. Our code would still look for parallel test/valid files to compute these metrics. You could either modify the code to not do any evaluation or add files with just one line with a simple example. Note: 'train.cpp_sa-java_sa.java_sa.pth' would be a train file with parallel cpp/java data and we don't use anything like that in TransCoder.
2. Yes the simplest way to use a parallel train set is to add mt_steps to train to translate. Both mt and bt steps are training using the cross-entropy. The difference is that we generate the source sentence in back-translation (allowing to train with monolingual datasets) while mt_steps use the sentence from the dataset.
3. It just limits the time spent evaluating MLM. It would probably be better to clean this up by making turning it into a parameter.

from codegen.

Thanks @brozi.

Can you please tell me how MT and BT steps are training using the cross-entropy loss ? Is it summing binary cross-entropy over all the tokens in the ground truth sequence? Any references or material you can point ?

from codegen.

@brozi thanks for your explanations, but to me it is not unclear how the parallel files like train.cpp_sa-java_sa.java_sa.pth are produced. the preprocessing pipeline seems to process each language independently. at which stage are they connected? tokenization, bpe, or binarization stage? do I need to put both languages in a single json, somehow? thanks

from codegen.