Question for validation and test sets about codegen HOT 1 OPEN

Hi and sorry for the late answer,
We created the tests by generating test cases for each problem, then running the generated tests on the ground truth. We considered the unit tests we generated to be correct if they all pass for the ground truth independently for each language. For instance, if the generated unit tests succeed for the Python version but not for the C++ version, we will add an example in the valid or test set for X -> Python but not for Y -> C++. That explains why we don't have the same number of valid unit tests for C++ -> Java and Java -> C++ for instance.
It is debatable whether that's the right thing to do, as we have seen a few examples where the functions were not exactly parallel (e.g. global variables in C++ but not in Python) and that was causing the tests to fail only in C++. Taking the intersection instead would reduce the number of valid/test examples and also probably increase the average quality of the sets and result in higher scores. On the other hand, if we assume that the translation will always fail when the functions are not parallel, the current version will still correlate well with model performance and have lower variance due to us accepting extra correct examples for which only one of the test suites failed.

About the difference between the number of unit tests in the unit test folder vs what we actually test on, we removed a few functions that had too many tokens from our test set and the generated tests are still in the test folder. That's why you have 1 or 2 examples that have unit tests but that we didn't test on (e.g. for C++ 231 + 466 = 697 so one was too long).
Tell me if you still have some questions.

More precisely, we remove all the sentences with more than 512 tokens in either the source or the target language, corresponding to these test IDs DYNAMIC_PROGRAMMING_SET_37_BOOLEAN_PARENTHESIZATION_PROBLEM for the test set and all language pairs and MOBILE_NUMERIC_KEYPAD_PROBLEM for Java and Python in the validation set.

from codegen.