How to extract functions? about codegen HOT 4 CLOSED

Whether CodeGen's preprocessing steps can extract comments? I find there is a parameter --keep_comments in TransCoder. But CodeGen doesn't have.

from codegen.

1/2 - This format python.*[0-4][0-9][0-9].json.gz includes python.000000000000.json.gz because of the *, so this works with this format as well. If this would be the error, then you will have had an error at the very first step "no json files found in folder". The error you have is very often due to the facts you ask to learn 50k codes on only a signle small python json. You should consider learning fewer codes and learning on more data.

3 - The others file are intermediate files necessary to run the pipeline, what you want to keep is only the .pth and their symlinks.

4- no feature in the pipeline to have description

5- we only keep the keep-comments=True by default as it is what we use in our models, but you can easily add a param to do so and set this params to False, as we still have keep-comments params in all tokenization/function extraction functions.

from codegen.

1/2 - This format python.*[0-4][0-9][0-9].json.gz includes python.000000000000.json.gz because of the *, so this works with this format as well. If this would be the error, then you will have had an error at the very first step "no json files found in folder". The error you have is very often due to the facts you ask to learn 50k codes on only a signle small python json. You should consider learning fewer codes and learning on more data.

3 - The others file are intermediate files necessary to run the pipeline, what you want to keep is only the .pth and their symlinks.

4- no feature in the pipeline to have description

5- we only keep the keep-comments=True by default as it is what we use in our models, but you can easily add a param to do so and set this params to False, as we still have keep-comments params in all tokenization/function extraction functions.

Do you keep the keep-comments=True by default? So I don't need to set keep-comments=True in CodeGen, right?
But I run the preprocess code and get the functions without comments:

When keep-comments is set to True, doesn't it mean keeping comments in the code? Why does the extracted function have no comments?

from codegen.

1/2 - This format python.*[0-4][0-9][0-9].json.gz includes python.000000000000.json.gz because of the *, so this works with this format as well. If this would be the error, then you will have had an error at the very first step "no json files found in folder". The error you have is very often due to the facts you ask to learn 50k codes on only a signle small python json. You should consider learning fewer codes and learning on more data.

3 - The others file are intermediate files necessary to run the pipeline, what you want to keep is only the .pth and their symlinks.

4- no feature in the pipeline to have description

5- we only keep the keep-comments=True by default as it is what we use in our models, but you can easily add a param to do so and set this params to False, as we still have keep-comments params in all tokenization/function extraction functions.

The description in the fourth question means comment. Now I can extract functions. But the comments can't be extracted. How can I extract functions with comments?

from codegen.