sriniiyer / codenn Goto Github PK

In your paper, you mentioned that you have conduct experiments on the tree structure of source code, could you tell me how to build the tree for C# using ANTLR? And it will be highly appreciated if you can share the code of handling abstract syntax trees.

I am trying to run the model so I installed antlr4 as follows:

(venv) user@MacBook-Pro-de-user:~/codenn/src/model$ sudo pip2 install 'antlr4-python2-runtime==4.5.2' --force-reinstall
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
WARNING: The directory '/Users/user/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/Users/user/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting antlr4-python2-runtime==4.5.2
Installing collected packages: antlr4-python2-runtime
  Found existing installation: antlr4-python2-runtime 4.7.2
    Uninstalling antlr4-python2-runtime-4.7.2:
      Successfully uninstalled antlr4-python2-runtime-4.7.2
Successfully installed antlr4-python2-runtime-4.5.2

Then , I tried to build the data like this:

(venv) user@MacBook-Pro-de-user:~/codenn/src/model$ ./buildData.sh
Total size = 23970
Total skipped = 1701
Total size = 3110
Total skipped = 216
Total size = 111
Total skipped = 0
Total size = 100
Total skipped = 0
line 3:4 token recognition error at: '$'
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'

However, I am having a number of token recognition errors. How can I fix this? Do I need to modify the grammar?

Thanks!

There is much broken data starting from line 2353 of csharp/train.txt(https://raw.githubusercontent.com/sriniiyer/codenn/master/data/stackoverflow/csharp/train.txt

), like

29768982	29768200	C# sorting multidimensional array by multiple columns	using System;
\nusing System.Collections.Generic;
\nusing System.Linq;
\nusing System.Text;
\n
\nnamespace ConsoleApplication19
\n{
\n    class Program
\n    {
\n        static void Main(string[] args)
\n        {
\n            List<List<int>> multiarray = new List<List<int>>{    
\n                new List<int> { 8, 63  },
\n                new List<int>  { 4, 2   }, 
\n                new List<int>  { 0, -55 }, 
\n                new List<int>  { 8, 57  }, 
\n                new List<int>  { 2, -120}, 
\n                new List<int>  { 8, 53  }  
\n            };
\n           
\n
\n            List<List<int>> sortedList = multiarray.OrderBy(x => x[1]).OrderBy(y => y[0]).ToList();
\n
\n        }
\n    }
\n}	0

Sorry for bothering, when I trying to run the setup.py to install and setup the sqlparse. It appear a error called "error in sqlparse setup command: use_2to3 is invalid.". The error still appear after I change the setuptools version = 57.5.0 or 58.

It would be very helpful if that can be provided i am not able to run this.

What should I do?
Total Tokens: 31667
Total Words: 7470
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-4410/cutorch/lib/THC/generic/THCTensorMath.cu line=35 error=48 : no kernel image is available for execution on the device /home/jinyuliu/torch/install/bin/luajit: ./encoder.lua:13: cuda runtime error (48) : no kernel image is available for execution on the device at /tmp/luarocks_cutorch-scm-1-4410/cutorch/lib/THC/generic/THCTensorMath.cu:35
stack traceback:
[C]: in function 'zero'
./encoder.lua:13: in function '__init'
/home/jinyuliu/torch/install/share/lua/5.1/torch/init.lua:91: in function </home/jinyuliu/torch/install/share/lua/5.1/torch/init.lua:87>
[C]: in function 'Encoder'
main.lua:8: in function 'setup'
main.lua:118: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...uliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/jinyuliu/torch/install/bin/luajit: cannot open <csharp.encoder> in mode r at /home/jinyuliu/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7f3ed7c9a460
[C]: in function 'DiskFile'
/home/jinyuliu/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
predict.lua:215: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...uliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

Excuse me... I have another error when I build dataset.
I set the PYTHONPATH and run ./buildData.sh and no vocab.data.sql file in workdir but vocab.sql
then I renamed this file to vocab.data.sql and run ./run.sh sql
and I getting this error now..

(py2) linux@ubuntu1604:~/codenn/src/model$ ./run.sh sql
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
main.lua:102: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
predict.lua:213: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

It's seemd that torch load() can't load this file .
Thanks for your help.

~/codenn/src/sqlparse$ sudo python setup.py install
/usr/local/lib/python2.7/dist-packages/setuptools/dist.py:285: UserWarning: Normalizing '0.1.16-dev' to '0.1.16.dev0'
normalized_version,
running install
running bdist_egg
running egg_info
writing sqlparse.egg-info/PKG-INFO
writing top-level names to sqlparse.egg-info/top_level.txt
writing dependency_links to sqlparse.egg-info/dependency_links.txt
reading manifest file 'sqlparse.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sqlparse.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
error: file '/home/jc/codenn/src/sqlparse/bin/sqlformat' does not exist

I'm getting this error when I try to install the modified sqlparse.

~/examples/codenn/src/model$ ./run.sh csharp
/opt/torch/install/bin/luajit: main.lua:102: attempt to concatenate local 'working_dir' (a nil value)
stack traceback:
main.lua:102: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
/opt/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/opt/torch/install/bin/luajit: predict.lua:213: attempt to concatenate local 'working_dir' (a nil value)
stack traceback:
predict.lua:213: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
/opt/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

when I run ./run.sh {csharp}, an error occurs.
I installed Torch7 in my PC. Is there a version mistake of torch?
/home/jinyuliu/torch/install/bin/luajit: cannot open <./workdir/vocab.data.{csharp}> in mode r at /home/jinyuliu/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7fd108b9e460
[C]: in function 'DiskFile'
/home/jinyuliu/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
main.lua:102: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...uliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/jinyuliu/torch/install/bin/luajit: cannot open <./workdir/vocab.data.{csharp}> in mode r at /home/jinyuliu/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7fdd87637460
[C]: in function 'DiskFile'
/home/jinyuliu/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
predict.lua:213: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...uliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

$ ./run.sh csharp

Total Tokens: 31667
Total Words: 7470
{
batch_size : 100
learningRate : 0.5
max_length : 20
init_weight : 0.35
decay : 0.8
gpu : 1
max_grad_norm : 5
decoder_emb : 400
max_code_length : 100
dev_ref_file : "/home/ronxin/codenn//data/stackoverflow/csharp/dev/ref.txt"
normalize : 1
rnn_size : 400
beam_size : 10
dropout : 0.5
max_nl_length : 100
layers : 1
encoder_emb : 400
}
/home/ronxin/torch/install/bin/luajit: /home/ronxin/torch/install/share/lua/5.1/nn/LookupTable.lua:75: bad argument #3 to 'index' (Tensor | LongTensor expected, got torch.CudaLongTensor)
stack traceback:
[C]: in function 'index'
/home/ronxin/torch/install/share/lua/5.1/nn/LookupTable.lua:75: in function 'func'
...e/ronxin/torch/install/share/lua/5.1/nngraph/gmodule.lua:345: in function 'neteval'
...e/ronxin/torch/install/share/lua/5.1/nngraph/gmodule.lua:380: in function 'forward'
main.lua:13: in function 'fp'
main.lua:160: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...nxin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x004064f0
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/ronxin/torch/install/bin/luajit: cannot open <csharp.encoder> in mode r at /home/ronxin/torch/pkg/torch/lib/TH/THDiskFile.c:649
stack traceback:
[C]: at 0x2ad99efec780
[C]: in function 'DiskFile'
/home/ronxin/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
predict.lua:215: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...nxin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x004064f0

This issue and this post seem to be related, but there is no posted solution yet.

Traceback (most recent call last):
File "buildData.py", line 1, in
from sql.SqlTemplate import SqlTemplate
ImportError: No module named sql.SqlTemplate
Traceback (most recent call last):
File "buildData.py", line 1, in
from sql.SqlTemplate import SqlTemplate
ImportError: No module named sql.SqlTemplate

I'm getting this error when i build dataset.

~/muse/codenn/src/model$ ./run.sh csharp
Total Tokens: 31667
Total Words: 7470
{
batch_size : 100
learningRate : 0.5
max_length : 20
init_weight : 0.35
decay : 0.8
gpu : 1
max_grad_norm : 5
decoder_emb : 400
max_code_length : 100
dev_ref_file : "/home/jc/codenn//data/stackoverflow/csharp/dev/ref.txt"
normalize : 1
rnn_size : 400
beam_size : 10
dropout : 0.5
max_nl_length : 100
layers : 1
encoder_emb : 400
}
predicting
python ../utils/bleu.py /home/jc/codenn//data/stackoverflow/csharp/dev/ref.txt < /tmp/lua_gGeOl0

Traceback (most recent call last):
File "../utils/bleu.py", line 198, in < module >
(goldMap, predictionMap) = computeMaps(predictions, reference_file)
File "../utils/bleu.py", line 159, in computeMaps
gf = open(goldfile, 'r')
IOError: [Errno 2] No such file or directory: '/home/jc/codenn//data/stackoverflow/csharp/dev/ref.txt'
/opt/torch/install/bin/luajit: main.lua:54: attempt to concatenate local 'bleu' (a nil value)
stack traceback:
main.lua:54: in function 'run_bleu'
main.lua:170: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
/opt/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/opt/torch/install/bin/luajit: cannot open < csharp.encoder > in mode r at /opt/torch/pkg/torch/lib/TH/THDiskFile.c:649
stack traceback:
[C]: at 0x7f1a2c5ff7b0
[C]: in function 'DiskFile'
/opt/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
predict.lua:215: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
/opt/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

Thank you for great job!

I am working on this dataset for a course project and benefits much from this repo.

In this repo, I only found a script that calculates BLEU score, could you please provide the script that calculates METEOR score?

Thank you very much!

I'm new to this field...
I was running my java data on the model, and after 22hours' running, the connection to my lab server failed...
How can I continue the process?

Besides, the java training data is 69k lines, and my lab's GPU is NVIDIA Tesla P4 (8 GB GDDR5), after 22 hours' training, the learning rate is still 0.32(init = 0.5), is it too slow?

Sorry I'm really a green hand and may ask some stupid questions... Thanks for your help:)