codeflaws / codeflaws Goto Github PK

This repository the benchmark with 3902 defects extracted from C programs in Codeforces (http://codeforces.com/)

License: MIT License

Python 12.87% Perl 3.30% Shell 83.83%

codeflaws's Introduction

Codeflaws

The Codeflaws benchmark is a collection of C programs with 3902 defects. Each defect are where the dataset is crawled from Codeforces

Download:

Codeflaws is available for download at tar-link

How to use codeflaws?

Install the time command:

sudo apt-get install time

Download codeflaws:

wget http://www.comp.nus.edu.sg/~release/codeflaws/codeflaws.tar.gz
tar xf codeflaws.tar.gz

Download individual repair tool
Select defects (there are in general 3902 defects) to run by creating a file filename that is a copy of the codeflaws/all-script/codeflaws-defect-detail-info.txt file.
Modify ./run-version-.sh by setting the appropriate variables. For example, in the run-version-genprog.sh file, you need to modify the following variables:

rundir="$rootdir/genprog-run" # directory in which genprog is called from, a temporary output directory where everything will be copied to during the repair
versiondir="$rootdir/codeflaws" #directory where the codeflaws.tar.gz is extracted
filename="$rootdir/run1" #should be a copy of the codeflaws-defect-detail-info.txt
genprog="/home/ubuntu/genprog-source-v3.0/src/repair"

run the script: ./run-version-genprog.sh

Structure of each subject folder

All the subject programs are in the benchmark directory. Each subject folder is named using the following convention: <contestid>-<problem>-bug-<buggy-submisionid>-<accepted-submissionid> Each folder contains:

Buggy submission with name <contestid>-<problem>-<buggy-submisionid>.c
Accepted submission with name <contestid>-<problem>-<accepted-submisionid>.c
Two sets of test scripts:
1. Repair Test script (test suite given to repair tools for generating repair): test-genprog.sh is for search-based repair tools (GenProg, SPR, Prophet), test-angelix.sh is for Angelix as it requires inserting special instrumentation.
2. Test script for patch validation (held-out test suite): test-valid.sh is for validating the correctness of patches
Test input files: input[0-9]+ file used by Test suite (i), and heldout-input[0-9]+ file used by Test suite (ii)
Test output files: output[0-9]+ file used by Test suite (i), and heldout-output[0-9]+ file used by Test suite (ii)
Makefile for compiling the buggy submission. This contains the CFLAGS options recommended by Codeforces. To compile the accepted submission, use the command make FILENAME=10-A-13543524
Makefile.genprog for compiling the buggy submission using cilly. This is for GenProg experiments as GenProg works on CIL representation.
Test configuration for SPR that specify the name for pass/fail test: <contestid>-<problem>-<buggy-submisionid>.c.revlog

Script for running each repair tool

All the files mentioned below are stored in the all-script directory

Angelix

Use the following files for running Angelix:

File for running Angelix: run-version-angelix.sh

GenProg

Use the following files for running GenProg:

File for GenProg general configuration: configuration-default
File for compilation configuration: compile.pl
File for running GenProg: run-version-genprog.sh
File for validating patches generated by GenProg (This script is called from run-version-genprog.sh): validate-fix-genprog.sh

SPR

Use the following files for running SPR:

File for compilation configuration: code-build.py
File for test configuration: run-test.py
File for running SPR: run-version-spr.sh
File for validating patches generated by SPR (This script is called from run-version-spr.sh): validate-fix-spr.sh

Prophet

Use the following files for running Prophet:

File for compilation configuration: code-build.py
File for test configuration: run-test.py
File for running Prophet: run-version-prophet.sh
File for validating patches generated by Prophet (This script is called from run-version-prophet.sh): validate-fix-prophet.sh
Parameter file with learned model (From Prophet original experiment): para-rext-all.out

Citing Codeflaws

If you use Codeflaws in an academic work, we would be really glad if you cite our paper using the following bibtex:

@inproceedings{Tancodeflaws,
  author    = {Tan, Shin Hwei and Yi, Jooyong and Yulis and Mechtaev, Sergey and Roychoudhury,Abhik},
  title     = {Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools},
  booktitle = {2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)}, 
  year      = {2017},
  pages = {180-182}
}

Other information

For more information/questions about the benchmark, refer to the following website: https://codeflaws.github.io/

codeflaws's People

Contributors

Stargazers

Watchers

Forkers

winbobob chdd chubbymaggie yodahaji xuejiangtao jlbanks praveen-2597 pdhung3012 weifeng-liu rshariffdeen panyuepeng samramehboob pdreiter

codeflaws's Issues

absolute paths in run-version-genprog.sh

Hi,

I am trying to run genprog on the benchmarks provided. There are some absolute paths in run-version-genprog.sh file (and may be other files as well). I'm not able to figure out which file is required here?

filename="$rootdir/run1" #previous defect-movestatement

Could you please explain?

# of defects in Codeflaws.

In the readme of Codeflaws repo, it has been said that the Codeflaws contains 4085 defects (twice). However, I found only 3902 defects in http://www.comp.nus.edu.sg/~release/codeflaws/codeflaws.tar.gz file.

Has Codeflaws evolved?

Python modules import error

when i run code-build.py,it tells there is no module named tester_common,which is in the 4th line of this file.how can i solve it?

Crawler

Hello

I was wondering if the crawler you used is available.
I read in your paper that you modified https://github.com/Nymphet/codeforces-crawler to suit the purpose of the codeflaws benchmark. I was not able to find it under this repository. Is it available?

Thanks

How should i set validate-fix-genprog.sh mannualy

i find in the top lines of this file should be set by the user self. compare to the run-version-genprog.sh,
i find the second line of this file lacks two " symble.

after set the first line as it is in run-version-genprog.sh,i find there should be 4 argument input by the user.
although the names are give,i think the meaning and the usage isn't very clear yet.

the first argument ,version, i have guessed genprog/71-A-bug-18359456-18359477/tempworkdir-71-A-bug-18359456-18359477/temp-71-A-bug-18359456-18359477.out/71-A-18359456.c/temp-71-A-bug-18359456-18359477 ,but every one seems wrong

the second argument , filen , i guessed this means the number of files, or the number of lines in the copy of codeflaws-defect-detail-info.txt . because i only tested one line,i put 1.

the third argument,repairdir, i make this same with the path to the genprog in the run-version-genprog.sh.

the fourth argument,timespent, i think this is the time spended in this test,but i don't get the stat , so i input 10 as a try.

after thar,the command line hints grep -1 : no such file or directory.

maybe the meaning of the first or the third argument are misunderstood by me. Or because i dont run run-version-genprog.sh?

sorry for again and again interrupt.

Programing Problem Description

Thanks for your dataset.
May I know how to get the corresponding original problem descriptions (written in natural languages)?

Permission denied

When I run Angelix with your provided script. The following error happened.

====
+++ Running angelix with defect Group 1, group size = 1
Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 430, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'my_outputinput-neg1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 528, in
shutil.rmtree(working_dir, onerror=rm_force)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.5/shutil.py", line 432, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 524, in rm_force
shutil.rmtree(name)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree

Traceback (most recent call last):
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 528, in
shutil.rmtree(working_dir, onerror=rm_force)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.5/shutil.py", line 432, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 524, in rm_force
shutil.rmtree(name)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 396, in _rmtree_safe_fd
onerror(os.listdir, path, sys.exc_info())
File "/usr/lib/python3.5/shutil.py", line 393, in _rmtree_safe_fd
names = os.listdir(topfd)
NotADirectoryError: [Errno 20] Not a directory: '/home/bach/workspace/emsedata/script/codeflaws/all-script/angelix-experiments/.angelix/frontend/my_outputinput-neg1'

Am I doing something wrong?

Bests,

From DefectTable, Github Link for each bug is not accessible

Hi,
I was trying to access type of bugs mentioned in defect table. Can you please update link in defect table or help me with this.
https://codeflaws.github.io/defect-table.html

Looking forward to your response.
Thank you.

License?

Hi All,

What is the license of this repository?

how to get buggy line number for each file?

Hi there,
Can you please let me know how to get buggy line number from each file. Do I need to use fault localization technique for this purpose or is there any other simpler way for codeflaws data. I mean to extract buggy-fixed line pair for each file.
Any help regarding this is much appreciated.
Thank you.

Inconsistencies in the test cases

I noticed that directories with the same corresponding problem have different test cases.
For example,
476-A-bug-16608008-16608059 has an extra file heldout-input-pos1 (with "10 2" in it) compared to 476-A-bug-15778161-15778167
I assume that one problem should correspond to one suite of test cases.
How do I deal with such inconsistencies? Thanks!

How should i use the benchmark with the 4 repair tools?

sorry for my foolishment. i am confused about this for days. i tried many methods but stiil cant make the repair tools work with the benchmark.
could you tell me ,if i have installed genprog in another folder different from this folder wich contains the all-script and codeflaws. Then what should i do to make genprog run with the codeflaw? i know there is compile.pl and run-version-genprog.sh and validate-fix-genprog.sh for genprog. but what should i do ?
should i run one of them? i tried all of them ,but every time it all fails.
when i run compile.pl ,it will asks one argument as folder,i have tested every folder include:the codeflaws, the all-script, the . , the .. , and i also tried the root and the src folder of the genprog, but all fails. even when i choose the root of the genprog as the argument,it seems to walk throuth the all disks in my computer.
and when i run-version-genprog.sh , it would hint thar there is no file named run1, when i see the script, i find it means the previous defect-movestatement,but what's that exactly? and the other lines in the top of this script also confused me .
then,when i run validate-fix-genprog.sh ,it seems like dive into a forever loop and never return.

could you tell me what i should do exactly to make this benchmark work with the repair tools?

Inconsistencies in the dataset tar file

Hi, thanks for the work on this interesting dataset. I've encountered some issues that I wanted to ask:

The repository description says there are 3902 bugs in the dataset. However, there are 3905 folders in the tar file and 3903 lines in the codeflaws-defect-detail-info.txt file.

The buggy and correct source of some bugs are identical. I found these:

71-A-bug-18359456-18359477
558-B-bug-12585906-12585920
382-A-bug-8368809-8368827
569-C-bug-12481867-12481905
289-D-bug-3473596-3473601
289-D-bug-3473592-3473601
431-C-bug-15194556-15194577

Some bugs don't have any held-out tests or their test-valid.sh script is empty:

254-B-bug-2772031-2772034
254-A-bug-3232299-3232307
7-B-bug-14690359-14690378
120-A-bug-3755986-3755994
234-A-bug-17608639-17608751
518-D-bug-10002685-10002791
397-B-bug-6197060-6197072
1-C-bug-15203581-15203678
35-A-bug-14752871-14752907
234-A-bug-17608640-17608751
234-B-bug-15831884-15832068
234-C-bug-3793024-3793031
435-C-bug-6755082-6755089
576-A-bug-13022096-13022115
35-A-bug-2853912-2853922
71-A-bug-18359456-18359477
234-B-bug-2494175-2494181
234-A-bug-2519091-2519130
120-A-bug-3756299-3756385
120-B-bug-3752196-3752201
379-B-bug-6978141-6978150
120-E-bug-3535976-3536067
364-C-bug-5274751-5274860
120-E-bug-7236446-7236513
120-B-bug-3754178-3754181

Some correct versions don't pass all the tests. For example, the correct version of 467-B-bug-15440653-15441226 doesn't pass p13 test case, and I have found that changing the gcc version changes the output. So, is there a preferred gcc version to use with the dataset?