Git Product home page Git Product logo

codeflaws's Introduction

Codeflaws

The Codeflaws benchmark is a collection of C programs with 3902 defects. Each defect are where the dataset is crawled from Codeforces

Download:

Codeflaws is available for download at tar-link

How to use codeflaws?

  1. Install the time command:
sudo apt-get install time
  1. Download codeflaws:
wget http://www.comp.nus.edu.sg/~release/codeflaws/codeflaws.tar.gz
tar xf codeflaws.tar.gz
  1. Download individual repair tool
  2. Select defects (there are in general 3902 defects) to run by creating a file filename that is a copy of the codeflaws/all-script/codeflaws-defect-detail-info.txt file.
  3. Modify ./run-version-.sh by setting the appropriate variables. For example, in the run-version-genprog.sh file, you need to modify the following variables:
rundir="$rootdir/genprog-run" # directory in which genprog is called from, a temporary output directory where everything will be copied to during the repair
versiondir="$rootdir/codeflaws" #directory where the codeflaws.tar.gz is extracted
filename="$rootdir/run1" #should be a copy of the codeflaws-defect-detail-info.txt
genprog="/home/ubuntu/genprog-source-v3.0/src/repair"
  1. run the script: ./run-version-genprog.sh

Structure of each subject folder

All the subject programs are in the benchmark directory. Each subject folder is named using the following convention: <contestid>-<problem>-bug-<buggy-submisionid>-<accepted-submissionid> Each folder contains:

  • Buggy submission with name <contestid>-<problem>-<buggy-submisionid>.c
  • Accepted submission with name <contestid>-<problem>-<accepted-submisionid>.c
  • Two sets of test scripts:
    1. Repair Test script (test suite given to repair tools for generating repair): test-genprog.sh is for search-based repair tools (GenProg, SPR, Prophet), test-angelix.sh is for Angelix as it requires inserting special instrumentation.
    2. Test script for patch validation (held-out test suite): test-valid.sh is for validating the correctness of patches
  • Test input files: input[0-9]+ file used by Test suite (i), and heldout-input[0-9]+ file used by Test suite (ii)
  • Test output files: output[0-9]+ file used by Test suite (i), and heldout-output[0-9]+ file used by Test suite (ii)
  • Makefile for compiling the buggy submission. This contains the CFLAGS options recommended by Codeforces. To compile the accepted submission, use the command make FILENAME=10-A-13543524
  • Makefile.genprog for compiling the buggy submission using cilly. This is for GenProg experiments as GenProg works on CIL representation.
  • Test configuration for SPR that specify the name for pass/fail test: <contestid>-<problem>-<buggy-submisionid>.c.revlog

Script for running each repair tool

All the files mentioned below are stored in the all-script directory

Angelix

Use the following files for running Angelix:

  • File for running Angelix: run-version-angelix.sh

GenProg

Use the following files for running GenProg:

  • File for GenProg general configuration: configuration-default
  • File for compilation configuration: compile.pl
  • File for running GenProg: run-version-genprog.sh
  • File for validating patches generated by GenProg (This script is called from run-version-genprog.sh): validate-fix-genprog.sh

SPR

Use the following files for running SPR:

  • File for compilation configuration: code-build.py
  • File for test configuration: run-test.py
  • File for running SPR: run-version-spr.sh
  • File for validating patches generated by SPR (This script is called from run-version-spr.sh): validate-fix-spr.sh

Prophet

Use the following files for running Prophet:

  • File for compilation configuration: code-build.py
  • File for test configuration: run-test.py
  • File for running Prophet: run-version-prophet.sh
  • File for validating patches generated by Prophet (This script is called from run-version-prophet.sh): validate-fix-prophet.sh
  • Parameter file with learned model (From Prophet original experiment): para-rext-all.out

Citing Codeflaws

If you use Codeflaws in an academic work, we would be really glad if you cite our paper using the following bibtex:

@inproceedings{Tancodeflaws,
  author    = {Tan, Shin Hwei and Yi, Jooyong and Yulis and Mechtaev, Sergey and Roychoudhury,Abhik},
  title     = {Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools},
  booktitle = {2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)}, 
  year      = {2017},
  pages = {180-182}
}

Other information

For more information/questions about the benchmark, refer to the following website: https://codeflaws.github.io/

codeflaws's People

Contributors

stan6 avatar mechtaev avatar

Stargazers

Daniel Oh avatar Emma avatar xrkk avatar Tianze Ds avatar  avatar  avatar  avatar Shiyu Sun avatar Reza Gharibi avatar Kyoosik Lee avatar SamraMehboob avatar Nghi D. Q. Bui avatar  avatar  avatar -O- avatar Nguyen Thanh Dat avatar Jooyong Yi avatar Ziqiang Li avatar Alex Yu avatar Nick Sumner avatar Bowen(Brad) Xu avatar Jordan Henkel avatar Chris Parnin avatar  avatar Taejoon Byun avatar yxliang avatar Xujie Si avatar Yushan avatar Manh Nguyen avatar  avatar Stanislav Beliaev avatar Chris Timperley avatar  avatar  avatar

Watchers

Huascar Sanchez avatar  avatar James Cloos avatar Ridwan Shariffdeen avatar Taejoon Byun avatar yxliang avatar  avatar

codeflaws's Issues

absolute paths in run-version-genprog.sh

Hi,

I am trying to run genprog on the benchmarks provided. There are some absolute paths in run-version-genprog.sh file (and may be other files as well). I'm not able to figure out which file is required here?

filename="$rootdir/run1" #previous defect-movestatement

Could you please explain?

Python modules import error

when i run code-build.py,it tells there is no module named tester_common,which is in the 4th line of this file.how can i solve it?

Crawler

Hello

I was wondering if the crawler you used is available.
I read in your paper that you modified https://github.com/Nymphet/codeforces-crawler to suit the purpose of the codeflaws benchmark. I was not able to find it under this repository. Is it available?

Thanks

How should i set validate-fix-genprog.sh mannualy

i find in the top lines of this file should be set by the user self. compare to the run-version-genprog.sh,
i find the second line of this file lacks two " symble.

after set the first line as it is in run-version-genprog.sh,i find there should be 4 argument input by the user.
although the names are give,i think the meaning and the usage isn't very clear yet.

the first argument ,version, i have guessed genprog/71-A-bug-18359456-18359477/tempworkdir-71-A-bug-18359456-18359477/temp-71-A-bug-18359456-18359477.out/71-A-18359456.c/temp-71-A-bug-18359456-18359477 ,but every one seems wrong

the second argument , filen , i guessed this means the number of files, or the number of lines in the copy of codeflaws-defect-detail-info.txt . because i only tested one line,i put 1.

the third argument,repairdir, i make this same with the path to the genprog in the run-version-genprog.sh.

the fourth argument,timespent, i think this is the time spended in this test,but i don't get the stat , so i input 10 as a try.

after thar,the command line hints grep -1 : no such file or directory.

maybe the meaning of the first or the third argument are misunderstood by me. Or because i dont run run-version-genprog.sh?

sorry for again and again interrupt.

Programing Problem Description

Thanks for your dataset.
May I know how to get the corresponding original problem descriptions (written in natural languages)?

Permission denied

When I run Angelix with your provided script. The following error happened.

====
+++ Running angelix with defect Group 1, group size = 1
Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 430, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'my_outputinput-neg1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 528, in
shutil.rmtree(working_dir, onerror=rm_force)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.5/shutil.py", line 432, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 524, in rm_force
shutil.rmtree(name)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree

Traceback (most recent call last):
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 528, in
shutil.rmtree(working_dir, onerror=rm_force)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.5/shutil.py", line 432, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/bach/workspace/repairtools/angelix/src/repair/main.py", line 524, in rm_force
shutil.rmtree(name)
File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.5/shutil.py", line 396, in _rmtree_safe_fd
onerror(os.listdir, path, sys.exc_info())
File "/usr/lib/python3.5/shutil.py", line 393, in _rmtree_safe_fd
names = os.listdir(topfd)
NotADirectoryError: [Errno 20] Not a directory: '/home/bach/workspace/emsedata/script/codeflaws/all-script/angelix-experiments/.angelix/frontend/my_outputinput-neg1'

Am I doing something wrong?

Bests,

License?

Hi All,

What is the license of this repository?

how to get buggy line number for each file?

Hi there,
Can you please let me know how to get buggy line number from each file. Do I need to use fault localization technique for this purpose or is there any other simpler way for codeflaws data. I mean to extract buggy-fixed line pair for each file.
Any help regarding this is much appreciated.
Thank you.

Inconsistencies in the test cases

I noticed that directories with the same corresponding problem have different test cases.
For example,
476-A-bug-16608008-16608059 has an extra file heldout-input-pos1 (with "10 2" in it) compared to 476-A-bug-15778161-15778167
I assume that one problem should correspond to one suite of test cases.
How do I deal with such inconsistencies? Thanks!

How should i use the benchmark with the 4 repair tools?

sorry for my foolishment. i am confused about this for days. i tried many methods but stiil cant make the repair tools work with the benchmark.
could you tell me ,if i have installed genprog in another folder different from this folder wich contains the all-script and codeflaws. Then what should i do to make genprog run with the codeflaw? i know there is compile.pl and run-version-genprog.sh and validate-fix-genprog.sh for genprog. but what should i do ?
should i run one of them? i tried all of them ,but every time it all fails.
when i run compile.pl ,it will asks one argument as folder,i have tested every folder include:the codeflaws, the all-script, the . , the .. , and i also tried the root and the src folder of the genprog, but all fails. even when i choose the root of the genprog as the argument,it seems to walk throuth the all disks in my computer.
and when i run-version-genprog.sh , it would hint thar there is no file named run1, when i see the script, i find it means the previous defect-movestatement,but what's that exactly? and the other lines in the top of this script also confused me .
then,when i run validate-fix-genprog.sh ,it seems like dive into a forever loop and never return.

could you tell me what i should do exactly to make this benchmark work with the repair tools?

Inconsistencies in the dataset tar file

Hi, thanks for the work on this interesting dataset. I've encountered some issues that I wanted to ask:

  1. The repository description says there are 3902 bugs in the dataset. However, there are 3905 folders in the tar file and 3903 lines in the codeflaws-defect-detail-info.txt file.

  2. The buggy and correct source of some bugs are identical. I found these:

    71-A-bug-18359456-18359477
    558-B-bug-12585906-12585920
    382-A-bug-8368809-8368827
    569-C-bug-12481867-12481905
    289-D-bug-3473596-3473601
    289-D-bug-3473592-3473601
    431-C-bug-15194556-15194577
    
  3. Some bugs don't have any held-out tests or their test-valid.sh script is empty:

    254-B-bug-2772031-2772034
    254-A-bug-3232299-3232307
    7-B-bug-14690359-14690378
    120-A-bug-3755986-3755994
    234-A-bug-17608639-17608751
    518-D-bug-10002685-10002791
    397-B-bug-6197060-6197072
    1-C-bug-15203581-15203678
    35-A-bug-14752871-14752907
    234-A-bug-17608640-17608751
    234-B-bug-15831884-15832068
    234-C-bug-3793024-3793031
    435-C-bug-6755082-6755089
    576-A-bug-13022096-13022115
    35-A-bug-2853912-2853922
    71-A-bug-18359456-18359477
    234-B-bug-2494175-2494181
    234-A-bug-2519091-2519130
    120-A-bug-3756299-3756385
    120-B-bug-3752196-3752201
    379-B-bug-6978141-6978150
    120-E-bug-3535976-3536067
    364-C-bug-5274751-5274860
    120-E-bug-7236446-7236513
    120-B-bug-3754178-3754181
    
  4. Some correct versions don't pass all the tests. For example, the correct version of 467-B-bug-15440653-15441226 doesn't pass p13 test case, and I have found that changing the gcc version changes the output. So, is there a preferred gcc version to use with the dataset?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.