Defects4J is a collection of reproducible bugs and a supporting infrastructure with the goal of advancing software engineering research.

Defects4J contains 395 bugs from the following open-source projects:

Each bug has the following properties:

• Issue filed in the corresponding issue tracker, and issue tracker identifier mentioned in the fixing commit message.
• Fixed in a single commit
• Minimized: the Defects4J maintainers manually pruned out irrelevant changes in the commit (e.g., refactorings or feature additions).
• Fixed by modifying the source code (as opposed to configuration files, documentation, or test files).
• A triggering test exists that failed before the fix and passes after the fix -- the test failure is not random or dependent on test execution order.

The (b)uggy and (f)ixed program revisions are labelled with <id>b and <id>f, respectively (<id> is an integer).

• Java 1.7
• Git >= 1.9
• SVN >= 1.8
• Perl >= 5.0.10

All bugs have been reproduced and triggering tests verified, using the latest version of Java 1.7. Note that using Java 1.8+ might result in unexpected failing tests on a fixed program version. The next major release of Defects4J will be compatible with Java 8.

All required Perl modules are listed in cpanfile. On many Unix platforms, these required Perl modules are installed by default. If this is not the case, you can use cpan (or a cpan wrapper) to install them. For example, if you have cpanm installed, you can automatically install all modules by running: cpanm --installdeps .

Defects4J generates and executes tests in the timezone America/Los_Angeles. If you are using the bugs outside of the Defects4J framework, set the TZ environment variable to America/Los_Angeles and export it.

1. Clone Defects4J:

   • git clone https://github.com/rjust/defects4j

2. Initialize Defects4J (download the project repositories and external libraries, which are not included in the git repository for size purposes and to avoid redundancies):

   • cd defects4j
   • ./init.sh

3. Add Defects4J's executables to your PATH:

   • export PATH=$PATH:"path2defects4j"/framework/bin

4. Check installation:

   • defects4j info -p Lang

1. Get information for a specific project (commons lang):

   • defects4j info -p Lang

2. Get information for a specific bug (commons lang, bug 1):

   • defects4j info -p Lang -b 1

3. Checkout a buggy source code version (commons lang, bug 1, buggy version):

   • defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy

4. Change to the working directory, compile sources and tests, and run tests:

   • cd /tmp/lang_1_buggy
   • defects4j compile
   • defects4j test

5. The scripts in framework/test/ are examples of how to use Defects4J, which you might find useful as inspiration when you are writing your own scripts that use Defects4J.

Use framework/bin/defects4j to execute any of the following commands:

Use defects4j export -p <property_name> [-o output_file] in the working directory to export a version-specific property:

The test execution framework for generated test suites (framework/bin) provides the following scripts:

• Scripts and annotations for evaluating FL techniques

• Scripts and annotations for evaluating APR techniques
• Patches generated with the Nopol, jGenProg, and jKali APR systems
• Repair actions and patterns for Defects4J v1.2.0

• "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs" René Just, Darioush Jalali, and Michael D. Ernst, ISSTA 2014 [download].

• "Are Mutants a Valid Substitute for Real Faults in Software Testing?" René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser, FSE 2014 [download].

More publications

Documentation for any script or module is available as html documentation.

The directory structure of Defects4J is as follows:

defects4j
   |
   |--- project_repos:     The version control repositories of the provided projects.
   |
   |--- major:             The Major mutation framework.
   |
   |--- framework:         Libraries and executables of the database abstraction and
       |                   test execution framework.
       |
       |--- bin:           Command line interface to Defects4J.
       |
       |--- core:          The modules of the core framework.
       |
       |--- lib:           Libraries used in the core framework.
       |
       |--- util:          Util scripts used by Defects4J.
       |
       |--- projects:      Project-specific resource files.
       |
       |--- test:          Scripts to test the framework.

MIT License, see license.txt for more information.