This document describes the rules of identifying the correctness of patches generated by APR tools. Please consider to cite the following paper if you are interested in the rules.

@inproceedings{liu2020efficiency,
  author    = {Kui Liu and Shangwen Wang and Anil Koyuncu and Kisub Kim and Tegawend{\'{e}} F. Bissyand{\'{e}} and Dongsun Kim and Peng Wu and Jacques Klein and Xiaoguang Mao and Yves Le Traon},
  title     = {On the efficiency of test suite based program repair: {A} Systematic Assessment of 16 Automated Repair Systems for Java Programs},
  booktitle = {Proceedings of the 42nd International Conference on Software Engineering},
  pages     = {615--627},
  publisher = {{ACM}},
  year      = {2020},
  url       = {https://doi.org/10.1145/3377811.3380338},
  doi       = {10.1145/3377811.3380338}
}

ACS and SimFix are available at: https://drive.google.com/drive/folders/1OS9-GG2WQjFpvOPARaLW-k9b-9ZpO_3J?usp=sharing

kPAR: https://github.com/SerVal-DTF/FL-VS-APR/tree/master/kPAR

AVATAR: https://github.com/SerVal-DTF/AVATAR

TBar: https://github.com/SerVal-DTF/TBar

The patch generated by an APR tool is identical to the developer's patch except for the diffs on format and comments.

ACS: [Chart-14], [Math-4];

ARJA: [Closure-126], [Math-98];

AVATAR: [Chart-1], [Chart-4], [Chart-24], [Closure-46], [Closure-62], [Closure-63], [Closure-73], [Lang-6], [Math-4], [Math-59];

Cardumen: [Math-70];

FixMiner: [Chart-1], [Chart-4], [Math-22], [Math-30], [Math-34], [Math-57], [Math-70], [Math-75], [Math-79];

jGenProg: [Math-5], [Math-53], [Math-70];

jMutRepair: [Chart-1], [Closure-62], [Closure-63], [Math-82], [Math-85];

kPAR: [Chart-1], [Chart-4], [Closure-73], [Math-58], [Math-70], [Math-75];

RSRepair: [Closure-126];

SimFix: [Chart-1], [Chart-7], [Chart-20], [Closure-14], [Closure-57], [Closure-62], [Closure-63], [Closure-73], [Lang-33], [Lang-43], [Lang-58], [Lang-60], [Math-5], [Math-41], [Math-53], [Math-57], [Math-59], [Math-70], [Math-(71)], [Math-75], [Math-79];

TBar: [Chart-1], [Chart-4], [Chart-9], [Chart-20], [Chart-24], Closure-62], Closure-63], Closure-73], [Lang-57], [Lang-59], [Math-5], [Math-57], [Math-70], [Math-75], [Math-79].

[AVATAR: [Chart-11];

Cardumen: [Chart-11];

FixMiner: [Chart-11];

TBar: [Chart-11].

The APR-generated patch has the same exception as the developer's patch, but does not specify the identical and concrete message.

ACS: [Chart-19], [Lang-35], [Math-25], [Math-61], [Math-89], [Math-90], [Math-99], [Time-15];

AVATAR: [Chart-19], [Math-89];

FixMiner: [Chart-19];

kPAR: [Chart-19], [Math-89];

TBar: [Chart-19], [Math-89].

The patch is generated by using a new initialized variable but not a default variable in the program, the two variables return the same value.

ARJA: [Math-58];

TBar: [Lang-47].

The APR patch is implemented by inserting new if statement but not a ternary operator to fix the bug.

TBar: [Lang-33].

All donor code of fixing a bug is rolled in a method, the patch unrolls the method but not invokes the method.

ACS: [Math-35];

ARJA: [Math-35].

The buggy variable is not replaced, its value is replaced with the correct one without other side effect.

Cardumen: [Chart-24];

FixMiner: [Chart-24].

The patch enumerates and negates the condition that is positively considered in the buggy code.

ACS: [Math_82], [Math_85].

The unnecessary code is not removed in the patch, such code will not be executed at the end or impact the execution of patched program.

ARJA: [Chart-12],[ Closure-115];

AVATAR: [Closure-115], [Lang-10];

GenProg-A: [Closure-115], [Closure-126];

RSRepairA: [Closure-115];

TBar: [Lang-10], [Math-65].

The patch checks the value of a boolean variable before other related boolean variables, and returns the related value if the condition is (not) satisfied. Developers however prefer to package the boolean variable with others and have a single return.

ACS: [Lang-24].

The APR-generated patch consists of more null checks than the corresponding developer's patch.

SimFix: [Lang-39].

The APR-generated patch considers unneeded check but such check is already considered in the context of non-buggy code.

ACS: [Math-5].

The patch fixes a partial code but passing all tests, and the patch code is identical to the partial code of developer's patch.

ACS: [Lang-7];

AVATAR: [Closure-2], [Lang-7];

jGenProg: [Closure-21], [Closure-22];

jKali: [Closure-21], [Closure-22], [Lang-7];

kPAR: [Closure-40], [Lang-7];

Nopol: [Lang-46];

RSRepair-A: [Closure-21], [Closure-22];

SimFix: [Closure-68], [Math-72];

TBar: [Lang-7].

The patch code is not compared with the same accurate value as the code of developer's patch.

ACS： [Math-3]；

AVATAR: [Closure-38];

TBar: [Math-11].

The patch uses a field but not a getter method that returns the field.

AVATAR: [Lang-57].

The APR-generated patch does not remove the buggy code, but make it un-actionable (it will never be executed).

DynaMoth: [Math-50];

jKali: [Closure-126], [Lang-10], [Math-50];

KaliA: [Closure-115], [Closure-126], [Math-50];

SimFix: [Closure-115].