Couldn't fix QuixBugs lis bugs about pyggi HOT 13 OPEN

Hi :), thanks for reaching us.
https://github.com/agb94/pyggi-quixbugs
👆 This repo contains the script & results that I used for the experiment.
Maybe you can try with those scripts. Please let me know if there’s a problem with using it. Thanks!

from pyggi.

Hello, I tried to run your scripts. It just hangs at program.py exec_cmd() shlex.split(cmd) forever. Timeout seems not working.
from terminal:
tmp: ./.pyggi/tmp_variants/java_programs/1638933294 cmd: None

code:
def exec_cmd(self, cmd, timeout=15):
cwd = os.getcwd()
os.chdir(self.tmp_path)
print('tmp: ', self.tmp_path)
print('cmd: ', cmd)
print(shlex.split(cmd))
print('after')
sprocess = subprocess.Popen(
shlex.split(cmd),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# stdout, stderr = process.communicate()
print('after open')
try:
start = time.time()
stdout, stderr = sprocess.communicate(timeout=timeout)
end = time.time()
return (sprocess.returncode, stdout.decode("ascii"), stderr.decode("ascii"), end-start)
except subprocess.TimeoutExpired:
sprocess.kill()
return (None, None, None, None)
finally:
os.chdir(cwd)

from pyggi.

Then could you please try it with the recent version of pyggi?
The exec_cmd is slightly updated.

from pyggi.

OK, which branch I should use? I used master branch in https://github.com/coinse/pyggi
Thanks!

from pyggi.

I merged a PR into the master branch yesterday, so you can use the current master branch!

from pyggi.

I just pulled the latest version. But it's still hanging on Popen. I think shlex.split(cmd) is still causing this. Also, your latest version misses StatusCode
Thank you!

def exec_cmd(self, cmd, timeout=15):
cwd = os.getcwd()
os.chdir(self.tmp_path)
print('tmp path: ', self.tmp_path)
print('split: ', shlex.split(cmd))
sprocess = subprocess.Popen(
shlex.split(cmd),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
preexec_fn=os.setsid)
print('after popen')
try:
start = time.time()
stdout, stderr = sprocess.communicate(timeout=timeout)
end = time.time()
return (sprocess.returncode, stdout.decode("ascii"), stderr.decode("ascii"), end-start)
except subprocess.TimeoutExpired:
os.killpg(os.getpgid(sprocess.pid), signal.SIGKILL)
_, _ = sprocess.communicate()
return (None, None, None, None)
finally:
os.chdir(cwd)

from pyggi.

Oh, right. The StatusCode class is changed into RunResult class (21b1fbc), so the pyggi-quixbug script should be revised. Sorry for the inconvenience.

• Have you separately run the shlex.split(cmd)? (I guess so because your code contains print('split: ', shlex.split(cmd))). Does it produce output?
• I think this problem may be related to the known Popen timeout issues, not the shlex, because the Popen's implementation is different across the system OS and the python version...: https://stackoverflow.com/questions/50338959/subprocess-run-isnt-timing-out-even-though-timeout-is-specified.

from pyggi.

print('split: ', shlex.split(cmd)) doesn't print anything. It just hangs there. So I think the problem might be shlex.split

from pyggi.

Are you using Windows?

from pyggi.

No I am using Mac

from pyggi.

that's weird... what is the value of cmd? I'll test it on my machine.

from pyggi.

I just printed cmd. It's None. Because self.test_command is None in run().

from pyggi.

I just found that the quixbug script on GitHub is completely not compatible with the current version of pyggi.
I may have the latest version of the script, but I guess it is on my previous laptop.
Inquixbugs-java.py, I defined the custom class MyLineProgram and override the run method, which is now evaluate_patch in pyggi/base/program.py.

• MyLineProgram/run in quixbugs-java.py

    def run(self, timeout=10):
        try_compile = self.exec_cmd(COMPILE_COMMAND.format(self.algo.upper()), timeout)
        if try_compile.stderr:
            result = self.__class__.Result(
                            status_code=StatusCode.PARSE_ERROR,
                            elapsed_time=try_compile.elapsed_time,
                            stdout=try_compile.stdout,
                            stderr=try_compile.stderr)
        else:
            failing = 0
            elapsed_time = 0
            for test in self.tests:
                test_in, test_out = test
                algo_input = ['"'+json.dumps(arg)+'"' for arg in copy.deepcopy(test_in)]
                try_test = self.exec_cmd(TEST_COMMAND.format(self.algo, ' '.join(algo_input)), timeout)
                #print(TEST_COMMAND.format(self.algo, ' '.join(algo_input)))
                #print(try_test.stdout.strip(), str(test_out))
                if try_test.stdout is None or try_test.stdout.rstrip() != str(test_out):
                    failing += 1
                elapsed_time += try_test.elapsed_time if try_test.elapsed_time else timeout
            result = self.__class__.Result(
                status_code=StatusCode.NORMAL,
                elapsed_time=elapsed_time,
                stdout=str(failing),
                stderr='')
        return result

• pyggi/base/program.py

 def evaluate_patch(self, patch, timeout=15):
        # apply + run
        self.apply(patch)
        return_code, stdout, stderr, elapsed_time = self.exec_cmd(self.test_command, timeout)
        if return_code is None: # timeout
            return RunResult('TIMEOUT')
        else:
            result = RunResult('SUCCESS', None)
            self.compute_fitness(result, return_code, stdout, stderr, elapsed_time)
            assert not (result.status == 'SUCCESS' and result.fitness is None)
            return result

To use the current pyggi version, Instead of overriding run, you should override evaluate_patch when defining the MyLineProgram,

ex)

class MyLineProgram(LineProgram):
   def evaluate_patch(self, patch, timeout=15):
          # apply the patch
          self.apply(patch)
         # try compile
         return_code, stdout, stderr, elapsed_time  = self.exec_cmd(COMPILE_COMMAND.format(self.algo.upper()), timeout)
         if stderr:
              return RunResult('Compile Error')
        else:
            failing = 0
            elapsed_time = 0
            for test in self.tests: # self.tests is defined in the __main__ function
                test_in, test_out = test
                algo_input = ['"'+json.dumps(arg)+'"' for arg in copy.deepcopy(test_in)]
                return_code, stdout, stderr, elapsed_time = self.exec_cmd(TEST_COMMAND.format(self.algo, ' '.join(algo_input)), timeout)
                if stdout is None or stdout.rstrip() != str(test_out):
                    failing += 1
                elapsed_time += elapsed_time if elapsed_time else timeout
        return RunResult('<something>', fitness=failing)

I haven't run this code, so it may have some minor issues, but I think you'll get the idea.

from pyggi.