sfalkner / spysmac Goto Github PK

this is also relevant for pysmac/pysmac/init.py

it is not obvious what changed in the final incumbent in comparison to the default configuration.
we should highlight and sort the parameters based on whether their value changed or not.

can we support other objective functions such as solution quality/cost/regret/...?

On SAT, someone suggested that SpySMAC would make a nice framework for benchmarking in general. When implementing a new feature that introduces new parameters, SpySMAC could be used to tune it. In that case, it would be helpful to be able to compare not only to the default, but also to older versions. I guess, the feature they would like to see if compare multiple SpySMAC runs in one report. If one requires the instances to be identical, that should not be a huge problem to implement.

Users are really interested to have also an ablation analysis.
We could add the original ablation implementation or we start a new research project and develop a more efficient ablation (ask me for details).

Steps to recreate:

1. Use a clean Ubuntu
2. Follow mini example instructions
3. During execution of make

    Compiling: core/Solver.o
    In file included from /home/osboxes/SpySMAC/examples/minisat/utils/Options.h:30:0,
                     from /home/osboxes/SpySMAC/examples/minisat/core/Solver.h:27,
                     from /home/osboxes/SpySMAC/examples/minisat/core/Solver.cc:24:
    /home/osboxes/SpySMAC/examples/minisat/utils/ParseUtils.h:27:18: fatal error: zlib.h: No such file or directory
    compilation terminated.

Possible fix:

apt-get install zlib1g-dev

I get a new error:

15:14:48.114 [main] ERROR c.u.c.b.smac.executors.SMACExecutor - Message: File: /pfs/work3/workspace/scratch/fr_tl1023-spysmac-exp-0/gcthreads/./out/scenario/state-run1/java_obj_dump-v2-it0.obj already exists
15:14:48.114 [main] ERROR c.u.c.b.smac.executors.SMACExecutor - Exception:java.lang.IllegalStateException
15:14:48.115 [main] ERROR c.u.c.b.smac.executors.SMACExecutor - StackTrace:java.lang.IllegalStateException: File: /pfs/work3/workspace/scratch/fr_tl1023-spysmac-exp-0/gcthreads/./out/scenario/state-run1/java_obj_dump-v2-it0.obj already exists
        at ca.ubc.cs.beta.aeatk.state.legacy.LegacyStateSerializer.save(LegacyStateSerializer.java:215)
        at ca.ubc.cs.beta.smac.configurator.AbstractAlgorithmFramework.saveState(AbstractAlgorithmFramework.java:431)
        at ca.ubc.cs.beta.smac.configurator.AbstractAlgorithmFramework.saveState(AbstractAlgorithmFramework.java:395)
        at ca.ubc.cs.beta.smac.configurator.AbstractAlgorithmFramework.run(AbstractAlgorithmFramework.java:539)
        at ca.ubc.cs.beta.smac.executors.SMACExecutor.oldMain(SMACExecutor.java:140)
        at ca.ubc.cs.beta.smac.executors.SMACExecutor.main(SMACExecutor.java:104)

The error only popped up after I ran simultaneously the validation of the default.
It's also reproducible that it is only the first SMAC run (seed 1) that throws this error.
Therefore, I assume we have a race-condition between the "real" SMAC runs and the validation run of the default configuration.

Best,
Marius