v0.2 Save trials object to resume on next run about tforce_btc_trader HOT 2 OPEN

For initial testing:
Instead of waiting for 20 runs to tell if this works, if you're testing it set max_evals = 5 or even max_evals = 1 to make sure that it loads and writes the pickle correctly. After that can set it back to 20 if there are no issues pickling the trials object. To reset your trials, simply delete the trial.pickle file then the next run will start fresh. If you want to start over but still have the ability to restore it back later, just back up the trial.pickle file to a different directory. If you add any additional hypers to search, delete the trials.pickle file or you'll get a key error when hyperopt attempts to resume trials.

Reminder, this is for the hyperopt implementation in v0.2 and does not apply to v0.1

A note on the hyperopt trials: If there is a hyper combo that causes the run to crash (example is a network depth that is too high for the amount of ram that you have resulting in the process getting killed), when hypersearch crashes it will NOT write the trials to pickle since it only saves at the end of max_evals. When the hypersearch is run again, it's very likely that the hyper combo that caused your process to get killed will be tried again at some point. I haven't run v0.2 yet, this is just from experience with Keras and sequential regression where a very particular combination of parameters returned loss as nan that fmin could not parse. We shouldn't have the nan problem since we are maximizing returns over holding (from what it looks like in v0.2) which is a little less finicky than a loss function.

from tforce_btc_trader.

I've been able to test this portion today and can report that pickling the trials is working with the code in the first post. Once the max_evals is reached, it saves the trials to disk. Next time you start hypersearch it resumes where it left off last. It takes a significant amount of runs for hyperopt to really get to refining things. It's effective but computationally very expensive. With how many hypers are realistically needed for RL, and how long it takes per run, saving the progress is a must. With 100 runs it's probably still throwing darts at a map, just standing a little closer.

@lefnire I'd like to get this SQL ready so it can fall in line with the rest of the framework but am not real SQL savvy, going to take me a bit to hack through that and there are some more pressing issues. It's your call if you want this in master as-is or hold until it's compatible with running multiple hypersearches in parallel. I'll get the fork ready and submit a PR with a reference to this issue to save some time if you decide to go with it.

from tforce_btc_trader.