🐞 Fix HyperOpt Results not reproducing about monigomani HOT 63 CLOSED

Good news! 🎉
My last batch of comparability tests with -j 1 and the same --random-state also reproduced!
They contain the late epoch results we where looking for to fully confirm that this issue lies in the parallelization while HyperOpting.

from monigomani.

This thread is better than a season on Netflix

from monigomani.

Finally!! 🥳 (Tests ran with 12 parallel workers)

Keep in mind that this is still on commit ee756f8 though.
Next I will apply the patches to my latest local MGM code to see if it will keep on reproducing there, but I am feeling positive on this one! 🙂

from monigomani.

Took ages, but seems like we're finally reproduction bug free! 🤩
Results on my latest (local) version of MGM:

Tonight I will continue on issue #38 to see if my new HyperOptLosses also are reproducing now with the bug in the main framework fixed.

After those have been confirmed to be reproducing results as intended, then it's finally, commit, push & merge time for the heap of pending changes that still need to reach the MGM repo 🙂

Progress train finally on the move again though! 🥳

from monigomani.

a very simple test to confirm if custom_info is leaking info between epochs:

    custom_info = []

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        # filter on one pair to avoid cluttering of output
        if metadata['pair'] == 'XRP/USDT':
            if self.custom_info:
                print(self.custom_info)
            else:
                self.custom_info.append(np.random.normal())
                print("not there")

Last night I implemented this test in MGM as following:

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        # Keep this call to populate the conditions responsible for the weights of your signals
        # filter on one pair to avoid cluttering of output
        dataframe = self._populate_trend('buy', dataframe, metadata)

        if metadata['pair'] == 'MATIC/USDT':
            if 'MATIC/USDT' in self.custom_info['unclogger_cooldown_pairs']:
                self.mgm_logger('custom', 'xmatthias test', f'custom_info found:\n{self.custom_info}')
            else:
                self.mgm_logger('custom', 'xmatthias test', f'MATIC/USDT not found in custom_info:\n{self.custom_info}')

        return dataframe

I manually stored some of the custom_info printouts, parsed them to a human readable .json format and came to some interesting conclusions!
HyperOptResults150_Pt1-07-08-2021-individual-epochs-jobs-1-MATIC-USDT-populate_buy_trend-custom_info-contents.zip

If we compare the lengths & contents of these files, then we can clearly see that the custom_info object is not being reset in-between epochs/backtests while HyperOpting! (We can see the custom_info object growing in size) 😮

We can also notice that the custom_info garbage collection seems to work correctly during the backtests,
but that the data left in there at the end of the backtesting timerange persists in between epochs while it should not!

I believe with these conclusions we finally found the cause of this nasty bug! 🥳
It would also explain why results start to be off only near the end of the HyperOpt (Because by then the custom_info object is filled up with a lot of unwanted data)

As I am writing this up I'm already running a new HyperOpt test with a potential fix.
In my test, during BackTesting/HyperOpting it will check if custom_info needs to be reset to it's initial (empty) state when calling _populate_trend()!

Fingers crossed that my following set of reproduction tests will finally reproduce! 🤞

from monigomani.

Code pushed, finally closing this one! 🥳

You can find the commit with the fix here: f97e373

Here is one last set of re-assuring results for good measure 😁

from monigomani.

So far 122232b is looking clear, but I noticed it only caught early epochs so I will queue another 2 tests on another random state:

from monigomani.

Commit c45fac5 has also been confirmed to be bug-free on my end! 🎉
This aligns with the findings by @slipx06 on the newer 2183fae commit posted a moment ago!

Reducing our commit search window on the user_data/strategies folder:

2721d03 (06/07/2021 22:32 - bugged)
...
ee756f8 (06/07/2021 21:44 - checking this commit next)
...
2183fae (04/07/2021 19:32 - bug free)

Only 5 commits in between now

I'm guessing the bug will have snuk in on either ee756f8 or c94eae6 🤔
Let's do a few more HO's till we have truly isolated to a single commit 🙂
Then I will start debugging the code in that commit further until I figure out the reason so I can fix it.

from monigomani.

Commit ee756f8 confirmed to be bugged!..

That means we have successfully concluded that the bug introduced itself on this commit (ee756f8)! 🎉

Conclusion of commit research on the user_data/strategies folder:

• ee756f8 (06/07/2021 21:44 - bugged)
• 2183fae (04/07/2021 19:32 - bug free)
  

Thank you so much @slipx06 for supporting me in in tracking this bug down this far down in the commits! ❤️
Now I'll take over from here, or if you feel capable, feel free to join in to try & solve this together with me! 🙂

from monigomani.

I assume that if "unclogger_enabled" was set to false in the config, it might be the reason it could be missed.

This is indeed appears to be the case and also why the issue didn't occur for @3ylo since he had the unclogger disabled on his recent signal tests

from monigomani.

When I was debugging I noticed following output:

==== ↓ Bugged Output ↓ ==============================================================================================
DEBUG - Open Trade Unclogger - Unclogging losing trade for pair (ENJ/USDT), Current Time: 2021-06-12 06:05:00+00:00...
DEBUG - Unclogger Cooldown - Minimal cooldown start time currently needed: 2021-05-16 23:49:00
DEBUG - Unclogger Cooldown - Start of cooldown period for pair (ENJ/USDT): 2021-06-12 06:05:00

From this we can learn that Freqtrade doesn't BackTest custom_sell & confirm_trade_entry in harmony, but rather,
run all the custom_sell loops, ... , then run all the confirm_trade_entry loops.
Which causes above bugged log output since MGM expected different Freqtrade behavior.

Currently running tests on a re-write that take this BackTest behavior into account! 🤞

from monigomani.

if you use different -j settings, the sequence will almost certainly be different (everything else would be coincidence) and potentially also results - as we "ask" the optimizer for fewer points per round - and the next round "more" data is known (results of that round's epochs) - so the direction the optimization might be going might be different (even if only slightly).

That's however neither an issue nor a bug - it's simply how this works and is expected / intended this way (if you consider it a bug, it's a bug in scikit-optimize - but once you think about it - it's fully expected as the calculations will be based on different "current" results.

from monigomani.

Tested 52b2784

from monigomani.

I have identical results with 80dae59. Will increase epochs to 1500 now

from monigomani.

This is proving a bit harder to track down. If you look at the tests from 80dae59 everything looks fine except for the one test with 1500 epochs that I aborted.

from monigomani.

Next step is to run some tests against a1b0aab

from monigomani.

Results from a1b0aab confirm the issue still present

Should I perform another run with a large epoch and same random state or move to 122232b?

from monigomani.

Commented out the code in question.

Did 3 comparability tests on it, of which all failed..

So this piece of code is not responsible for the results not reproducing

Confirmed with my testing as well

from monigomani.

Sorry I meant v0.12 of MGM (main branch)

from monigomani.

Trying to test that version but getting the following error. Any ideas

2021-07-23 09:53:23,975 - freqtrade - ERROR - Fatal exception!
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3081, in get_loc
   return self._engine.get_loc(casted_key)
 File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
 File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
 File "pandas/_libs/hashtable_class_helper.pxi", line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item
 File "pandas/_libs/hashtable_class_helper.pxi", line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'adx'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 431, in _process_worker
   r = call_item()
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 285, in __call__
   return self.fn(*self.args, **self.kwargs)
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 595, in __call__
   return self.func(*args, **kwargs)
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/parallel.py", line 262, in __call__
   return [func(*args, **kwargs)
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/parallel.py", line 262, in <listcomp>
   return [func(*args, **kwargs)
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/externals/loky/cloudpickle_wrapper.py", line 38, in __call__
   return self._obj(*args, **kwargs)
 File "/home/home/freqtrade/freqtrade/optimize/hyperopt.py", line 300, in generate_optimizer
   bt_results = self.backtesting.backtest(
 File "/home/home/freqtrade/freqtrade/optimize/backtesting.py", line 393, in backtest
   data: Dict = self._get_ohlcv_as_lists(processed)
 File "/home/home/freqtrade/freqtrade/optimize/backtesting.py", line 201, in _get_ohlcv_as_lists
   self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
 File "/home/home/freqtrade/freqtrade/optimize/hyperopt_auto.py", line 31, in populate_buy_trend
   return self.strategy.populate_buy_trend(dataframe, metadata)
 File "/home/home/freqtrade_new/freqtrade/user_data/strategies/MoniGoManiHyperStrategy.py", line 190, in populate_buy_trend
   dataframe = self._populate_trend('buy', dataframe, metadata)
 File "/home/home/freqtrade/user_data/strategies/MasterMoniGoManiHyperStrategy.py", line 1211, in _populate_trend
   self._add_signal(signal_name, space, dataframe, condition_func(dataframe))
 File "/home/home/freqtrade_new/freqtrade/user_data/strategies/MoniGoManiHyperStrategy.py", line 17, in <lambda>
   'adx_strong_up': lambda df: (df['adx'] > 25),
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/pandas/core/frame.py", line 3024, in __getitem__
   indexer = self.columns.get_loc(key)
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3083, in get_loc
   raise KeyError(key) from err
KeyError: 'adx'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/home/freqtrade/freqtrade/main.py", line 37, in main
   return_code = args['func'](args)
 File "/home/home/freqtrade/freqtrade/commands/optimize_commands.py", line 87, in start_hyperopt
   hyperopt.start()
 File "/home/home/freqtrade/freqtrade/optimize/hyperopt.py", line 455, in start
   f_val = self.run_optimizer_parallel(parallel, asked, i)
 File "/home/home/freqtrade/freqtrade/optimize/hyperopt.py", line 368, in run_optimizer_parallel
   return parallel(delayed(
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/parallel.py", line 1054, in __call__
   self.retrieve()
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/parallel.py", line 933, in retrieve
   self._output.extend(job.get(timeout=self.timeout))
 File "/home/home/freqtrade/.env/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
   return future.result(timeout=timeout)
 File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
   return self.__get_result()
 File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
   raise self._exception
KeyError: 'adx'

from monigomani.

Ok I've managed to fix it. Running the tests now. PEBKAC :)

from monigomani.

+1

from monigomani.

9e0daf0 looks OK

from monigomani.

122232b is looking clear once more, still not an epoch in the 600-1000 range as I would have liked for my tests though, I'll try the random state used by @slipx06 next on the same commit, I really want to be sure on this.

@slipx06 good work! 🦾 Now we can finally start narrowing this down further. I would suggest next you test one of the commits done on the user_data/strategies folder between 15/07/2021 19:22 - 25/06/2021 23:47

That means we currently are looking at following commit window on the user_data/strategies folder:

a1b0aab (15/07/2021 19:22 - bugged)
...
122232b (25/06/2021 23:47 - looking bug free, testing more to be sure)
...
9e0daf0 (19/06/2021 18:00 - bug free)

from monigomani.

I'm busy checking 122232b and a1b0aab again

from monigomani.

Happy to announce that 122232b has been confirmed as bug-free! 🎉

Reducing our commit search window on the user_data/strategies folder:

a1b0aab (15/07/2021 19:22 - bugged)
...
2721d03 (06/07/2021 22:32 - next commit I plan on testing, since it's +- in the middle of our search window)
...
122232b (25/06/2021 23:47 - bug free)

from monigomani.

I ran another test on a1b0aab just to double check and still seeing discrepancies.

Will skip 122232b since you confirmed it twice now

from monigomani.

Commit 2721d03 has been confirmed to be bugged!..

Reducing our commit search window on the user_data/strategies folder:

2721d03 (06/07/2021 22:32 - bugged)
...
3318628 (01/07/2021 18:48 - checking this commit next)
...
122232b (25/06/2021 23:47 - bug free)

I'm happy to see we are finally rapidly closing in on it! 🥳

from monigomani.

I was busy testing 983469b and it confirms your results above. No issues. Moving onto 2183fae

from monigomani.

Ok commit 2183fae looks good. This is from July 4th

Only 5 commits in between now

from monigomani.

Great. Glad we hunted it down :)

I assume that if "unclogger_enabled" was set to false in the config, it might be the reason it could be missed.

My coding is not great but I'm more than happy to run some more testing when needed

from monigomani.

I temporarily removed all the unclogger cooldown code on my end & results started reproducing again.

However it's still unclear what causes this bug, it just confirms that we don't need to look further back down the commit history.
Next I will add a bare confirm_trade_entry that returns true at all times to check if results will keep on reproducing with that function implemented.

from monigomani.

After a talk with @xmatthias we believe that the bug could be caused due to a parallelization issue within the joblib library (A dependency of Freqtrade).

I just finished 2 comparability tests done with --jobs 1.
Sadly it didn't find any late epochs (Around epoch 700 - 800 would be ideal since we know that the issue always occurs there),
so I will soon do another set of comparability tests on another --random-state (To try and get results with later epochs).

However on the bright side, the tests I just finished did reproduce! 🥳

from monigomani.

I just finished 2 comparability tests done with --jobs 1.
Sadly it didn't find any late epochs (Around epoch 700 - 800 would be ideal since we know that the issue always occurs there),
so I will soon do another set of comparability tests on another --random-state (To try and get results with later epochs).

Will test on my end to confirm this as well. With only 1 job the ETA is 6hours. 😲

from monigomani.

from monigomani.

a very simple test to confirm if custom_info is leaking info between epochs:

    custom_info = []

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        # filter on one pair to avoid cluttering of output
        if metadata['pair'] == 'XRP/USDT':
            if self.custom_info:
                print(self.custom_info)
            else:
                self.custom_info.append(np.random.normal())
                print("not there")

if there is information leakage in custom_info - it'd apply after the first epochs ...

as it's however printing not there all the time - it's not a parallelization issue - but must be an issue in the strategy.

As you only seem to reprocduce it with "all" cpu's - maybe try with half of them.
Maxing out the CPU over a long period of time might cause the cpu / memory to act strangely (and maybe you're experiencing some bit-flips down the road).

from monigomani.

Running the same test using the SwingHighToSkyHyperStrategy and we see the same behaviour.

from monigomani.

Current Findings of BugHunting (check newer findings below)

@slipx06 posted in the Discord server that identical results where found on commit ce9a3b9:

So at that point in time the bug did not introduce itself yet in the software!

Further I also tested commit e779564 without the -> list appended to def informative_pairs(self): in MoniGoManiHyperStrategy.py. However this did not fix the issue:

Commit window where bug introduced itself

In user_data/strategies

(Most likely that one of these changes caused it)

In the whole MoniGoMani repository

from monigomani.

In the whole MoniGoMani repository

I just went through all of the code in this commit window and I cannot believe the the issue introduced itself here!
All that happened in this commit window was adding @topscoder's GitHub Actions/WorkFlows and the removal of some Python Caches that got accidentally pushed (Both of these do not influence the working of MGM at all)

@slipx06 posted in the Discord server that identical results where found on commit ce9a3b9:

However starting to question our assumptions pulled on these test results by @slipx06.
We should re-test commit ce9a3b9 using another random state then on the test already done. Since in this test we only have a single best epoch found near the beginning of the HyperOpt.

I noticed that the results often still align near the beginning of a HyperOpt, so we would want a test done on ce9a3b9 with more best results found, at a later time in the HyperOpt.
Starting to believe this commit will be bugged after all and that the bug introduced itself at an earlier commit done on the contents of the user_data/strategies folder

from monigomani.

I have identical results with 80dae59. Will increase epochs to 1500 now

Thank you for your quick tests! (Currently taking me roughly 5hrs to test a random state twice..)
This time it's later in the HO but still only 1 best result found.
I think cranking it up to 1000 or 1200 epochs should already be enough.
(Normally we tend to not go over 1000 to stay away from overfitting, but for these testing purposes it should be ok)

from monigomani.

Current Findings of BugHunting

Commit ce9a3b9 has been confirmed to be bugged after all:

Meaning all commits in the window we where first looking at will all contain the bug.
We need to look further back since the bug will have introduced itself at an earlier commit done on the contents of the user_data/strategies folder

Commit window where bug introduced itself

@3ylo mentioned on Discord that MGM Commit a1b0aab in combination with Freqtrade commit 3503fdb4 is free from the bug.

With what we learned from our previous findings I propose following new commit window to search through:
a1b0aab (Bug Free) - ce9a3b9 (Bugged)

However if we compare with the corresponding commit windows I would find it hard to believe that commit a1b0aab is Bug Free, because no changes happened to the MGM Framework & Strategy files in that timespan.

This would lead me to believe that the bug might have been introduced even earlier then commit a1b0aab.
Because all that got committed in that timespan where @topscoder's GitHub Actions/Workflows and mgm-hurry updates/enhancements, which should be completely separate from the MGM Framework & Strategy files/working.

In user_data/strategies

In the whole MoniGoMani repository

from monigomani.

Next step is to run some tests against a1b0aab

It's "good" to see that your findings are aligning with mine! 🦾
Fingers crossed for a1b0aab 🤞 But I fear we might have to dive deeper back after all

from monigomani.

Confirmed that other strategies do not suffer from this issue, so the bug is definitely in MGMs codebase & not Freqtrade's:

@slipx06 is also using the "vanilla" freqtrade commands and not the mgm-hurry commands so the bug is also not related to the mgm-hurry code

from monigomani.

Should I perform another run with a large epoch and same random state or move to 122232b?

Yes I would recommend that epoch and 1000-1200 epochs 🙂

from monigomani.

Commented out the code in question.

Did 3 comparability tests on it, of which all failed..

So this piece of code is not responsible for the results not reproducing

from monigomani.

Next up rolling back to commit: 122232b
And will fire up 2 identical comparability tests on it.

Feel free to already dive a bit further back in the code.
At this point in time we must first look for a commit where the bug did not introduce itself in the software yet.

Once we found the commit window, where the bug was not in the code yet <=> when we first registered it being in the code (For now this is commit: a1b0aab), then we can start to reflect upon it, to check if I can spot a feature in it that I would find suspicious.
Or we could start using git-bisect to help us track down the bug more easily in the commit window.

We will have to be careful to use the corresponding go-to commands + mgm-config.json for older MGM commits! (Freqtrade version too to be honest, but I regret not documenting the Freqtrade commits to use in the past.. Only recently started doing that on MGM's development branch)

from monigomani.

Have we confirmed that this bug is not present in the stable branch?

from monigomani.

Have we confirmed that this bug is not present in the stable branch?

Not explicitly no @slipx06, however I did already confirm that the issue is not present on other strategies using Freqtrade development (commit: 3503fdb4).

Confirmed that other strategies do not suffer from this issue, so the bug is definitely in MGMs codebase & not Freqtrade's:

Feel free to try it out though! Would save us a lot of unneeded searching if it would prove to be a Freqtrade issue after all.

from monigomani.

Sorry I meant v0.12 of MGM (main branch)

Ah no problem, no this has not been tested yet.
Just checked the dates and the v0.12.0 release (commit: 9e0daf0) was done 1 week earlier before commit 122232b (Which I'll test shortly).
So testing v0.12.0 - main sounds like a good plan to me! 👍

from monigomani.

OK. I'll test 9e0daf0

from monigomani.

Trying to test that version but getting the following error. Any ideas

 File "/home/home/freqtrade_new/freqtrade/user_data/strategies/MoniGoManiHyperStrategy.py", line 190, in populate_buy_trend
   dataframe = self._populate_trend('buy', dataframe, metadata)
 File "/home/home/freqtrade/user_data/strategies/MasterMoniGoManiHyperStrategy.py", line 1211, in _populate_trend
   self._add_signal(signal_name, space, dataframe, condition_func(dataframe))
 File "/home/home/freqtrade_new/freqtrade/user_data/strategies/MoniGoManiHyperStrategy.py", line 17, in <lambda>
   'adx_strong_up': lambda df: (df['adx'] > 25),
   raise KeyError(key) from err
KeyError: 'adx'

Cleaned up the stacktrace for the important pieces.
Seems like the adx indicator is missing (Back then it was used for core trend detection in the master file and as a weighted signal in the strategy file. Nowadays we changed the core trend detection indicators used in the master file so adx is only used as a weighted signal in the strategy file).
Just checked the code in commit 9e0daf0 and in the _populate_core_trend() function I can see that the adx row should indeed be getting populated in the dataframe See Line 295 here 🤔
You rolled back both the master, the strategy and the config files correct?

from monigomani.

Performed a clean build of freqtrade and new install of 9e0daf0 . Trying to figure out why I'm getting the error below

2021-07-23 10:48:42,764 - freqtrade.resolvers.iresolver - WARNING - Could not import /home/home/freqtrade_test/user_data/strategies/MoniGoManiHyperStrategy.py due to 'No module named 'MasterMoniGoManiHyperStrategy''
2021-07-23 10:48:42,765 - freqtrade - ERROR - Impossible to load Strategy 'MoniGoManiHyperStrategy'. This class does not exist or contains Python code errors.

from monigomani.

Performed a clean build of freqtrade and new install of 9e0daf0 . Trying to figure out why I'm getting the error below

2021-07-23 10:48:42,764 - freqtrade.resolvers.iresolver - WARNING - Could not import /home/home/freqtrade_test/user_data/strategies/MoniGoManiHyperStrategy.py due to 'No module named 'MasterMoniGoManiHyperStrategy''
2021-07-23 10:48:42,765 - freqtrade - ERROR - Impossible to load Strategy 'MoniGoManiHyperStrategy'. This class does not exist or contains Python code errors.

Which Freqtrade version are you using? I'm on Freqtrade commit 3503fdb4 and besides some warnings about Freqtrade stuff in the older mgm-config.json soon becoming deprecated, MoniGoMani commit 122232b appears to have started it's comparability test HyperOpt without issues on my end 😮

You are not renaming the MGM master & strat files right? That can cause issues since they are also hard coded in some places in the code.

from monigomani.

@slipx06 I would pick your next commit between 2721d03 (06/07/2021 22:32 - still testing, but already spotting misalignments!)
and 122232b (25/06/2021 23:47 - confirmed to be bug free)

from monigomani.

Commit 3318628 has been confirmed to be bug-free! 🎉

Reducing our commit search window on the user_data/strategies folder:

2721d03 (06/07/2021 22:32 - bugged)
...
c45fac5 (03/07/2021 22:45 - checking this commit next)
...
3318628 (01/07/2021 18:48 - bug free)

Halved once more! 🥳

from monigomani.

I'm testing c94eae6 now

from monigomani.

Bare confirm_trade_entry described above is reproducing so that's not the issue.

from monigomani.

My tests are taking much longer than anticipated. Over 15 hours for just 1 run so far. Plan on running the same --random-state with multiple jobs to confirm the issue. Might only have this on Monday at this speed.

from monigomani.

Thank you @slipx06, this confirms once again that the issue lies in the parallelization while HyperOpting!
However I'm trying to reproduce this issue on a lightweight strategy and haven't been able to do that so far.

After my discussion with @bzed on Discord yesterday, I believe it might only occur when the dataframe reaches a certain size.
That or a bug in the MGM code after all, however this seems very unlikely due to --jobs 1 reproducing multiple times by now!

from monigomani.

I have 32 cores. Ran some tests using the same random state but different number of workers and got completely different results

from monigomani.

Running the same test using the SwingHighToSkyHyperStrategy and we see the same behaviour.

I noticed that too, however I believe that different results due to different parallel workers is another separate issue/bug in Freqtrade itself.

Because on my end comparability tests with the same --jobs & --random-state lead to identical results on SwingHighToSky.

However if I compared multiple SwingHighToSky results done on the same --random-state but different amount of --jobs then the results where different too! 🤔

from monigomani.

That's however neither an issue nor a bug - it's simply how this works and is expected / intended this way (if you consider it a bug, it's a bug in scikit-optimize - but once you think about it - it's fully expected as the calculations will be based on different "current" results.

Nice! I'm glad you are already aware of this & that it's expected behavior instead of another bug! 😌

from monigomani.

Just finished a long debug test with --jobs 1 and custom logging.
All custom_info kept on resetting properly throughout the whole test on my end! 🥳

Delaying my reproduction tests.
Next up I will first do testing with multiple parallel workers to check if custom_info will keep on behaving properly there too 👀

from monigomani.