discrepancy between simulation trial variability in HNN vs hnn-core about hnn-core HOT 10 CLOSED

Also note that the input histograms are drastically different.

from hnn-core.

sorry I didn't look at the zip file. Might be better to share via gist, so I can copy paste. You can have the param file as a dictionary in the Python script to avoid loading it.

could it be that sync_evinput is ignored somewhere? The input histograms being different is certainly weird.

from hnn-core.

@jasmainak try this.

could it be that sync_evinput is ignored somewhere? The input histograms being different is certainly weird.

Possibly, though I don't understand why some trials would still have a small amount of jitter s.t. entire clumps of trials are still exactly the same. Note that sync_evinput = True for this parameter set, however, the problem persist when sync_evinput = False.

from hnn-core.

I also tried running the same simulation with 25 trials using the JoblibBackend and got the following image. @blakecaldwell isn't the JoblibBackend supposed to provide jitter across trials?

from hnn-core.

it could be that using seeds to jitter the trial may not be working in this case for whatever reason. Will need to investigate. Should we set up another pair programming session @rythorpe ?

By the way, I do think we should move away from "let's compare to HNN as ground truth" philosophy at some point. Because HNN does not have any unit tests either, so there isn't really a strong notion of correctness in that sense. What we have in HNN is outputs which make sense for some scenarios.

I have seen software where 2 bugs cancel each other and the output makes sense for some inputs but not for others. We need a stronger definition of what's a bug, otherwise it will hamper development of the software.

from hnn-core.

it could be that using seeds to jitter the trial may not be working in this case for whatever reason. Will need to investigate. Should we set up another pair programming session @rythorpe ?

@jasmainak sure, let's give it a shot!

By the way, I do think we should move away from "let's compare to HNN as ground truth" philosophy at some point. Because HNN does not have any unit tests either, so there isn't really a strong notion of correctness in that sense. What we have in HNN is outputs which make sense for some scenarios.

I have seen software where 2 bugs cancel each other and the output makes sense for some inputs but not for others. We need a stronger definition of what's a bug, otherwise it will hamper development of the software.

We can discuss this more in-person, but as a contributor, I completely agree that HNN shouldn't be used as ground truth. As a user with a previously-created, HNN-generated param file, I just want to identify what is wrong with my param file so that I can run similar (i.e., not necessarily identical) simulations in hnn-core.

I think this issue represents a case where a (potential) bug has emerged that prevents hnn-core from accomplishing a basic use-case that HNN currently implements correctly (i.e. providing uniform stochasticity across simulation trials). Most likely, the discrepancy outlined in this issue is not a bug in hnn-core but rather a difference in how HNN and hnn-core reference some elusive parameter. However, if a bug does indeed exist that has eluded us despite hnn-core's tests, the functionality of HNN can provide us with a useful sanity check.

from hnn-core.

By the way, I do think we should move away from "let's compare to HNN as ground truth" philosophy at some point. Because HNN does not have any unit tests either, so there isn't really a strong notion of correctness in that sense. What we have in HNN is outputs which make sense for some scenarios.

At some point, hnn-core will be considered "ground truth". More accurately, the "current state of the model", It is important to systematically understand where differences are coming from, whether a bug in new code or a fix in new code. Both have been found in developing hnn-core.

from hnn-core.

@rythorpe hnn-core master in my experience has never provided variable results for different trials, no matter the backend. There is a bug in hnn-core. I forgot exactly where, maybe the seeds for different trials.

If we can reproduce HNN seeding in the first pull request, then we can at least run sanity checks for new code. We can then have another issue/PR for more logical seeding of random variables in each trial.

from hnn-core.

@rythorpe hnn-core master in my experience has never provided variable results for different trials, no matter the backend. There is a bug in hnn-core. I forgot exactly where, maybe the seeds for different trials.

@blakecaldwell but try this:

$ git checkout 015124c36ba0632391a2532a1166c201e1153841
$ python examples/plot_simulate_evoked.py

it did have randomness before ...

from hnn-core.

@jasmainak, indeed, but there is still a problem with the seeds where more than one trial use the same seed (and have identical results). Described in more detail in #113

from hnn-core.