Automatic windowing method fails for gapped and irregular chains about pyerrors HOT 4 CLOSED

Indeed, your solution to determine gapsize is much better than mine! I'll adapt my code to open a PR.

from pyerrors.

I have found some cases with very large autocorrelation, where the gap is correctly identified but the windowing does not properly work - more investigation is needed.

from pyerrors.

The following code snippet seems to resolve the problems:

                else:
                    # Standard automatic windowing procedure
                    tau = self.S[e_name] / np.log((2 * self.e_n_tauint[e_name][gapsize::gapsize] + 1) / (2 * self.e_n_tauint[e_name][gapsize::gapsize] - 1))
                    g_w = np.exp(- np.arange(1, len(tau) + 1) / tau) - tau / np.sqrt(np.arange(1, len(tau) + 1) * e_N)
                    for n in range(1, w_max):
                        if n < w_max // 2 - 2:
                            _compute_drho(gapsize * n + gapsize)
                        if g_w[n - 1] < 0 or n >= w_max - 1:
                            n *= gapsize
                            self.e_tauint[e_name] = self.e_n_tauint[e_name][n] * (1 + (2 * n + 1) / e_N) / (1 + 1 / e_N)  # Bias correction hep-lat/0306017 eq. (49)
                            self.e_dtauint[e_name] = self.e_n_dtauint[e_name][n]
                            self.e_dvalue[e_name] = np.sqrt(2 * self.e_tauint[e_name] * e_gamma[e_name][0] * (1 + 1 / e_N) / e_N)
                            self.e_ddvalue[e_name] = self.e_dvalue[e_name] * np.sqrt((n + 0.5) / e_N)
                            self.e_windowsize[e_name] = n
                            break

All plots and, most importantly, the resulting error seem to be fine.

The dection of the gapsize is quite stable when the criterion in the if clause is a bit relaxed, say, to 1e-10 . The fft should be the best method to detect the gapsize.

There is the general question of the unit of the window size and the autocorrelation time. Currently, we have a factor of gapsize, when we deal with regular and gapped chains. In the proposed fix, the unit for these numbers is one unit in terms of idl for the window size for gapped and irregular chains. The autocorrelation time however contains the implicit factor of gapsize. We could think of changing the output for regular chains (i.e., those with a range as idl) to have consistent estimates in terms of MDU or to print the gapsize, either determined with the new code or from the ranges

from pyerrors.

I was aware of this issue but never took the time to really dig into it. I like your solution for the modification of the windowing procedure. Maybe I am overlooking something but there might be a simpler and more stable way of obtaining the gap size directly from the idl, for example

gapsize = np.min(np.diff(self.idl[e_name]))

I see the problem related to the output but I am not sure about the best solution. Do you maybe want to open a PR with a proposal and then we can run a few test cases on the new code?

from pyerrors.