Possible bug in Corr.dump(datatype="json.gz") about pyerrors HOT 4 CLOSED

Good point, when I wrote the json export routine for the correlators I did not have this case in mind and assumed that None values would only occur at the very beginning or end of the correlator (through the padding). Accounting for this special case could be a bit more complicated. Is this an essential feature for you? Otherwise we could just check for these special cases and issue an Exception.

from pyerrors.

Honestly, this is pretty important to me. The module is set up in such a way, that a value could be None for any number of reasons. It would be pretty weird, if you could do everything with a correlator that has missing values, but you could not save it.

from pyerrors.

I don't see a very simple solution to this problem. The relevant line of code can be found in https://github.com/fjosw/pyerrors/blob/develop/pyerrors/input/json.py#L167
For now I cut out the paddings and convert the remainder into a three dimensional numpy array which I then write to the file. If the array contains None entries the conversion to the numpy array does not work properly (numpy issues an explicit warning "Creating an ndarray from ragged nested sequences"). I see two options:

• We could split the correlator at the None entries and write the individual blocks as an array. The problem with this approach is that reassembling the block might be non trivial and the information about the blocks has to be saved somewhere.
• Alternatively we could replace the None entries by numpy.arrays of the correct size containing a "special" value (something like np.inf? This has to play well with the json parser). When reading the file one could then reconstruct these blocks as None.

from pyerrors.

Yes, i was just about to suggest the second approach.
I would use np.nan since it is a bit clearer. Having some matrices of nan will probably be compressed anyway. So i do not see a huge downside to doing it this way.

from pyerrors.