Cannot open certain stitched .nd2 files from Elements about nd2reader HOT 15 OPEN

This should now be fixed in the latest master version with PR #40 (thanks again @ggirelli !)
If anyone could test this with their images, that would be appreciated.

from nd2reader.

Hey @rbnvrw, mine is the sample_data from:
https://downloads.openmicroscopy.org/images/ND2/karl/

from nd2reader.

Amazing! Now I can read my stitched nd2 files! Thanks a lot guys @rbnvrw @ggirelli ! Please bump the version number to signify this fix.

from nd2reader.

@ggirelli could you test this again with the latest fix from #50?

from nd2reader.

@ggirelli congratulations on submitting your thesis! do you have time to test this now? :-)

from nd2reader.

Hi @rbnvrw , thanks and sorry, it's been crazy a crazy period before/after the defense :)
I will take care of these in the next couple of weeks!

from nd2reader.

Same problem for me. Any updates?

As an alternative, pims_nd2 handles the same dataset without problems.

from nd2reader.

Hi Ruben (@rbnvrw ),
I'm having a similar issue with a stitch image:

ValueError: cannot reshape array of size 1038565 into shape (1019,1019)

metadata:

{'height': 1019, 'width': 1019, 'date': datetime.datetime(2017, 6, 6, 11, 15, 6), 'fields_of_view': range(0, 1), 'frames': [0], 'z_levels': range(0, 21), 'z_coordinates': None, 'total_images_per_channel': 21, 'channels': ['CSU far red RNA', 'CSU red RNA', 'CSU green RNA', 'CSU blue RNA', 'CSU BF'], 'pixel_microns': 1.0153102411107793, 'num_frames': 1, 'experiment': {'description': 'ND Acquisition', 'loops': []}, 'events': []}

Dimension:

Dimensions: XY(1) x λ(5) x Z(21)

What I found in

def _get_raw_image_data(self, image_group_number, channel_offset, height, width):
[...]
image_group_data[image_data_start::number_of_true_channels]

the slicing is good for the first 5(channels) x 1019 pixels then there is an extra 0 that is added to the data that need to be removed, I added a very crude:

unwanted = []
        for i in range(5099, 5192828, 5095):
            unwanted.append(i)
        
        for ele in unwanted:
            del image_group_data[ele]

5099 is 1019x5+4(image_data_start), 5192828 is the length of image_group_data and 5095 is 1019x5. Obviously it was just me trying to figure out what was wrong.
I wonder if the XY(1) is Dimensions is an indication that something is wrong, but, unfortunately, I don't have more stitched images to try.

from nd2reader.

Thank you for the addition Cedric. @bioimage-analysis @ColinDTaylor @aaristov could you maybe all add a link to a sample ND2 file that I can download?
I'd really like to solve this, but I don't have any files I can try to analyze. Thanks! 👍

from nd2reader.

The fix seems to break things for 3D multi-channel images... I am looking into it.

from nd2reader.

@ggirelli OK thank you. please keep us posted. Does it happen for images with (x,y,z,c) channels only or also (x,y,z,t) or (x,y,c,t) images for example?

from nd2reader.

@rbnvrw sorry, I am back after a short break after submitting my thesis :)
I'll test this later this week and post here about it!

from nd2reader.

@rbnvrw better late than ever, I just tested it on my files. All good on my end 😄 🎉 🎆

from nd2reader.

Hi guys!

So I hate to ruin the good mood but I am running into a similar issue.

I'll copy-paste it here it for convenience and it's completely fine for me to share the file (36.6 MB) if you can help me. I saw you might want the metadata from NIS Viewer, I'll see if I can locate it.

Interestingly it is only one of the two channels that has the issue (so I can use images[0] as normal but iterating or images[1] will trigger the ValueError).

Some nd2 images has images seem to miss one data point, making it impossible to reshape their array which also makes the file non-iterable.

Notably 18105583 == 18105584 - 1, where 18105584 == 3892 * 4651 so it would seem one data point is missing.

NIS-viewer can open the file without issues.

Error

ValueError: cannot reshape array of size 18105583 into shape (3892,4651)

Code to reproduce

#! /bin/python3

from nd2reader import ND2Reader

INPUT = "/Users/tobiasfrick/PycharmProjects/demos/test-data/multiple-channels.nd2"


def iterate():
    with ND2Reader(INPUT) as images:
        for _ in images:
            pass


if __name__ == '__main__':
    iterate()

Versions

% python --version
Python 3.8.10
% conda list | grep nd2reader
nd2reader                 3.3.0              pyhd8ed1ab_0    conda-forge

Traceback

python main.py
Traceback (most recent call last):
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 277, in _get_raw_image_data
    image_data = np.reshape(image_group_data[image_data_start::number_of_true_channels], (height, width))
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: cannot reshape array of size 18105583 into shape (3892,4652)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 18, in <module>
    main()
  File "main.py", line 13, in main
    for i in images:
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 227, in <genexpr>
    return (self._get(i) for i in self.indices)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 207, in _get
    return self._ancestor[key]
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 188, in __getitem__
    return self._get(indices)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/pims/base_frames.py", line 98, in __getitem__
    return self.get_frame(key)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/pims/base_frames.py", line 592, in get_frame
    result = self._get_frame_wrapped(**coords)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/reader.py", line 88, in get_frame_2D
    return self._parser.get_image_by_attributes(t, v, c, z, y, x)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 103, in get_image_by_attributes
    timestamp, raw_image_data = self._get_raw_image_data(image_group_number, channel,
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 280, in _get_raw_image_data
    image_data = np.reshape(image_group_data[image_data_start::number_of_true_channels], (height, new_width))
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: cannot reshape array of size 18105583 into shape (3892,4651)

from nd2reader.

Update

I'm getting the same error for any image I have taken where I am using more than one channel. So possibly this merits a separate issue as I can reproduce it even when I have non-stitched images.

For every image I test on from my most recent dataset, channel=0 always works whereas channel=1 never works (for example in a syntax like this image = images.get_frame_2D(c=channel)).

from nd2reader.