once just before calling read_index:
https://github.com/micro-manager/NDTiffStorage/blob/main/python/ndtiff/nd_tiff_current.py#L187

and once inside read_index:
https://github.com/micro-manager/NDTiffStorage/blob/main/python/ndtiff/nd_tiff_current.py#L556

please let me know if i'm missing something trivial

Imaging data generated using pycromanager can be opened with other readers within the TIFF ecosystem, although there are some restrictions outside of the pycromanager.Dataset route.

In the following, I am using the TIFF file generated by democam.py in micro-manager/pycro-manager#76 as a reference:

1. ImageJ itself does not recognize acquisition axes, i.e. the output stack shown is flat (instead of having - in this specific case - 2 sliders as in the original acquisition window)
2. Reading tags using tifffile reveals that there is a superfluous axis 'channel' within 'ImagrStorageAxesPositions' / 'StorageSuperChannelName'; the order of axes is also not preserved (likely due to specifying a 'standard' axis name - I'm using Python 3.8, so dict should preserve order).
3. The Micro-Manager Image file stack header appears to be incompletely written (see comment in cgohlke/tifffile#23); while I generated the file in question, I cannot verify this myself.
4. As far as I understand there doesn't appear to be an option for compressed storage (this is not a portability issue per se, but would be nice)

I am unfortunately not sufficiently familiar with Java/TIFF details to tackle this (my background is C++/Python and a few others, and I don't have the bandwidth to go beyond). It also does not appear to be an issue that is specific to an individual use case.

Dask arrays created by the ndtiff library cannot be pickled. Pickling is important for multiprocessing applications.

For example,

dataset = Dataset(\path\to\ndtiff\dataset)
data_dask_array = dataset.as_array()
pickle.dumps(data_dask_array)

first throws a AttributeError: Can't pickle local object 'NDTiffDataset.as_array.<locals>.read_one_image', which is fixed by #108. It then throws TypeError: cannot pickle '_thread.RLock' object, which I don't think I'll be able to fix myself.

A workaround may be to open a new dask array in every worker rather than pickling the opened dask array for distribution to workers.

Hey @henrypinkard I would like to open a ndtiff Dataset that is currently being acquired in a separate process for viewing with napari. I need to do that because I am running two Acquisitions in parallel and would like to view the data from both. Do you see a problem with opening a current dataset in a different process? Would the builtin thread lock protect from read/write conflicts?

The change from axis indices being integers to allowing all objects breaks code working with NDTiffStorage. It makes writing (backward compatible) code to work with NDTiffStorage difficult and awkward. Would it not be much easier to stick to integers as axis indices, and provide an (optional) dictionary mapping indices to Objects (or highly preferable, just Strings) in summary metadata?

At the very least, according to Semver this should be a major version increase, i.e. 4.0 rather than 3.2.

Dataset.read_image().shape returns (Y, X) dimensions, while Dataset.as_array().shape[-2:] returns (X, Y) dimensions

I am using pycromager to acquire many very large datasets. I have a lot of very large ndtiff Dataset files where I want to keep most of the data but I would like to remove an axis (ex: channel 1, or z:0 ) because it does not contain anything valuable. I like the file format of ndtiff so I would like to keep it rather then using another package like tifffile to write the desired arrays to other files. How can I write a new dataset file in python including only the axes I care about?

From a packager viewpoint it would be great if you add git tags to the commits used for releasing a version on PyPI. For example, I need the test files corresponding to the released sdist.

@henrypinkard ... Acquisitionevents in pycro-manager create TIFF files, which, however, appear to be flat (i.e. they aren't ND Hyperstacks). Is there a way to preserve axes defined in events?

Further, none of the metadata shown in the micro-magellan window appears to be saved to TIFF.

PS: The code I'm using is

import pycromanager as pm

if __name__ == '__main__':

    exposures = [.01, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]
    with pm.Acquisition(directory='.', name='darkframes') as acq:
        events = []
        for rep in range(50):
            for idx, exposure in enumerate(exposures):
                evt = {'axes': {'time': 0, 'repetition': rep, 'exposure': idx}, 'exposure': exposure}
                events.append(evt)

        acq.acquire(events)

From: micro-manager/pycro-manager#172 (comment)

Roughly 1,600 KB each file, which would correspond to ~2 images.

_000 is 4,186,782 KB and _776 is 3,299,682 KB.

_777 is only 290,140 KB - but I'm assuming that is because it didn't need to fill the TIFF container for the last images in the acquisition.

Likely due to some error in calculating how much space to reserve at the end of files

Hello,

I faced an unexpected behavior in the function as_array when using the stitched=True option. The returned dask array seems to be in the wrong shape. When converting it to a numpy array however the shape assumes the right dimensions.

I have tow-channel images of dimensions 2016x852px, which are arranged in two columns and eight rows (no overlap).

Experienced behavior:

d = data.as_array(stitched=True, z=0)
print(d)
>>> a=np.asarray(d)
print(a.shape)
>>> dask.array<read_one_image, shape=(2, 2016, 852), dtype=uint16, chunksize=(1, 2016, 852), chunktype=numpy.ndarray>
a=np.asarray(d)
print(a.shape)
>>>(2, 6816, 4032)

Note that beside the fact that the dimensions are wrong the width and the height of the images are swapped: In the dask-array the columns come before the rows, whereas in the numpy array the rows come before the columns.

Expected behavior:

d = data.as_array(stitched=True, z=0)
print(d)
>>> a=np.asarray(d)
print(a.shape)
>>> dask.array<read_one_image, shape=(2, 6816, 4032), dtype=uint16, chunksize=(1, 6816, 4032), chunktype=numpy.ndarray>
a=np.asarray(d)
print(a.shape)
>>>(2, 6816, 4032)

Fix:

Changing the following rows

483    w = self.image_width if not stitched else self._tile_width
484    h = self.image_height if not stitched else self._tile_height

...

553    chunks += (w, h)

to

483    w = self.image_width if not stitched else self._tile_width*len(self.axes["column"])
484    h = self.image_height if not stitched else self._tile_height*len(self.axes["row"])

...

553    chunks += (h, w)

in nd_tiff_v2.py seems to solve this problem.

Suggestion:

This way however the chunks consists of the already stitched images. For very large stitched images it might be better, though, to have a single chunk for every acquired image, thus a dask array of properties
dask.array<read_one_image, shape=(2, 6816, 4032), dtype=uint16, chunksize=(1, 852, 2016), chunktype=numpy.ndarray>
This might be useful for some people.

The bug is related to the already fixed issue reported by ptbrown1729: micro-manager/pycro-manager#278

Otherwise very nice package! Thanks for developing this open source project!

Problem

If people are going to rely on pycromanager to load their datasets then it would be good to have 100% test coverage of the dataset code. Otherwise the potential cost of breakage is high.

Proposed Solution

add tests for dataset. This basically works by having some simple known test images embedded into the repo. and then you use the dataset loading code to load them and do some asserts that everythign is equal.

For example see how aicsimageio does it: https://github.com/AllenCellModeling/aicsimageio/tree/main/aicsimageio/tests/readers

Hi team, in working on the conda forge package for the latest release

conda-forge/ndtiff-feedstock#20

We noticed that the tarball is not including the requirements.txt file:
https://github.com/micro-manager/NDTiffStorage/blob/main/python/setup.py#L14

Which will result in a broken package when installing from a tarball

I am going to add a dummy requirements.txt as the rest of the conda dependencies will be provided by the recipe meta.yaml.

Using this sort of install procedure (relying on an external requirements.txt) file is not recommended.

On another note it seems the package is using sortedcontainers but it has not been declared as a dependency.
See:

Could you look into updating the missing dependencies and look into other ways of specifying then? For instance using a setup.cfg file as described here.

Cheers!

See micro-manager/pycro-manager#467

NDTiff v3.0 had a bug where in certain cases it would encode metadata in not i UTF-8 and maybe in UTF-16 or ISO-8859-1

fixed in v3.1

It might be a good addition to try to deserialize (in dataset.read_metadata()) with other encodings if it fails with JSON

Is it OK to change the signature of MultiResNDTiffAPI

void increaseMaxResolutionLevel(int newMaxResolutionLevel);

to:

Future<Integer> increaseMaxResolutionLevel(int newMaxResolutionLevel);

where Integer denotes the newMaxResolutionIndex?

This would make it possible to immediately display the new resolution image, rather than having to wait for a user action to update the UI. Currently, in Magellan in Explore mode, zooming out regularly leads to a completely black canvas that only gets filled in when the user moves the mouse. or otherwise forces the display to update.

While some limited usage examples are provided here, the NDTiffStorage format and API are largely undocumented, and module dependencies are not clear (is the format usable/readable without installing micro-manager?).

See also: #10

Mirroring the changes for 11 bit in #102