Crashes reading a large file about jupyterlab-h5web HOT 6 CLOSED

Hello Jon, thanks for trying the extension and for the feedback !

Axis selection

I wish I could figure out how to select x and y axes for a plot?

Well, h5web is a "dumb" viewer: it will only display visualizations corresponding to the content of the file. It is not meant to be a visualization tool.
The only way to select x and y axes for a plot would be to use a NXData group with an attribute axesas the NeXus standard is supported by h5web.

Reasons of the crash when reading a large dataset

This is due to a limitation in the Line visualisation: we have a feature (auto-scale off) where the axis limits are set to the limits of the full dataset. As a consequence, when using the Line, h5web fetches the full dataset. In this case, I believe this is around 256 GB (😱) making the whole Jupyter server crash. I still need to investigate the exact reason.

Note that the Heatmap suffers not from this limitation: it only fetches the slice. This is why the first display of /1.1/measurement/eiger works. It is the switch to the 1D dataset /1.1/measurement/fpico6 that make h5web switch to the Line visualisation when coming back to /1.1/measurement/eiger.

What is next, then?

Is there a way to use hdf5 slice operations (maybe combined with fast histograms) so you only hold in memory what is going to be displayed on the screen (e.g. maximum data is a 2D image)?

It would indeed make sense to fetch only the slice even for a Line visualization. The Auto-scale feature puts a large limitation for large datasets and we need to work somehow around that.

We have an issue in h5web where we track our ideas and improvements to fetch large datasets: silx-kit/h5web#616. The discussion about the auto-scale will surely continue there and any implementation fixing the crash will be mentioned there.

In the mean time, use the Heatmap ? 😅

from jupyterlab-h5web.

@jonwright thanks for the +ve feedback.

@loichuder thanks for the explanations. It seems like we are missing a tool to do flexible viewing of Nexus files i.e. selecting what to display against what. AM I right to say that users have to build their own tool with a mixture of h5py and matplotlib for now? Does bragy address this?

from jupyterlab-h5web.

This is outside of the scope of Braggy, for sure. It's always possible to make a new GUI, but note that a solution to this problem is to generate a NeXus-compliant HDF5 file with external links to the relevant datasets, and then open this file in H5Web. Obviously not as practical as a GUI, but we could easily provide Python utilities to make generating this sort of file a breeze (perhaps these utilities already exist, even).

from jupyterlab-h5web.

There is already some helpers to save NXData: nexusformat or silx.io.nxdata.save_NXdata.

Otherwise since this runs in a notebook, using matplotlib or any other plot library is probably best suited for tailored plots if not saved as NXData.

BTW, in silx view, there is a feature to create "virtual NXData" by dragging and dropping datasets as signal and axes, but to me it is a bit complex since one needs to know about NeXus to use it.

from jupyterlab-h5web.

Following on the crash issue, we have something in the works to solve it: silx-kit/h5web#616 (comment)

I will close this once this is shipped in a jupyterlab-h5web release.

from jupyterlab-h5web.

silx-kit/h5web#616 (comment) was integrated in v0.1.0 that is now deployed in jupyter-slurm.

from jupyterlab-h5web.