usnistgov / jsfive Goto Github PK

I need to serve on-demand cropped 2D tiles from a 3D image volume for an application I am developing. Instead of saving each cropped 2D image as a PNG, I thought about directly cutting them from the corresponding h5 file.

I get a single, consecutive array using your library to read the data (1D array of length z*y*x).
Is this the intended behavior meant by the sentence "It is only for reading HDF5 files as an ArrayBuffer representation of the file.", or is it possible to maintain a 3D array structure when loading?

I've included the script, but I still am facing this issue. Please help me solve this soon. Thank you.
`<script src="https://cdn.jsdelivr.net/gh/usnistgov/jsfive@master/dist/hdf5.js"></script>

<script>
    angular.module('myApp', [])

        .controller("myCtrl", function ($scope, $http) {
            fetch("Link")
                .then(function (response) {
                    console.log(response);
                    return response.arrayBuffer()
                })
                .then(function (buffer) {
                    console.log(buffer);
                    var f = new hdf5.File(buffer, filename);
                    // do something with f;
                    // let g = f.get('group');
                    // let d = f.get('group/dataset');
                    // let v = d.value;
                    // let a = d.attrs;
                    console.log(f);
                });
        });
</script>`

HI @bmaranville,

You worked very helpfully with my colleague previously regarding ONT's VBZ Compression (#26).

We noticed an error when working with very large 500MB+ vbz compressed fast5 files.

It seems that while reading the file, the offset is out of bounds. It essentially jumps from a nominal magnitude of 10^6 to 10^19 which is greater than the byteLength.

The error occurs in the core.js file here:

Do you know what could be causing this?

Part of our project at Oxford Nanopore Tech, we need to read fast5 files that are VBZ compressed. We would like to extend the JSFive Project to support VBZ compression.

Any support/guidance would be appreciated please, thank you.

Hi, awesome project. Forgive me if this has been answered or if I've missed something.

I'm interested in reading objects (root group and specific named datasets) from large hdf5 files remotely, that is by URL, without reading the entire file into an array buffer. First is this possible out-of-the box? I assume it is not having spent some hours experimenting and looking at source, but perhaps I have missed something.

Thanks for your attention.

First off, thank you for your work on this project. It has been a great help to me!

Secondly, are there any plans to implement the h5 compound datatype? I tried, then was thrown the message "not yet implemented!"

The length of the "length of link name" field from https://support.hdfgroup.org/HDF5/doc/H5.format.html#LinkMessage is not being calculated correctly by the JS library. This appears to be a typo when turning the python library this is based off of into a JS one, as the python library does this calculation correctly.

The value of the length field should be derived from the flags such that:

• flags & 3 = 0 means length of link name is 1 byte
• flags & 3 = 1 means length of link name is 2 bytes
• flags & 3 = 2 means length of link name is 4 bytes
• flags & 3 = 3 means length of link name is 8 bytes

This can be calculated by taking the 2^X of flags & 3 however the JS library does X^2 instead, leading to incorrect results when parsing the name of the file, and subsequent operations within the link message

Using version 0.3.3 in a site created with create-react-app (https://github.com/facebook/create-react-app) causes a compilation error, as in this simple example:

1. $ npx create-react-app test-jsfive
2. $ cd test-jsfive
3. $ npm install [email protected]
4. Add to src/App.js:

	import * as hdf5 from 'jsfive';

5. $ npm start

Failed to compile.
./node_modules/jsfive/esm/high-level.js 30:8
Module parse failed: Unexpected token (30:8)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|    * @member {Array<string>} _keys;
|    */
>   parent;
|   file;
|   name;

Version 0.2.3 does work, however (i.e., replace step 3 with $ npm install [email protected] and there is no error at step 5).

I have a trivial test program to use jsfive. Here is the code:

async function helloJSFive(file) {

    const response = await fetch(file)
    const buffer = await response.arrayBuffer()
    const cndb = new hdf5.File(buffer, file.name)

    cndb.close()
}

jsfive throws and an error in the line:

new hdf5.File(buffer, file.name)

here is the error:

throw "unsupported superblock version: " + version_hint.toFixed();

Can someone enlighten me to why this is happening. The file I am trying to read is read correctly in the Python file reader I have that uses h5py

I am using jsfive in a React project created with CRA.

• react version is ^17.0 (17.0.2 is installed)
• react-scripts version is ^4.0 (4.0.3 is installed)

Everything was working fine until I upgraded jsfive to latest 0.3.8 version. I am currently having problems when using the library.

Here is what I have tried so far.

I was using import * as hdf5 from "jsfive/dist/index"; before and here is the compile error I receive after update (using index.js, using without index are also giving the same error):

./node_modules/jsfive/dist/index.js 5186:14
Module parse failed: Unexpected token (5186:14)
File was processed with these loaders:
 * ./node_modules/react-scripts/node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
| };
| var BTreeV1 = class extends AbstractBTree {
>   B_LINK_NODE = /* @__PURE__ */new Map([["signature", "4s"], ["node_type", "B"], ["node_level", "B"], ["entries_used", "H"], ["left_sibling", "Q"], ["right_sibling", "Q"]]);
|
|   _read_node_header(offset, node_level) {

I tried using import * as hdf5 from "jsfive"; as suggested in the latest release (and README), no compile error but opening a file gives the following error in the console:

Uncaught TypeError: jsfive__WEBPACK_IMPORTED_MODULE_1__.File is not a constructor
    onloadend HDF5File.js:66

Finally I try const hdf5 = import("jsfive"); as suggested in README (but without await), I am receiving the following in the console:

Uncaught TypeError: hdf5.File is not a constructor
    onloadend HDF5File.js:68
    HDF5File HDF5File.js:67
    React 5
    unstable_runWithPriority scheduler.development.js:468
    React 8
    unstable_runWithPriority scheduler.development.js:468
    React 15
    js index.js:7
    js main.chunk.js:3022
    Webpack 7

How can I import and use correctly?

package.json

{
  "name": "name",
  "description": "description",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "start": "CHOKIDAR_USEPOLLING=1 react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "dependencies": {
    "@fortawesome/fontawesome-svg-core": "^1.2",
    "@fortawesome/free-solid-svg-icons": "^5.15",
    "@fortawesome/react-fontawesome": "^0.1",
    "axios": "^0.25",
    "jsfive": "^0.3",
    "papaparse": "^5.3",
    "react": "^17.0",
    "react-alert": "^7.0",
    "react-alert-template-basic": "^1.0",
    "react-dom": "^17.0",
    "react-scripts": "4.0",
    "react-table": "^7.7",
    "web-vitals": "^2.1"
  },
  "devDependencies": {
    "@testing-library/jest-dom": "^5.16",
    "@testing-library/react": "^12.1",
    "@testing-library/user-event": "^13.5"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Hi, thanks for maintaining jsfive!

Quick question: how come this package is not available on NPM?
Are there any legal reasons for it to not be published?

Cheers & thanks again
Patrik

Hey, back in December there was a PR: #25 that has been merged into master.

In our project when we install jsfive directly from github master branch, it has the latest changes, it works fine for us after adding the babel plugin in our bundler.

However we noticed that the NPM registry may not be updated with the latest changes.

Thank you!

Are there any plans to make this available/work in Node.js?
I could only find two hdf5 packages for JavaScript; this one and https://github.com/HDF-NI/hdf5.node. However hdf5.node does not work on Windows currently.

I've recently upgraded to v0.3.9 from v0.3.8. Since then jsfive always gives me a type error when trying to create a new file:

TypeError: hdf5.File is not a constructor

That's the code that worked fine until v0.3.9:

import * as hdf5 from "jsfive"

export function deserializeRawResultBuffer(arrayBuffer: ArrayBuffer): Result {
    var hdf5File = new hdf5.File(arrayBuffer);
    var results = hdf5File.get('results');
    var scaling = results.attrs.scaling;
    if (scaling && scaling != 1) {
        return new FixedPointResult(new Uint16Array(results.value), scaling);
    } 
    else {
        return new Result(new Float32Array(results.value));
    }    
}

When installing latest jsfive, there's an issue with one of the imports inside the filters file.

./node_modules/jsfive/esm/filters.js
Module not found: Can't resolve '../node_modules/pako/dist/pako.esm.mjs'

RangeError: Offset is outside the bounds of the DataView
at DataView64.getUint8 (<anonymous>)
at Struct.unpack_from (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\core.js:145:28)
at _unpack_struct_from (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\core.js:4:24)
at BTreeV1RawDataChunks._read_node_header (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\btree.js:85:16)
at BTreeV1RawDataChunks._read_node (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\btree.js:161:21)
at BTreeV1RawDataChunks._read_root_node (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\btree.js:36:26)
at BTreeV1RawDataChunks.init (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\btree.js:17:10)
at new BTreeV1RawDataChunks (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\btree.js:155:10)
at DataObjects._get_chunked_data (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\dataobjects.js:725:23)
at DataObjects.get_data (E:\_d_a_t_e_n\Studium\Inf\Seminar mit Bachelorarbeit SE_10e\OpenCFS_experiments\read_hdf5\node_modules\jsfive\esm\dataobjects.js:633:19)

I don't understand the code immediately just by stepping through the code but maybe the problem comes from this code in dataobjects.js?!

var data_offset;
...
else if (version == 3) {
  var [dims, address] = struct.unpack_from(
    '<BQ', this.fh, property_offset);
  data_offset = property_offset + struct.calcsize('<BQ');
}

The address returned is '18446744073709552000'.
This becomes the offset in the constructor of the BTreeV1RawDataChunks in _get_chunked_data.
I loose track here, but maybe this offset is the problem?

I can send you the hdf5 file I try to read if you want. I have two files both created with the same software (OpenCFS). jsfive is working perfectly fine with one file. It sends the above errors with the other file.

I use jsfive version 0.3.13

I am trying to load an HDF5 file in node v14.17.5 and getting an error when creating a new hdf5.File - new hdf5.File(ab.buffer);

Sample file: https://water.weather.gov/precip/downloads/current/nws_precip_1hour_conus.nc

As far as I can tell in my debugging, it is processing the very start of the file. It's (erroneously?) expecting to see a 'FRHP' signature, but the signature is the standard HDF signature that should be at the start of the file.

As a side note, the assert function does not handle false very well, trying to call it as a function.

home/rtaylor/src/jsfive/dist/index.js:156
    thing();
    ^

TypeError: thing is not a function
    at assert (/home/rtaylor/src/jsfive/dist/index.js:156:5)
    at new FractalHeap (/home/rtaylor/src/jsfive/dist/index.js:9424:22)
    at DataObjects._iter_links_btree_v2 (/home/rtaylor/src/jsfive/dist/index.js:10331:16)
    at _iter_links_btree_v2.next (<anonymous>)
    at DataObjects._iter_link_from_link_info_msg (/home/rtaylor/src/jsfive/dist/index.js:10325:19)
    at _iter_link_from_link_info_msg.next (<anonymous>)
    at DataObjects.iter_links (/home/rtaylor/src/jsfive/dist/index.js:10224:21)
    at iter_links.next (<anonymous>)
    at Function.fromEntries (<anonymous>)
    at DataObjects.get_links (/home/rtaylor/src/jsfive/dist/index.js:10214:19)```

Hi!

I have been getting the following error when attempting to parse a specific file's attrs. Running latest jsfive-0.3.8 from relases.

It appears like something is off with my file (importing from another system outside of my control) as the library reports a string with length of 256 bytes where only "Second" is the value. I believe that the value field is null terminated and the library attempts to parse the full 256 bytes instead of up to the null, leading to a crash while trying to decode.

Is this expected behavior for such cases? I would expect that if there is a null in a string that is not escaped, the string would terminate vs filtering the null and continuing.

"URIError: URI malformed
    at decodeURIComponent (<anonymous>)
    at DataView64.getString (file:///C:/Users/claytonwhite/Documents/experiments/hdf5.js:207:14)
    at DataObjects._attr_value (file:///C:/Users/claytonwhite/Documents/experiments/hdf5.js:5407:34)
    at DataObjects.unpack_attribute (file:///C:/Users/claytonwhite/Documents/experiments/hdf5.js:5349:24)
    at DataObjects.get_attributes (file:///C:/Users/claytonwhite/Documents/experiments/hdf5.js:5279:34)
    at Group.get attrs [as attrs] (file:///C:/Users/claytonwhite/Documents/experiments/hdf5.js:6009:41)
    at FileReader.reader.onloadend (file:///C:/Users/claytonwhite/Documents/experiments/hd5viewer.html:204:42)"

"Secondà�hî����Ö&è�.1>�¥)Dáz�¿����ÿÿÿÿo���û�����®��	�¯���Ù»H��*;�òí����ÇU��°±����²�������ÇU��T®ï3����ÇU��Ö��î��8¯������áé���"

Relevant code here:

The following snippet:

const arrayBuffer = (await axios(temporaryDownloadUrl, {
        responseType: 'arraybuffer'
    })).data.buffer;
    const data = new hdf5.File(arrayBuffer);

Returns:

RangeError: The "ascii" encoding is not supported
RangeError[ERR_ENCODING_NOT_SUPPORTED]: The "ascii" encoding is not supported
at new NodeError(node: internal / errors: 393: 5)
at new TextDecoder(node: internal / encoding: 403: 15)
at DataObjects._decode_link_msg(/home/deb4693 / nodevenv / domains / xxx.xx / node_dev / 18 / lib / node_modules / jsfive / dist / cjs / index.js: 5552: 16)
at DataObjects._iter_links_btree_v2(/home/deb4693 / nodevenv / domains / xxx.xx / node_dev / 18 / lib / node_modules / jsfive / dist / cjs / index.js: 5586: 40)
at _iter_links_btree_v2.next( < anonymous > )
at DataObjects._iter_link_from_link_info_msg(/home/deb4693 / nodevenv / domains / xxx.xx / node_dev / 18 / lib / node_modules / jsfive / dist / cjs / index.js: 5571: 19)
at _iter_link_from_link_info_msg.next( < anonymous > )
at DataObjects.iter_links(/home/deb4693 / nodevenv / domains / xxx.xx / node_dev / 18 / lib / node_modules / jsfive / dist / cjs / index.js: 5499: 21)
at iter_links.next( < anonymous > )
at Function.fromEntries( < anonymous > )

Environment:
Node.js 18.9.1

Cause:
Many production Node.js hosting environments are built with small-icu and not the full icu data, so the module can't use encodings other than utf-8, utf-16, utf-16be in TextDecoder(encoding).

Using the latest version 0.3.8 in a site created with create-react-app (https://github.com/facebook/create-react-app) works for npm start but fails for npm test, as shown in this simple example:

1. $ npx create-react-app test-jsfive
2. $ cd test-jsfive
3. $ npm install [email protected]
4. Add to src/App.js:

   /* eslint-disable-next-line no-unused-vars */
   import * as hdf5 from 'jsfive';
   

5. $ npm start
   Works.
6. $ npm test

   SyntaxError: Unexpected token 'export'
   
     3 |
     4 | /* eslint-disable-next-line no-unused-vars */
   > 5 | import * as hdf5 from 'jsfive';
       | ^
     6 |
     7 | function App() {
     8 |   return (
   
     at Runtime.createScriptFromCode (node_modules/jest-runtime/build/index.js:1728:14)
     at Object.<anonymous> (src/App.js:5:1)
   

Reverting to version 0.3.5 and using the older import syntax solves the problem:

1. $ npm uninstall jsfive
2. $ npm install [email protected]
3. Change the import line in src/App.js to:

   import * as hdf5 from 'jsfive/dist';
   

4. npm start
   Works.
5. npm test
   Works; the one test passes, and the warning that "ReactDOM.render is no longer supported in React 18" seems unrelated.

Hello! Thank you for all the work on this package. We are exciting for a browser only HDF5 reader.

Would you be willing to bundle into an es5 compatible script for use in currently active es5 applications? I understand the push towards the es6 syntax, but I would guess there is still a use case to be able to load this using a simple script tag.

We grabbed your latest and had success making a bundle with parcel, but this could be done with any compiler. Would be great to have this alongside the releases!