Add hdf5 about array_storage_benchmark HOT 4 OPEN

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

from array_storage_benchmark.

will h5 file have problem with CPU memory？I encounter some problem which I think is caused by the memory consumption of loading h5file，especially when using multi-workers

from array_storage_benchmark.

@PingHGao I think this might not be the best place to get help with your problem sorry

from array_storage_benchmark.

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

Hi, is this related to chunk caching of HDF5? If you're writing or reading the same content(chunk) multiple times, real I/O will be executed once.

If you're using h5py, you can set the rdcc_* arguments when opening a file to disable the cache effect. https://docs.h5py.org/en/stable/high/file.html#chunk-cache

from array_storage_benchmark.