Data Loading/Storage about thermo HOT 1 CLOSED

Hi Daniel,
It is kind of you to have done all of this research for the project. At the time I didn't look at it but Pandas did some changes to their CSV parser and I took another look today.

It is important to realize that speed is a combination of lots of factors, and while I have no doubt the HD5 format is very efficient, it does not appear that Pandas's interface to it is as performant as reading CSV files.

import pandas as pd
def load_old():
    CRC_organic_data = pd.read_csv('/tmp/Physical Constants of Organic Compounds.csv', sep='\t', index_col=0)
%timeit load_old()

18.9 ms ± 329 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

CRC_organic_data = pd.read_csv('/tmp/Physical Constants of Organic Compounds.csv', sep='\t', index_col=0)

CRC_organic_data.to_hdf('/tmp/example.h5', 'my_hdf_table', mode='w', format='table', complib='blosc:zstd', complevel=9 )#
%timeit pd.read_hdf('/tmp/example.h5', key='my_hdf_table')

CRC_organic_data.to_hdf('/tmp/example.h5', 'my_hdf_table', mode='w', format='table')#
%timeit pd.read_hdf('/tmp/example.h5', key='my_hdf_table')

CRC_organic_data.to_hdf('/tmp/example.h5', 'my_hdf_table', mode='w', format='fixed', complib='blosc:zstd', complevel=9 )#
%timeit pd.read_hdf('/tmp/example.h5', key='my_hdf_table')

CRC_organic_data.to_hdf('/tmp/example.h5', 'my_hdf_table', mode='w', format='fixed')#
%timeit pd.read_hdf('/tmp/example.h5', key='my_hdf_table')

34.7 ms ± 874 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
32 ms ± 1.43 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
26.6 ms ± 3.82 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
24.2 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

No matter the settings it seems to be considerably slower. The CSVs also have the benefit of having track-able changes in git.
I am closing the issue because I am cleaning up and it seems certain to me that csv-style files read through Pandas is good enough.

Sincerely,
Caleb

from thermo.