Structuring with large (>~180MB) Arbin CSV files hangs about beep HOT 4 CLOSED

I just let it run for a while, and you're right @patrickherring-TRI it does eventually process after not appearing to make any progress for a while. I was able to process all of the files with an average time of 15 min / file. Interestingly, the RAM utilization (at least in the beginning, while appearing not to progress) is quite low. Not sure what the slowdown is. The files of size < 160 MB process in around 1 min or so.

Thanks for looking into this!

from beep.

@jkoeller Hi, thanks for reporting this!
I tried out a couple of the files that you showed above and it looks like the structuring succeeds although it does take longer than I would expect. I will look into what process is taking up so much CPU time.

One thing that might be happening is an out of memory issue causing your kernel to crash. Generally, the structuring process takes ~5x the memory footprint of the raw file (pandas being what it is). So a 300MB file might require ~2GB of RAM to complete successfully.

After structuring, the structured file has a considerably smaller memory footprint and is much easier to work with.

from beep.

@patrickherring-TRI some of these memory issues might be fixed by using in place operations for the dataframes. For example, in RawCyclerRun.get_interpolated_data() the dataframe seems to be copied in memory several times

from beep.

This delay appears to be due to the initial data loading process which for large files can be resource intensive. New pandas functions might be able to speed up this process, but a better solution will be the larger refactor of the structuring module. Closing this issue for now since it is system dependent.

from beep.