new function savejson_fastfile to speed up exporting directly to files about jsonlab HOT 3 CLOSED

@spichardo thanks for pointing out the efficiency issue from string concatenations.

As I mentioned in the File Exchange discussions, we have been focusing on optimizing loadjson because it was the (much) slower one between the two. We haven't paid much attention to savejson because most data I tested were simple data structures.

I was able to download and run your test cases, and I confirm that the provided savejson_fastfile script is about 2x faster than the latest savejson in the git (45 s vs 92 s on an i7 Ubuntu box with ssd). However, the implementation in savejson_fastfile has limitations. It requires users to provide a file and writes to disk with file IO, and it can not return JSON strings directly in memory.

Since the key issue for the performance difference is the string concatenation, I was able to update savejson to use a more efficient way to combine strings. Here is my latest commit:

5ae51f1

In this update, I replaced all sprintf() based string concatenations to string cell operations, and flattened the string cell once at the end of each sub-function. Such change essentially eliminates the overhead due to string concatenations (except for matdata2json because we have user specified printing format).

With this updated savejson, I was able to cut the run time for your benchmark from 92 seconds to 39 seconds, ~15% faster than savejson_fastfile. The outputs are identical (except for a pair of empty square brackets due to other prior changes).

Can you try out the new savejson and let me know if this works for you?

Again, thank you for helping making savejson faster!

from jsonlab.

Excellent, I just tested it, and now definitively using cell concatenation and then flattening gives a huge boost. Funny because I should have thought on that first myself since I used cell arrays precisely to facilitate dynamic growing with minimal impact to performance. Since myself I was purely focused in output files, that's the reason I thought in simply redirecting the output to files.

In my conditions with a double Xeon proc, the updated savejson takes 48.3s vs 53 s with savejson_fastfile, more importantly, when doubling the cell array to 2000, the penalty remains linear: 96 s with savejson vs 104.9s with savejson_fastfile

Thanks a lot for taking a look on this

Cheers

from jsonlab.

terrific, thanks for confirming.

I am closing this issue for now. I am pretty sure there are places in savejson can be further accelerated. Patches and suggestions are always welcome!

from jsonlab.