uploading slow with tons of files about hfs HOT 11 CLOSED

its very slow with over 5000 files
and it affects on LAN too

from hfs.

opening multiple tabs slows down the other uploads

from hfs.

i tried to reproduce this problem, but maybe my computer is too fast (m1 pro).
I even tried artificially slowing down with Chrome

still i could get over 200MB/s on a single large file, even with over 5k files in queue.
Can you still replicate the problem?
The easiest way to try is to "pick folder" and select a big one. You can do it multiple times, doesn't matter the files are duplicate.

I suspect the problem is not with the list in itself, but with its update on the screen. So, my idea is to not show all the items, but cut at 500.

from hfs.

my pc is very fast aswell could be even faster than yours am not sure (i5-13400f)
the issue is with uploading multiple small files it takes an age to get to the next file that completes instantly due to it being so small
so 90% of the time uploading is spent waiting for the next file to start uploading while theres multiple in queue

from hfs.

i tried to reproduce this problem, but maybe my computer is too fast (m1 pro). I even tried artificially slowing down with Chrome

still i could get over 200MB/s on a single large file, even with over 5k files in queue. Can you still replicate the problem? The easiest way to try is to "pick folder" and select a big one. You can do it multiple times, doesn't matter the files are duplicate.

I suspect the problem is not with the list in itself, but with its update on the screen. So, my idea is to not show all the items, but cut at 500.

https://hdun.org/random/hfs_upload_issues.mp4

from hfs.

ah yes, you got files that are exceptionally small, 10 bytes each.
There's no decent way at the moment to get good performances.
Of course you may zip them before uploading.
I think I can add an "unzip" command without much effort. It could be available if you have upload permission.
Anyway, I also devised an optimization that should improve things already.
Let's see how faster you can get
0.52-beta6.2

from hfs.

ok, i made a test myself.
we are passing from a performance index of 7 to 44 with the version above, but then i've managed to improve my technique and reach 75.
To my surprise, the main slowing factor was repainting of the list.

from hfs.

from hfs.

something like that

from hfs.

from hfs.

that's roughly what i did

from hfs.