Rate limiting causes chunks not to be uploaded about web HOT 9 CLOSED

Good catch!
It seems like there are 3 possible solutions to this:

1. Limiting the upload speed.
2. Using the retry_after parameter after the blocked request.
3. Using the x-ratelimit-* headers before the blocked request.

If there's no penalty on passing the rate limit too often (which I will research about), I'm opting for the second option, as it will be a cleaner and simpler solution imo. What do you think?

Either way, please share your upload speed so we can get a general sense of the rate limit

Thanks!

from web.

Upload speed should be around 125mbps to 150mbps max, but is usually a bit lower than that. Speed test link: https://www.speedtest.net/result/13941504387
After looking those two things up, it does seem like retry_after is a good option. Not really sure though.

from web.

Hi again!
I ended up fixing the issue using the third option (x-ratelimit-* headers) over at #16

However, my upload speed isn't high enough to properly test this.
Could you clone the branch and test it locally (As explained in #10) and comment there if it fixed your issue?
You can also review the changes if you want to😄

from web.

Oddly, it was working quite well, but then at around 50% of the file (it's about 11.4 GB) it started doing this again.
Checking the discord channel also shows that chunks are being skipped
(Also, I would review them but I know next to nothing about coding lol)

Link to the file I used is here: https://placedata.reddit.com/data/canvas-history/2022_place_canvas_history.csv.gzip (it's just the 2022 history for the Reddit Place thing)

from web.

Thank you! I really appreciate your help with this.
I'll invesitgate this and will probably ask you to test a new version.

I suspect the fix might not be perfect because they x-ratelimit-retry-after only displays seconds and not milliseconds.

It is weird though, because I would expect it to round up. Just to make sure, you cloned the bugfix/upload-ratelimit branch, right?

from web.

i downloaded the zip from github, but i tried cloning it just now using git (i made sure it was the right branch) and its doing the same thing

from web.

Thanks again, I'll look into it and let you know!

from web.

Just wanted to add, I'm also getting this from trying to download a file.

from web.

The PR #16 for this seems to fix this issue for me.

from web.