Add custom delay inbetween requests (prevent ban) about lychee HOT 8 OPEN

I'd like a feature like rate-limit in muffet which applies across all requests.

I'm happy to offer a bounty of sorts of 100€ (payable via PayPal or SEPA) for whoever implements this, if multiple people work on it I'm happy to split the money.

from lychee.

There is no delay between the requests at the moment.
In fact, we run many requests concurrently; even to the same host.

We could add a --request-delay argument and in combination with --max-concurrency 1 it would resolve your issue, but it would just be a band-aid to the underlying problem, which is the rate limiting itself. You see, delays need to be tweaked by humans to find a sweet spot between throughput and stability. This can't be solved with a global delay.

Instead, my proposal is to add better rate-limiting support per website. I wrote https://github.com/mre/rate-limits a while ago and would like to integrate it into lychee. We would keep track of the current rate-limits for each host in a HashMap (a concurrent HashMap actually) or maybe even create one reqwest client per host; I don't know which one is the better option right now.

E.g. the hash map could look like this:

use rate_limits::ResetTime;
use std::collections::HashMap;
use time::OffsetDateTime;

let mut rate_limits = HashMap::new();
rate_limits.insert(
    "github.com".to_string(),
    ResetTime::DateTime(OffsetDateTime::from_unix_timestamp(1350085394).unwrap()),
);

An entry would be inserted once we get rate-limited.
Before every request, we would check if the current time is after the ResetTime.
If not, we'd wait the difference and finally remove the rate limit entry from the map.
This would scale much better than a global delay and would cover more use-cases.

What do you think?

from lychee.

It is pretty common for APIs, but not for websites I would guess. Realistically we might still need both, the rate-limit headers and a way to configure the delay.

Let's start with rate-limit headers, though, because that's a common way for a website to tell us to slow down. Another common way is the infamous 429. We currently retry those requests with an exponential backoff, which is a start. (We could do better by estimating an optimal request interval based on the response time with various request delays, but let's not get ahead of ourselves.)

from lychee.

Instead, my proposal is to add better rate-limiting support per website.

👆This!

Checking my awesome-falsehood project for dead links reveals some false positives for news.ycombinator.com and twitter.com domains. Of course these two protects themselves from abuse, and concurrent access by Lychee are seen as such.

--max-concurrency 1 solves the issue.

But we forfeit any performance. The ideal solution would be a way to have either --max-concurrency-per-domain 1 or --delay-per-domain 1.5s option.

from lychee.

You are absolutely right that the needed delay would need to be tweaked to fit all queried hosts... and there is likely no common ground. And of course concurrency needs to be set to 1 and everything would be slowed down.

Your proposal sounds like a pretty smart solution!
How common is the usage of this headers already? I see that the newest IETF document also covers APIs, but everything is still in draft?

from lychee.

Did you manage to check twitter links lately? It's failing on my end, even with our workaround to use nitter instead. Maybe the concurrency is what's killing it for me.

Haven't encountered any issues with HN yet, even though it's probably a matter of not triggering their rate-limiting. Out of curiosity, how many requests to news.ycombinator.com does it take until you encounter any issues?

from lychee.

Did you manage to check twitter links lately?

It's more complicated than that: --max-concurrency 1 fixed it from my machine. But it doesn't from my GitHub action workflows. So there is hard rate-limiting by Twitter from requests originating from GitHub.

how many requests to news.ycombinator.com does it take until you encounter any issues?

Around 4 request:

Source: kdeldycke/awesome-falsehood#159

from lychee.

That's great to hear! To whoever might be interested in tackling that, feel free to post a comment here.
Generally, we can follow the approach of muffet that was linked above.

from lychee.