too many requests error on github profile links even if use --github-token about lychee HOT 4 CLOSED

There currently is no automatic rate limiting, so it depends on how many GitHub links you want to check.
When using GITHUB_TOKEN, the rate limit is 1,000 requests per hour per repository.

You can check with

lychee --dump --include github README.md | wc -l

If the number is big, you can try to manually configure the number of concurrent requests:

--max-concurrency 10

Also see the lychee docs.

from lychee.

Thank you for your response, I ran your command and confirmed that there are ~700 GitHub links in the file. However, even if I limit the max-concurrency to 1, there are still ~100 links with 429 error code. Is this normal?

from lychee.

If you run lychee twice, that would be 1400 links and so on the second run I'd expect 400 links to fail if all requests were done asynchronously. With --max-concurrency 1, the entire runtime would be the sum of all individual requests, so the request budget might hold a bit longer, so the 100 failing links sound about right then. Not sure if that's the case.

Nonetheless, per host rate limiting is on the roadmap, but it's not there yet. The next best thing is probably caching requests with --cache.

from lychee.

My previous comment sums up all existing possibilities as of today. Will close this in favor of #989, which will tackle per-host rate limiting.

from lychee.