Read timed out about cdx_toolkit HOT 5 CLOSED

The code already retries 10 times for 30 seconds each.

Common Crawl's index server has been having problems with someone who's fetching data from thousands of AWS EC2 nodes. That person is making the CC server more unreliable than usual.

If you'd like to restart your computation roughly where it went off, you can use timestamps to do that.

If you'd like to make sure your code doesn't get one of these errors in the middle of the iteration, you can raise the maximum in the line if connect_errors > 10 in myrequests.py

from cdx_toolkit.

okay got it. can you give support to increase this on itor with kwargs ?

from cdx_toolkit.

Yeah, I recall thinking about that potentially being a problem when I wrote the code :-) really I should raise the limits until it always works for the typical cdx server -- I just need to convince it to complain visibly after enough failures (i.e. --source hostname that resolves but no one's there) without making the test suite run longer (it's already a little annoying.)

from cdx_toolkit.

I just pushed a new version which tries 10x more times to connect, while failing fast if the hostname DNS fails. That ought to be enough to battle through CC's occasional problems.

from cdx_toolkit.

thanks

from cdx_toolkit.