next(CDXFetcherIter) hangs when cc request returns 400/404 about cdx_toolkit HOT 2 CLOSED

I did not use cdx server api before so I don't know its behavior before but it seems that with a large page number, the request would eventually return 400, for example with page=0
https://index.commoncrawl.org/CC-MAIN-2020-50-index?url=http%3A%2F%2Fwww.illinoishomepage.net%2Fnews%2Flocal-news%2Fman-who-fell-through-ice-dies&output=json&page=0
return 404, text is "error no capture".
But with page=1,
https://index.commoncrawl.org/CC-MAIN-2020-50-index?url=http%3A%2F%2Fwww.illinoishomepage.net%2Fnews%2Flocal-news%2Fman-who-fell-through-ice-dies&output=json&page=1
returns 400 and get_more() will move to next endpoint.

But for my example above, for CC-MAIN-2017-34-index, no matter what the page number is, for example, page=1000000, https://index.commoncrawl.org/CC-MAIN-2017-34-index?url=http%3A%2F%2Fwww.illinoishomepage.net%2Fnews%2Flocal-news%2Fman-who-fell-through-ice-dies&output=json&page=1000000
still returns 404 with text "error no capture". And get_more() will keep increasing the page number and retry.

from cdx_toolkit.

Sebastian fixed this bug on the Common Crawl side, it was a missing index file for just that one index. It happened because the index moved to a new server recently. Thanks for finding it!

I will add some kind of guard to the code, so this infinite loop is at least a visible infinite loop.

from cdx_toolkit.