eutils rate limits exceeded (with relatively small input set of accessions) about fetchngs HOT 4 OPEN

Yeah, fair point. I would put this as another pointer towards using ffq #110 If you have the time and it's important to you right now, introducing ffq should be quite straight forward.

from fetchngs.

thanks @Midnighter - looks like @drpatelh is on top of it already (#100)! I haven't looked closely at that but I'll just add that I'd guess that simply swapping in calls to ffq to the current parallel structure would likely end up hitting the same rate limit issue ( see https://github.com/pachterlab/ffq#failure-modes number 4). So probably trying to use batching for the metadata calls would be relevant in any case?

from fetchngs.

Still haven't had time to look into adding ffq properly I'm afraid 😏 Going to get a maintenance release out in the next couple of days. We can maybe assess this after we have added ffq support.

from fetchngs.

Btw, the only way to avoid this is by calling the run info resolution in a single process that can control the request rate. As soon as multiple processes are started in parallel, all bets are off.

from fetchngs.