Offchain Indexers unreliably processing data about digital-asset-rpc-infrastructure HOT 3 OPEN

I took a bit of look at this and a few ideas from my side:

1. We should add a configurable timeout and a configurable number of retries, probably also MAX_TASK_BATCH_SIZE should be configurable too so we could tweak these values accordingly. In this way we can experiment with different values for these and track results (see 2)

2. We should capture the metrics for error rates and compare, right now it looks like it's quite different across different envs. We have one env where it's about 30% at the moment. Is the bgtask_succes/bgtask_error sufficiently finegrained metric or is there another one we should look at ?

3. When it comes to the priority, we could add a priority field to the DB and add a second ordering here: https://github.com/linuskendall/digital-asset-rpc-infrastructure/blob/fix_das_api_frontend_panic/nft_ingester/src/tasks/mod.rs#L149 ? We could also if we wanted to get fancy add the ability for a bgtask runner to only process tasks with priority > N, so you can deploy priority runners in your infra and others. This would require some tooling to interface with the DB and mark certain assets as priority for downloads.

Potential future:

4. Maybe we should have something like a download manager for these downloads? For example I am wondering if we should ratelimit our fetches from a single upstream service.

from digital-asset-rpc-infrastructure.

This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.

I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.

It's possible that the http lib we are using doesnt handle many many connections to the same host very well.

from digital-asset-rpc-infrastructure.

This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.

I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.

It's possible that the http lib we are using doesnt handle many many connections to the same host very well.

We would really benefit form setting a TTL on off-chain data. So we don't re-index an offchain until we have passed the TTL duration. This would dramatically decrease the load and especially help with services like arweave or ipfs gateways that rate limit by IP.

from digital-asset-rpc-infrastructure.