Comments (3)
I took a bit of look at this and a few ideas from my side:
-
We should add a configurable timeout and a configurable number of retries, probably also MAX_TASK_BATCH_SIZE should be configurable too so we could tweak these values accordingly. In this way we can experiment with different values for these and track results (see 2)
-
We should capture the metrics for error rates and compare, right now it looks like it's quite different across different envs. We have one env where it's about 30% at the moment. Is the bgtask_succes/bgtask_error sufficiently finegrained metric or is there another one we should look at ?
-
When it comes to the priority, we could add a priority field to the DB and add a second ordering here: https://github.com/linuskendall/digital-asset-rpc-infrastructure/blob/fix_das_api_frontend_panic/nft_ingester/src/tasks/mod.rs#L149 ? We could also if we wanted to get fancy add the ability for a bgtask runner to only process tasks with priority > N, so you can deploy priority runners in your infra and others. This would require some tooling to interface with the DB and mark certain assets as priority for downloads.
Potential future:
- Maybe we should have something like a download manager for these downloads? For example I am wondering if we should ratelimit our fetches from a single upstream service.
from digital-asset-rpc-infrastructure.
This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.
I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.
It's possible that the http lib we are using doesnt handle many many connections to the same host very well.
from digital-asset-rpc-infrastructure.
This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.
I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.
It's possible that the http lib we are using doesnt handle many many connections to the same host very well.
We would really benefit form setting a TTL on off-chain data. So we don't re-index an offchain until we have passed the TTL duration. This would dramatically decrease the load and especially help with services like arweave or ipfs gateways that rate limit by IP.
from digital-asset-rpc-infrastructure.
Related Issues (20)
- Potential issue with redeem indexing of transactions HOT 1
- When 1.16 is out upgrade to use stack height changes from blockbuster HOT 1
- Feat: Out of order updates: if we see a missing mint, push that to be backfilled HOT 1
- All Seller Fee Basis Points Indexed as 0 HOT 1
- Outdated init.sql HOT 2
- Consider removing `name` and `symbol` from chain data and just use raw_name and raw_symbol
- Return metadata in raw format
- Question: Get owner from PDA if the owner is off-curve
- Reading from snapshot during the startup
- Decompression indexing authority HOT 1
- Validator slot falling behind when enabled `nft_ingester`. HOT 4
- [Feature] Tree Sequence Based Backfilling HOT 5
- [Bug] saving cl_audit records is not idempotent HOT 1
- Replace `cl_audits` table with `tree_transactions` HOT 2
- [Proposal] Project Structure and DAS Operations HOT 3
- Ingestion congestion from inserting backfill_items
- [Proposal] getAssetSignatures method response schema
- [RFC] Solana Transaction Structures HOT 4
- Store transaction status in plerkle serialised format
- Proposal: gRPC based indexer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from digital-asset-rpc-infrastructure.