iquidus / blockspider Goto Github PK
View Code? Open in Web Editor NEWAn open source blockchain crawler and kafka producer.
License: MIT License
An open source blockchain crawler and kafka producer.
License: MIT License
make lint
fails to run. Investigate and fix.
Add options to format blocks in various ways, eg:
Root level
{
Type: uint (only eth based networks supported at present, type = 0)
}
More types to be added in future, e.g bitcoin, substrate, cosmos.
Per topic/stream
{
Compact: bool (if true only include required fields [number, hash, parentHash])
Transactions: bool (if true include transaction objects)
Logs: bool (if true include log objects)
}
make all
& make test
both fail.
update build/ci logic to fix issues.
Polling with eth_getlogs on 3rd party node providers such as Alchemy consumes too much compute. Add support for alchemy webhooks, so we don't have to crawl chains that Alchemy supports.
https://docs.alchemy.com/reference/custom-webhooks-quickstart
blockspider was initially built using the older ethereum PoW block format, (e.g Ubiq, Ethereum Classic), extend type to support the newer Ethereum PoS blocks.
If the daemon is not running, and the block it last stored in local state is reorg'd, when restarting the daemon it fails to determine common ancestor and handle the reorg'd blocks/events correctly.
Note: this does not affect reorgs that occur while the daemon is running (it handles them correctly).
Consider storing entire block cache in local state.
Complete introduction, configuration & setup docs. Include examples of common configurations eg:
All blocks
All events
ERC20 transfers (all tokens)
ERC20 events (single token)
ERC721 transfers (all tokens)
ERC721 events (single collection)
Remove "S1000: should use for range instead of for { select {} }"
from excludes and address lint errors.
Current test coverage is too small
go run build/ci.go test
>>> /usr/local/go/bin/go test -timeout=20m -tags=ckzg -p 1 ./...
ok github.com/iquidus/blockspider/cache (cached)
? github.com/iquidus/blockspider/cmd/blockspiderd [no test files]
? github.com/iquidus/blockspider/cmd/consumer [no test files]
? github.com/iquidus/blockspider/cmd/reorgd [no test files]
? github.com/iquidus/blockspider/common [no test files]
? github.com/iquidus/blockspider/crawler [no test files]
? github.com/iquidus/blockspider/internal/build [no test files]
? github.com/iquidus/blockspider/kafka [no test files]
? github.com/iquidus/blockspider/params [no test files]
? github.com/iquidus/blockspider/rpc [no test files]
? github.com/iquidus/blockspider/state [no test files]
ok github.com/iquidus/blockspider/syncronizer (cached)
? github.com/iquidus/blockspider/util [no test files]
Finish writing tests. This will likely require some refactoring.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.