iquidus / blockspider Goto Github PK

make lint fails to run. Investigate and fix.

Add options to format blocks in various ways, eg:

Root level

{
  Type: uint (only eth based networks supported at present, type = 0)
}

More types to be added in future, e.g bitcoin, substrate, cosmos.

Per topic/stream

{
  Compact: bool (if true only include required fields [number, hash, parentHash])
  Transactions: bool (if true include transaction objects)
  Logs: bool (if true include log objects)
}

make all & make test both fail.

update build/ci logic to fix issues.

Polling with eth_getlogs on 3rd party node providers such as Alchemy consumes too much compute. Add support for alchemy webhooks, so we don't have to crawl chains that Alchemy supports.

https://docs.alchemy.com/reference/custom-webhooks-quickstart

blockspider was initially built using the older ethereum PoW block format, (e.g Ubiq, Ethereum Classic), extend type to support the newer Ethereum PoS blocks.

If the daemon is not running, and the block it last stored in local state is reorg'd, when restarting the daemon it fails to determine common ancestor and handle the reorg'd blocks/events correctly.

Note: this does not affect reorgs that occur while the daemon is running (it handles them correctly).

Consider storing entire block cache in local state.

Complete introduction, configuration & setup docs. Include examples of common configurations eg:

All blocks
All events

ERC20 transfers (all tokens)
ERC20 events (single token)
ERC721 transfers (all tokens)
ERC721 events (single collection)

Remove "S1000: should use for range instead of for { select {} }" from excludes and address lint errors.

Current test coverage is too small

go run build/ci.go test
>>> /usr/local/go/bin/go test -timeout=20m -tags=ckzg -p 1 ./...
ok  	github.com/iquidus/blockspider/cache	(cached)
?   	github.com/iquidus/blockspider/cmd/blockspiderd	[no test files]
?   	github.com/iquidus/blockspider/cmd/consumer	[no test files]
?   	github.com/iquidus/blockspider/cmd/reorgd	[no test files]
?   	github.com/iquidus/blockspider/common	[no test files]
?   	github.com/iquidus/blockspider/crawler	[no test files]
?   	github.com/iquidus/blockspider/internal/build	[no test files]
?   	github.com/iquidus/blockspider/kafka	[no test files]
?   	github.com/iquidus/blockspider/params	[no test files]
?   	github.com/iquidus/blockspider/rpc	[no test files]
?   	github.com/iquidus/blockspider/state	[no test files]
ok  	github.com/iquidus/blockspider/syncronizer	(cached)
?   	github.com/iquidus/blockspider/util	[no test files]

Finish writing tests. This will likely require some refactoring.