Git Product home page Git Product logo

41north / besu-exflo Goto Github PK

View Code? Open in Web Editor NEW
17.0 3.0 7.0 3.76 MB

A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.

License: Apache License 2.0

Kotlin 53.55% Dockerfile 0.96% Java 20.09% Shell 0.37% JavaScript 4.44% Solidity 20.59%
ethereum besu kafka postgres kotlin crypto hyperledger hyperledger-besu

besu-exflo's People

Contributors

aldoborrero avatar brianmcgee avatar dependabot[bot] avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

freight-trust iss-lab sambacha neoxolotl gnangla 0x3n3jvhpuv developerkevin

besu-exflo's Issues

Capture state changes resulting from hard forks

There are certain state changes as a result of hard forks that we currently do not capture. An example would be the refunds introduced in block 1.92 million for the DAO issue.

We need a strategy to define and emit the hard coded state changes at a given block. There is a similar pattern worth reviewing which is already implemented within Besu for handling protocol changes at specific blocks.

Update gradle to 6.4

Gradle has released version 6.4. We should update without major issues Exflo to take advantage of it.

Add Dockerfile with besu and bundled plugins

In order to ease the usage of Exflo, we should setup a Dockerfile that bundles Besu + Exflo together.

Optionally, we can setup a Github Action to publish automatically the image to Docker Hub.

Replace usage of Klaxon library with Jackson

Currently, we are using a mixture of jackson libraries for several parts of the project that can be found on:

  • buildSrc: deserialization from YAML | auto code generation of web3 entities and so on
  • base: allows to specify a custom ClassLoader

And klaxon that can be found on testutil package.

In order to simplify things, and dependencies, I propose to migrate testutil as well to use directly jackson instead.

Improve README

We need a better README that express intentions and clarifies how to properly configure / develop / run Exflo. It should also address common configuration issues and problems.

Things that need a better clarification are:

  • What is the output of the plugin? We need a way to showcase the produced output for Postgres and Kafka
  • How do I jump and help to contribute?
  • How I can execute the plugin using JAR / Docker options?
  • Better organization
  • ...

Improve detection of configured Besu instance

Currently, we are detecting different Besu configuration options by resorting to pre-configured environment variables.

In reality, this showcases the current limitation of the plugin system (a thing we should raise to Besu community), meanwhile, I think it's better to resort to reflection in order to obtain current instances so we are not missing neither constructing by hand needed object instances.

Update flatbuffers to 1.12.0

There's a new version major version of flatbuffers that has been released. In Exflo we are currently using version 1.11.0.

This task should take care of updating all required dependencies, entities and codebase to version 1.12.0. Things to take care of:

  • Update Dockerfile to use version 1.12.0
  • Update java dependency to use version 1.12.0
  • Regenerate entities with gradle
  • Correct and fix errors

Add support for IBFT 2.0 consensus algorithm

Algorithm is detailed here

It is highly likely given how we integrated and followed the lead of Besu for things such as block reward etc that we already support IBFT 2.0, however this needs verified.

In addition we should capture the voting events and epoch history.

Improvements to truffle/bin/capture-test-blocks.sh script

truffle/bin/capture-test-blocks.sh is the script in charge of generating a test blockchain to execute tests. The script in its current state is very naive and simplistic in how it handle error failures.

If for example we want to generate blocks and somehow the testing suite fails, a dangling docker besu image is kept running without being terminated.

A good approach could be using trap signals like this good article explains. So the script is allowed to handle the error and finish its state cleanly.

Update Kafka plugin to use an import queue

Right now, the current implementation of Kafka is very simple and straightforward.

When traces are enabled, some of them takes time to process depending on the smart contract and the amount of CREATE opcodes that it has. Average time ranges between: 80ms to the extreme of 2 seconds.

Clearly we can benefit of using the concept of import queue on Kafka and be as concurrent as we can while preserving the nature of the topic.

Add support for Clique consensus protocol

Algorithm is detailed here

It is highly likely given how we integrated and followed the lead of Besu for things such as block reward etc that we already support Clique, however this needs verified.

In addition we should capture the voting events and epoch history.

Upgrade Besu dependencies to 1.4.4

Upgrade to the latest release of Besu 1.4.4.

Test for any compatibility issues and review release notes for anything interesting / useful.

Tracing is failing with Ropsten after upgrade to Besu 1.4

Expected Behavior

Traces being added to postgres database and published via kafka when importing the Ropsten chain

Current Behavior

An exception is being thrown quite early in the chain

besu_1      | java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 0
besu_1      |   at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
besu_1      |   at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
besu_1      |   at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
besu_1      |   at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
besu_1      |   at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
besu_1      |   at io.exflo.ingestion.tracer.ExfloOperationTracer.tracePrecompileCall(ExfloOperationTracer.kt:253) ~[plugin-e15a7da.jar:?]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.MainnetMessageCallProcessor.executePrecompile(MainnetMessageCallProcessor.java:134) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.MainnetMessageCallProcessor.start(MainnetMessageCallProcessor.java:61) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.AbstractMessageProcessor.process(AbstractMessageProcessor.java:167) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:361) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:306) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:47) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at org.hyperledger.besu.ethereum.mainnet.TransactionProcessor.processTransaction(TransactionProcessor.java:169) ~[besu-core-1.4.4.jar:1.4.4]
besu_1      |   at io.exflo.ingestion.tracer.BlockTracer$trace$1.invoke(BlockTracer.kt:58) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockTracer$trace$1.invoke(BlockTracer.kt:39) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockReplay$block$1.invoke(BlockReplay.kt:57) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockReplay$block$1.invoke(BlockReplay.kt:36) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockReplay.performActionWithBlock(BlockReplay.kt:90) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockReplay.block(BlockReplay.kt:43) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracer.BlockTracer.trace(BlockTracer.kt:48) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.tracker.BlockReader.trace(BlockReader.kt:125) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.postgres.tasks.TraceImportTask$run$1.apply(TraceImportTask.kt:84) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.exflo.ingestion.postgres.tasks.TraceImportTask$run$1.apply(TraceImportTask.kt:46) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.reactivex.rxjava3.internal.operators.parallel.ParallelMap$ParallelMapSubscriber.onNext(ParallelMap.java:114) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.reactivex.rxjava3.internal.operators.parallel.ParallelRunOn$RunOnSubscriber.run(ParallelRunOn.java:273) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) ~[plugin-e15a7da.jar:?]
besu_1      |   at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) ~[plugin-e15a7da.jar:?]
besu_1      |   at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
besu_1      |   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
besu_1      |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
besu_1      |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
besu_1      |   at java.lang.Thread.run(Thread.java:834) [?:?]

Steps to Reproduce

  1. Build the exflo docker image:
docker build -t 41north/exflo -f docker/exflo/Dockerfile .
  1. Modify docker/exflo/docker-compose.postgres.yml, adding the following environment variable:
BESU_PLUGIN_EXFLO_POSTGRES_PROCESSING_LEVEL: TRACES
  1. Run the postgres example:
docker-compose -f docker/exflo/docker-compose.postgres.yml up

Alternatively just apply step 2 to your intellij run config for Postgres & Ropsten.

KafkaCache: Could not read to end offsets

Expected Behavior

If I execute the following command with no previous volumes set:

docker-compose -f docker/exflo/docker-compose.kafka.yml up

It should start processing blocks and inserting those into Kafka.

Current Behavior

What is the current behavior?

As soon as Kafka is initalized, Exflo outputs the following error:

besu_1       | 2020-05-04 11:37:39.170+00:00 | main | WARN  | ConsumerConfig | The configuration 'topic' was supplied but isn't a known config.
besu_1       | 2020-05-04 11:37:39.170+00:00 | main | INFO  | AppInfoParser | Kafka version: 2.4.0
besu_1       | 2020-05-04 11:37:39.170+00:00 | main | INFO  | AppInfoParser | Kafka commitId: 77a89fcf8d7fa018
besu_1       | 2020-05-04 11:37:39.170+00:00 | main | INFO  | AppInfoParser | Kafka startTimeMs: 1588592259170
besu_1       | 2020-05-04 11:37:39.182+00:00 | main | INFO  | KafkaConsumer | [Consumer clientId=kafka-cache-reader-_exflo-import-cache, groupId=kafkacache] Subscribed to partition(s): _exflo-import-cache-0
besu_1       | 2020-05-04 11:37:39.185+00:00 | main | INFO  | KafkaCache | Seeking to beginning for all partitions
besu_1       | 2020-05-04 11:37:39.186+00:00 | main | INFO  | SubscriptionState | [Consumer clientId=kafka-cache-reader-_exflo-import-cache, groupId=kafkacache] Seeking to EARLIEST offset of partition _exflo-import-cache-0
besu_1       | 2020-05-04 11:37:39.186+00:00 | main | INFO  | KafkaCache | Initialized last consumed offset to {0=-1}
besu_1       | 2020-05-04 11:37:39.187+00:00 | main | INFO  | KafkaCache | KafkaTopicReader thread started for kafka-cache-reader-_exflo-import-cache.
besu_1       | 2020-05-04 11:37:39.193+00:00 | main | INFO  | Metadata | [Consumer clientId=kafka-cache-reader-_exflo-import-cache, groupId=kafkacache] Cluster ID: 1o2lE4tqRC24CfgQlD3jnw
besu_1       | 2020-05-04 11:37:39.216+00:00 | main | INFO  | SubscriptionState | [Consumer clientId=kafka-cache-reader-_exflo-import-cache, groupId=kafkacache] Resetting offset for partition _exflo-import-cache-0 to offset 0.
besu_1       | 2020-05-04 11:37:39.226+00:00 | kafka-producer-network-thread | producer-1 | INFO  | Metadata | [Producer clientId=producer-1] Cluster ID: 1o2lE4tqRC24CfgQlD3jnw
besu_1       | 2020-05-04 11:37:39.284+00:00 | main | INFO  | KafkaCache | During init or sync, processed 1 records from topic _exflo-import-cache
besu_1       | 2020-05-04 11:37:39.285+00:00 | main | INFO  | BlockImportTask | Syncing import cache
besu_1       | 2020-05-04 11:37:39.285+00:00 | kafka-cache-reader-thread-_exflo-import-cache | INFO  | ShutdownableThread | Starting
besu_1       | 2020-05-04 11:37:39.286+00:00 | main | WARN  | KafkaCache | Could not read to end offsets
besu_1       | org.apache.kafka.common.errors.WakeupException: null
besu_1       |  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:511) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:275) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.Fetcher.fetchOffsetsByTimes(Fetcher.java:537) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.Fetcher.beginningOrEndOffset(Fetcher.java:578) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.internals.Fetcher.endOffsets(Fetcher.java:567) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:2193) ~[plugin-6bda013.jar:?]
besu_1       |  at org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:2165) ~[plugin-6bda013.jar:?]
besu_1       |  at io.kcache.KafkaCache$WorkerThread.readToEndOffsets(KafkaCache.java:634) ~[plugin-6bda013.jar:?]
besu_1       |  at io.kcache.KafkaCache$WorkerThread.waitUntilEndOffsets(KafkaCache.java:802) [plugin-6bda013.jar:?]
besu_1       |  at io.kcache.KafkaCache$WorkerThread.access$100(KafkaCache.java:565) [plugin-6bda013.jar:?]
besu_1       |  at io.kcache.KafkaCache.sync(KafkaCache.java:205) [plugin-6bda013.jar:?]
besu_1       |  at io.exflo.ingestion.kafka.tasks.BlockImportTask.<init>(KafkaImportTask.kt:113) [plugin-6bda013.jar:?]
besu_1       |  at io.exflo.ingestion.kafka.KafkaBlockWriter.<init>(KafkaBlockWriter.kt:36) [plugin-6bda013.jar:?]
besu_1       |  at io.exflo.ingestion.kafka.ExfloKafkaPlugin$implKoinModules$1$3.invoke(ExfloKafkaPlugin.kt:47) [plugin-6bda013.jar:?]
besu_1       |  at io.exflo.ingestion.kafka.ExfloKafkaPlugin$implKoinModules$1$3.invoke(ExfloKafkaPlugin.kt:37) [plugin-6bda013.jar:?]
besu_1       |  at org.koin.core.instance.DefinitionInstance.create(DefinitionInstance.kt:54) [plugin-6bda013.jar:?]
besu_1       |  at org.koin.core.instance.FactoryDefinitionInstance.get(FactoryDefinitionInstance.kt:37) [plugin-6bda013.jar:?]
besu_1       |  at org.koin.core.definition.BeanDefinition.resolveInstance(BeanDefinition.kt:70) [plugin-6bda013.jar:?]
besu_1       |  at org.koin.core.scope.Scope.resolveInstance(Scope.kt:165) [plugin-6bda013.jar:?]
besu_1       |  at org.koin.core.scope.Scope.get(Scope.kt:128) [plugin-6bda013.jar:?]
besu_1       |  at io.exflo.ingestion.ExfloPlugin.start(ExfloPlugin.kt:236) [plugin-6bda013.jar:?]
besu_1       |  at org.hyperledger.besu.services.BesuPluginContextImpl.startPlugins(BesuPluginContextImpl.java:140) [besu-1.4.4.jar:1.4.4]
besu_1       |  at org.hyperledger.besu.cli.BesuCommand.startPlugins(BesuCommand.java:1106) [besu-1.4.4.jar:1.4.4]
besu_1       |  at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:958) [besu-1.4.4.jar:1.4.4]
besu_1       |  at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.4.jar:1.4.4]
besu_1       |  at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.4.jar:1.4.4]
besu_1       |  at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]
besu_1       |  at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1074) [besu-1.4.4.jar:1.4.4]
besu_1       |  at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:942) [besu-1.4.4.jar:1.4.4]
besu_1       |  at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.4.jar:1.4.4]
besu_1       | 2020-05-04 11:37:39.350+00:00 | main | INFO  | RunnerBuilder | Detecting NAT service.
besu_1       | 2020-05-04 11:37:39.424+00:00 | ExfloExecutorThread-%d | ERROR | BlockImportTask | Critical failure

It seems we should manage the WakeupException accordingly (as described here), this is happening mostly Besu has not started syncing blocks as is finding peers.

We should listen probably as well to main Besu events in order to start processing instead of just trying to do so.

Steps to Reproduce

As described above:

docker-compose -f docker/exflo/docker-compose.kafka.yml up

Make sure volumes are empty (to double check docker-compose -f docker/exflo/docker-compose.kafka.yml down -v)

Improve logging

Currently Besu has a document where it explains how they properly log on the project. We should take a similar approach on this plugin and improve logging messages where applicable.

This task should be in charge of reviewing all calls to the logger object and see if appropriate messages are displayed.

Also, for those classes that doesn't display any output, consider the option of adding a logger.

Improvements to error handling

The current behavior we have, although correct, doesn't signal specifically that something wrong has happened to Exflo. Right now, if something fails, Besu keeps running while Exflo is not.

We should discuss and see what is the best approach.

Add small dataset for Postgres and Kafka

In order to showcase what info we are able to extract, we can setup a a demo postgres and kafka dump.

This task should take care of:

  • Create a demo postgres dataset.
  • Create a demo kafka dataset.
  • Depending on the size, setup Git LFS (although better to avoid it as much as possible).
  • Add section to README so people can access it easily.

We can create the datasets from the dev testing network as are small enough and it includes most of the usecases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.