Git Product home page Git Product logo

eth-indexer's Introduction

High Performance Ethereum Indexer

Syncs events from Ethereum and indexes them for fast retrieval. This is useful and often essential for production Dapps.

  • Support for 6 different persistence stores
  • Includes "single-node friendly" stores such as flat-file JSON
  • Comprehensive error retry logic when communicating with Ethereum node
  • Maximum throughput via JSON RPC using batched parallel fetching

Todo

  • Keep track of blockNumbers in persistence for fast-resume
  • Benchmark documentation
  • Support going forward syncing for Ethereum
  • NPM package
  • Improved failsafes for data consistency
  • Split up stores into separate dependencies

Supported Indexing Stores and Benchmarks

The following indexing implementations have been benchmarked on an Ethereum blockchain section with a relatively large number of events (EtherDelta exchange blocks 4800000 - 5002718) on a single 8-core 2017 Macbook Pro:

Store Name Store ID Events Per Second Blocks Per Second
Local Memory memory 4072.7/s 464.5/s
Redis redis 2049.55/s 235.25/s
Local Flat File file 1798.75/s 201.75/s
MongoDB mongodb 795.5/s 94.6/s
Elasticsearch elasticsearch 452.25/s 57.1/s
LevelDB level 207.35/s 26.95/s

Install Dependencies

yarn install

Testing & Linting

yarn test
yarn lint

Integration tests (requires synced ethereum node at localhost:8545):

yarn test:integration

Example: Sync trading and balance events from decentralized trading contract: EtherDelta

Contract address: https://etherscan.io/address/0x8d12a197cb00d4747a1fe03395095ce2a5cc6819

See examples/etherdelta/sync.js.

Define which events to index on which keys:

const indexing = {
  events: {
    Withdraw: {
      keys: ['user'],
    },
    Trade: {
      keys: ['tokenGive', 'tokenGet', 'get', 'give'],
    },
  },
};

Then create a store:

const store = new LevelStore(indexing, '/tmp/etherdelta.db');

Now boot up the indexer:

const indexer = new Indexer(store, EtherdeltaABI, '0x8d12a197cb00d4747a1fe03395095ce2a5cc6819');
await indexer.syncAll({
  fromBlock: 4906764,
});

Once indexing is complete events can be retrieved as follows:

const events = await store.get('Withdraw', 'user', '0x13d8d38421eb02973f3f923a71a27917bd483190');

Directory Structure

  • package.json - Configure dependencies
  • dist/* - Files generated by babel
  • src - All source code
  • src/*/__tests__ - Unit tests
  • integration/__tests__ - Integration tests

eth-indexer's People

Contributors

dominiek avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.