graphprotocol / graph-node Goto Github PK

Graph Node indexes data from blockchains such as Ethereum and serves it over GraphQL

License: Apache License 2.0

Rust 94.47% CSS 0.71% HTML 0.13% TypeScript 2.00% PLpgSQL 2.22% Dockerfile 0.07% Shell 0.28% JavaScript 0.07% Solidity 0.05% Makefile 0.02%

graphql graphql-api blockchain protocol ethereum ipfs graphql-server developer-tools

graph-node's Introduction

Graph Node

The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL.

Graph Node is an open source Rust implementation that event sources the Ethereum blockchain to deterministically update a data store that can be queried via the GraphQL endpoint.

For detailed instructions and more context, check out the Getting Started Guide.

Quick Start

Prerequisites

To build and run this project you need to have the following installed on your system:

Rust (latest stable) – How to install Rust
- Note that rustfmt, which is part of the default Rust installation, is a build-time requirement.
PostgreSQL – PostgreSQL Downloads
IPFS – Installing IPFS
Profobuf Compiler - Installing Protobuf

For Ethereum network data, you can either run your own Ethereum node or use an Ethereum node provider of your choice.

Minimum Hardware Requirements:

To build graph-node with cargo, 8GB RAM are required.

Docker

The easiest way to run a Graph Node is to use the official Docker compose setup. This will start a Postgres database, IPFS node, and Graph Node. Follow the instructions here.

Running a Local Graph Node

This is a quick example to show a working Graph Node. It is a subgraph for Gravatars.

Install IPFS and run ipfs init followed by ipfs daemon.
Install PostgreSQL and run initdb -D .postgres -E UTF8 --locale=C followed by pg_ctl -D .postgres -l logfile start and createdb graph-node.
If using Ubuntu, you may need to install additional packages:
- sudo apt-get install -y clang libpq-dev libssl-dev pkg-config
In the terminal, clone https://github.com/graphprotocol/example-subgraph, and install dependencies and generate types for contract ABIs:

yarn
yarn codegen

In the terminal, clone https://github.com/graphprotocol/graph-node, and run cargo build.

Once you have all the dependencies set up, you can run the following:

cargo run -p graph-node --release -- \
  --postgres-url postgresql://USERNAME[:PASSWORD]@localhost:5432/graph-node \
  --ethereum-rpc NETWORK_NAME:[CAPABILITIES]:URL \
  --ipfs 127.0.0.1:5001

Try your OS username as USERNAME and PASSWORD. For details on setting the connection string, check the Postgres documentation. graph-node uses a few Postgres extensions. If the Postgres user with which you run graph-node is a superuser, graph-node will enable these extensions when it initializes the database. If the Postgres user is not a superuser, you will need to create the extensions manually since only superusers are allowed to do that. To create them you need to connect as a superuser, which in many installations is the postgres user:

    psql -q -X -U <SUPERUSER> graph-node <<EOF
create extension pg_trgm;
create extension pg_stat_statements;
create extension btree_gist;
create extension postgres_fdw;
grant usage on foreign data wrapper postgres_fdw to <USERNAME>;
EOF

This will also spin up a GraphiQL interface at http://127.0.0.1:8000/.

With this Gravatar example, to get the subgraph working locally run:

yarn create-local

Then you can deploy the subgraph:

yarn deploy-local

This will build and deploy the subgraph to the Graph Node. It should start indexing the subgraph immediately.

Command-Line Interface

USAGE:
    graph-node [FLAGS] [OPTIONS] --ethereum-ipc <NETWORK_NAME:FILE> --ethereum-rpc <NETWORK_NAME:URL> --ethereum-ws <NETWORK_NAME:URL> --ipfs <HOST:PORT> --postgres-url <URL>

FLAGS:
        --debug      Enable debug logging
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --admin-port <PORT>                           Port for the JSON-RPC admin server [default: 8020]
        --elasticsearch-password <PASSWORD>
            Password to use for Elasticsearch logging [env: ELASTICSEARCH_PASSWORD]

        --elasticsearch-url <URL>
            Elasticsearch service to write subgraph logs to [env: ELASTICSEARCH_URL=]

        --elasticsearch-user <USER>                   User to use for Elasticsearch logging [env: ELASTICSEARCH_USER=]
        --ethereum-ipc <NETWORK_NAME:[CAPABILITIES]:FILE>
            Ethereum network name (e.g. 'mainnet'), optional comma-separated capabilities (eg full,archive), and an Ethereum IPC pipe, separated by a ':'

        --ethereum-polling-interval <MILLISECONDS>
            How often to poll the Ethereum node for new blocks [env: ETHEREUM_POLLING_INTERVAL=]  [default: 500]

        --ethereum-rpc <NETWORK_NAME:[CAPABILITIES]:URL>
            Ethereum network name (e.g. 'mainnet'), optional comma-separated capabilities (eg 'full,archive'), and an Ethereum RPC URL, separated by a ':'

        --ethereum-ws <NETWORK_NAME:[CAPABILITIES]:URL>
            Ethereum network name (e.g. 'mainnet'), optional comma-separated capabilities (eg `full,archive), and an Ethereum WebSocket URL, separated by a ':'

        --node-id <NODE_ID>
            A unique identifier for this node instance. Should have the same value between consecutive node restarts [default: default]

        --http-port <PORT>                            Port for the GraphQL HTTP server [default: 8000]
        --ipfs <HOST:PORT>                            HTTP address of an IPFS node
        --postgres-url <URL>                          Location of the Postgres database used for storing entities
        --subgraph <[NAME:]IPFS_HASH>                 Name and IPFS hash of the subgraph manifest
        --ws-port <PORT>                              Port for the GraphQL WebSocket server [default: 8001]

Advanced Configuration

The command line arguments generally are all that is needed to run a graph-node instance. For advanced uses, various aspects of graph-node can further be configured through environment variables. Very large graph-node instances can also split the work of querying and indexing across multiple databases.

Project Layout

node — A local Graph Node.
graph — A library providing traits for system components and types for common data.
core — A library providing implementations for core components, used by all nodes.
chain/ethereum — A library with components for obtaining data from Ethereum.
graphql — A GraphQL implementation with API schema generation, introspection, and more.
mock — A library providing mock implementations for all system components.
runtime/wasm — A library for running WASM data-extraction scripts.
server/http — A library providing a GraphQL server over HTTP.
store/postgres — A Postgres store with a GraphQL-friendly interface and audit logs.

Roadmap

🔨 = In Progress

🛠 = Feature complete. Additional testing required.

✅ = Feature complete

Feature	Status
Ethereum
Indexing smart contract events	✅
Handle chain reorganizations	✅
Mappings
WASM-based mappings	✅
TypeScript-to-WASM toolchain	✅
Autogenerated TypeScript types	✅
GraphQL
Query entities by ID	✅
Query entity collections	✅
Pagination	✅
Filtering	✅
Block-based Filtering	✅
Entity relationships	✅
Subscriptions	✅

Contributing

Please check CONTRIBUTING.md for development flow and conventions we use. Here's a list of good first issues.

License

The Graph is dual-licensed under the MIT license and the Apache License, Version 2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for the specific language governing permissions and limitations under the License.

graph-node's People

Contributors

Stargazers

Watchers

Forkers

johansten rkapur123 soros1321 ch4r10t33r nomikeeper thirdplanetio coderfi flada-auxv hiddentao html5cat fragosti samparsky llogiq sriharikapu renatrazumov adamsoffer chattg1 junbeomlee pklinger miogreen nunofernandes-plight dillonhows blockarraygroup bruceherve dorgjelli madhur4444 evm-andrey yosriady cristianmalfesi tenfinney miguelramosfdz fleupold ana0 yssofer iameli moshiach770 akeem jpgonzalezra jesstada joachim-lebrun orenyodfat dappsinc amxx earthwater abarmat bernarcio luckyyang andypeng2015 isgasho robertdigital defiboy flowstake k-ho randomcomposition pmespresso lutter johnrjj amadeobrands levalleux-ludo pier-two teller-protocol mikehathaway ltlmaps samnaidups chrischiedo metmirr adzialocha jaggedsoft gen-crypto tommy-venator theophoric pkrasam mingderwang akegaviar crazylandd cpfiffer epiqueras joesixpack a8uhnf vvvictorlee conflux-network-global rexeee melvillian lstarby newswap ajrocpanda marlinprotocol tpscrpt wighawag cag my-swarm swift1911 fransixles alexv-n pynchmeister x5engine stakesquid khangle27 syndykator19 riccardobiosas

graph-node's Issues

Implement a simple query runner

We will assume the schema and query used during the execution have already been validated. This will be the responsibility of a separate Query Validator component.

The query runner must adhere to the execution section of the GraphQL spec but must ignore mutations: http://facebook.github.io/graphql/October2016/#sec-Execution

Add pipeline task to benchmark the GraphQL server

Make local node use clap and parse a Postgres URL argument to pass to the diesel store

Rename the-graph-service repository to the-graph-network

Create Rust project skeleton

Implement a Data Source Runtime wrapper (index.js)

This script would load the data-source.json file, load the files referenced in the data source definition (from files/), instantiate contracts using web3, set up IPC and eventually pass everything to the default export function of mapping.js.

Add mock implementations for all components

Update rust toolchain on BuildKite nodes to Rust 1.26

Add GraphQL interfaces support

Implement missing schema introspection features

These are:

__type arguments: the signature should be __type(name: String!): __Type but we don't currently support the name argument).
Union types (we only handle unions when looking at whether fragments apply; everywhere else they are not supported yet).
The "interfaces" field of any types inside "possibleTypes" (with the current eager construction of the introspection schema, filling this in would result in infinite recursion).
Deprecation support.

Implement a simple schema provider for the local node

This schema provider should simply take the schemas from the data source provider and combine them into one.

Translate GraphQL arguments into store queries

Add data to MockStore and schemas to MockDataSourceProvider

Extract entity types / IDs from GraphQL arguments in the StoreResolver

As @fordN pointed out rightly in #67 (review), this is still missing and we'll need it to resolve relationships between entities (and be able to query specific entities via the singular entity query fields).

Implement Store::find for String attribute values and non-composite filters (no And/Or)

We will track support for non-string attribute values and filters for those in another issue.

Mock out and wire up all standalone node components

Set up cluster monitoring with Google Stackdriver

We'll want to monitor our standalone node (uptime etc.). Google Stackdriver can do this for us and will hopefully integrate well with Kubernetes.

Create traits and types for the components of the standalone node

Add a Diesel/Postgres store skeleton

Write testing, running and install instructions for the local node

Add GraphQL arguments support

Deploy the-graph-network docs on Kubernetes as part of CI

Set up a CI cluster with preconfigured BuildKite agents

Create master-only pipeline to deploy the standalone node

Report store errors back to client instead of panicking

Example: We won't support all filters for all types, so we should

Try to only expose filters that we support in the GraphQL schema
Allow the Store to return useful errors when filters are not supported for certain value types.

Add Stackdriver uptime check for the staging standalone node

We'll later want the same for the public gateway nodes.

Create local node deployment with a co-located private testnet on the staging cluster

This should be preconfigured to use a local district0x data source.

The private testnet should have some sample data transacted into it that we can query.

Parse incoming GraphQL queries

We'll want to parse any queries that come in via POST / so that we can run them in the query runner.

Track crashes and other errors using Sentry

This would allow us to be notified whenever our deployed instance(s) crash or trigger an error that we consider a bug. It would also allow us to create GitHub issues from automatically generated crash reports and assign these to developers for fixing.

Create BuildKite pipeline definitions

Implement a RuntimeAdapter in thegraph-runtime-nodejs

This DataSourceConnector would take in a DataSourceDefinition (see #60) and instantiate data source runtime that lives in another process and is based on Node.js.

This involves:

Create a temporary directory.
Creating a package.json with default dependencies (like web3, and IPFS)
Serialize the data source definition to a data-source.json file in the temporary directory using serde_json
Copy the files referenced in the data source definition into the temporary directory under files/
Write the mapping script from the data source definition to an mapping.js.
Create a index.js that connects the indexing code in mapping.js to the DataSourceConnector
- This means we need to stand up an IPC interface
- Run index.js with Node, calling the default exported function from mapping.js with data initialized from data-source.json
- Connect to the IPC interface

Return query results in the right (JSON) format

For this we'll need to serialize query results (and server errors) into JSON according to http://facebook.github.io/graphql/October2016/#sec-Response.