What is fast-near?

fast-near aims to provide the fastest RPC implementation for @NEARProtocol using high performance storage backends like:

in-memory storage in Redis.
SSD optimized storage using LMDB.

It is optimized for view call performance and ease of deploy and scaling.

It currently doesn't sync with network on its own, data needs to be loaded either from NEAR Lake or from https://github.com/vgrichina/near-state-indexer.

Why

nearcore RPC performance isn't good enough for novel use cases like https://web4.near.page.

fast-near achieves better performance by using

in-memory storage using Redis
client-side caching to save on Redis I/O
V8 WebAssembly implementation
disabled gas metering (timeout works fine for view calls)
simpler REST API (no JSON wrapper if passing large binaries, etc)
good compatibility with caching at HTTP layer (using Nginx, etc)

fast-near is also a good fit if you want to run RPC node serving limited subset of accounts (e.g. supporting your app exclusively) on a smaller hardware. This works well if data is sourced from NEAR Lake.

How to

Run directly from npm:

FAST_NEAR_REDIS_URL=<redis_ip> FAST_NEAR_NODE_URL=<rpc_endpoint> npx fast-near

Build and run via yarn:

yarn
FAST_NEAR_REDIS_URL=<redis_ip> FAST_NEAR_NODE_URL=<rpc_endpoint> yarn start

Build and run with docker:

docker build -t fastrpc .
docker run -d -e FAST_NEAR_REDIS_URL=<redis_ip> -e FAST_NEAR_NODE_URL=<rpc_endpoint> fastrpc

Pull data into Redis

To load from NEAR Lake (use --help to learn more about options):

node scripts/load-from-near-lake.js near-lake-data-mainnet --batch-size 50 --history-length 1 --dump-changes

Pull data into LMDB (experimental)

To load from NEAR Lake (use --help to learn more about options):

FAST_NEAR_STORAGE_TYPE=lmdb node scripts/load-from-near-lake.js near-lake-data-mainnet --batch-size 50 --history-length 1 --dump-changes

Run server:

FAST_NEAR_STORAGE_TYPE=lmdb yarn start

Load data only for your app's smart contracts

To load data for app1.near, app2.near and all subaccounts of superapp.near:

node scripts/load-from-near-lake.js near-lake-data-mainnet --include app1.near --include app2.near --include '*.superapp.near' --dump-changes

Exclude some undesired accounts

To load data for all accounts except aurora and sweat subaccounts:

node scripts/load-from-near-lake.js near-lake-data-mainnet --exclude aurora.* --exclude sweat.* --dump-changes

Different data sink options

Currently there are such options to dump data loaded from NEAR Lake:

--dump-changes - dumps state changes into storage. Use FAST_NEAR_STORAGE_TYPE to specify storage type. Defaults to redis

Load data from a local node

See https://github.com/vgrichina/near-state-indexer for Rust implementation running full nearcore node.

CLI options

Environment variables

PORT - port to listen on (default: 3000)
FAST_NEAR_STORAGE_TYPE - storage type to use (default: redis). Supported values: redis, lmdb.
FAST_NEAR_ENABLE_CACHE - enable client-side caching (default: true).
FAST_NEAR_LMDB_PATH - path to LMDB database (default: ./lmdb-data). This is only used if FAST_NEAR_STORAGE_TYPE is set to lmdb.
FAST_NEAR_REDIS_URL - Redis URL (default: redis://localhost:6379)
FAST_NEAR_NODE_URL - NEAR RPC endpoint (default: https://rpc.mainnet.near.org). This is only used as a fallback for JSON-RPC endpoint.
FAST_NEAR_ARCHIVAL_NODE_URL - NEAR RPC endpoint for archival node (default: https://rpc.mainnet.internal.near.org). This is only used as a fallback for JSON-RPC endpoint for data unavailable in Redis or on non-archival RPC.
FAST_NEAR_ALWAYS_PROXY - Always proxy JSON-RPC requests to FAST_NEAR_NODE_URL (default: false).
FAST_NEAR_START_BLOCK_HEIGHT - Minimum block height expected to be present in Redis (default: 0).
FAST_NEAR_WORKER_COUNT - Number of workers to use for execution of WASM code. (default: 4).
FAST_NEAR_CONTRACT_TIMEOUT_MS - Timeout for contract execution in milliseconds (default: 1000).

HTTP API

Public endpoints

Testnet: https://rpc.web4.testnet.page/account/testnet
Mainnet: https://rpc.web4.near.page/account/near

Call view method

POST

You can post either JSON or binary body, it's passed raw as input to given method.

URL format:

https://rpc.web4.near.page/account/<contract_account_id>/view/<method_name>

Examples

http post https://rpc.web4.near.page/account/vlad.tkn.near/view/ft_balance_of account_id=vlad.near

GET

Parameters are passed as part of URL query.

URL format:

https://rpc.web4.near.page/account/<contract_account_id>/view/<method_name>?<arg_name>=<string_arg_value>&<arg_name.json>=<json_arg_value>

Examples

String parameters:

curl 'https://rpc.web4.near.page/account/vlad.tkn.near/view/ft_balance_of?account_id=vlad.near'

https://rpc.web4.near.page/account/vlad.tkn.near/view/ft_balance_of?account_id=vlad.near

JSON parameters:

curl --globoff 'https://rpc.web4.near.page/account/lands.near/view/web4_get?request.json={"path":"/"}'

https://rpc.web4.near.page/account/lands.near/view/web4_get?request.json={"path":"/"}

Number parameters (passed as JSON):

curl 'https://rpc.web4.near.page/account/lands.near/view/getChunk?x.json=0&y.json=0'

https://rpc.web4.near.page/account/lands.near/view/getChunk?x.json=0&y.json=0

Get account info

GET

URL format:

https://rpc.web4.near.page/account/<account_id>

Example

curl 'https://rpc.web4.near.page/account/vlad.near'

https://rpc.web4.near.page/account/vlad.near

Get access key info

GET

URL format:

https://rpc.web4.near.page/account/<account_id>/key/<public_key>

Example

curl 'https://rpc.web4.near.page/account/vlad.near/key/ed25519:JBHUrhF61wfScUxqGGRmfdJTQYg8MzRr5H8pqMMjqygr'

https://rpc.web4.near.page/account/vlad.near/key/ed25519:JBHUrhF61wfScUxqGGRmfdJTQYg8MzRr5H8pqMMjqygr

Download contract WASM code

GET

URL format:

https://rpc.web4.near.page/account/<account_id>/contract

Example

curl 'https://rpc.web4.near.page/account/vlad.tkn.near/contract'

https://rpc.web4.near.page/account/vlad.tkn.near/contract

Get contract methods list

GET

URL format:

https://rpc.web4.near.page/account/<account_id>/contract/methods

Example

curl 'https://rpc.web4.near.page/account/lands.near/contract/methods'

https://rpc.web4.near.page/account/lands.near/contract/methods

Roadmap

Some of the planned and already implemented components. Is not exhaustive list.

Loading data
- Allow loading from NEAR Data Lake
- Allow loading from NEAR ZeroMQ Indexer for smaller latency
- Compress history to given time window
- [?] Update near-state-indexer to load latest format in Redis
- [?] Update nearcore to load latest format in Redis
- Load account keys
- Filter accounts when loading
- Load recent transactions results
- Manage lowest known block height dynamically
- Delegate to another fast-near REST API instance if given account data not present
- Delegate to another nearcore JSON-RPC instance if given account data not present?
REST API
- Call view methods
- View contract WASM
- View contract methods
- View account
- View contract state
- View account access key
- View account access keys list
- View transaction results
- Submit transaction
- Structured error handling
JSON-RPC API
- Call view methods
- View account
- Proxy to another node if not implemented / hitting archival
- Decide whether needs to be supported (e.g. Pagoda can allocate grant)
NEAR P2P Protocol
- Basic data structures
- POC downloading blocks with transactions
- Submit transaction
- Load and execute transactions
WASM Runtime
- Basic view method support
- Implement missing imports for view methods
- State change method support
Storage
- Redis
- Abstract storage API
- LMDB storage
- Load storage selectively from another fast-near instance
- Browser-based storage
Tests
- Test compress-history
- Test view calls
- Integration test with loading near-lake mainnet data
- Full coverage of runtime methods
- More robust integration tests

Endpoint `/account/:accountId/data/:keyPattern` returns inconsistent data and `iterator` doesn't work

I'm trying to fetch the most recent state of priceoracle.testnet contract but the response differs from the one that RPC request returns.

How to reproduce:

Make RPC call

Request:

curl --location --request POST 'https://rpc.testnet.near.org' \
--header 'Content-Type: application/json' \
--data-raw '{
    "jsonrpc": "2.0",
    "id": "dontcare",
    "method": "query",
    "params": {
        "request_type": "view_state",
        "finality": "final",
        "account_id": "priceoracle.testnet",
        "prefix_base64": "U1RBVEU=" -> "STATE" in base64
    }
}'

Response:

[
    {
        "key": "U1RBVEU=",
        "value": "AgAAAABpBAAAAAAAAAACAAAAAGsEAAAAAAAAAAIAAAAAdgIAAAABaQoAAAAAAAAAAgAAAAFrCgAAAAAAAAACAAAAAXZaAAAAEwAAAHByaWNlb3JhY2xlLnRlc3RuZXQAAAAlpAAKi8oiBAAAAAAA"
    }
]

Make Fast-Near HTTP call:

Request:

curl --location --request GET 'https://rpc.web4.testnet.page/account/priceoracle.testnet/data/STATE?encoding=base64'

Response:

{
    "data":
    [
     ..996 elements
    ],
    "iterator": "02643a70726963656f7261636c652e746573746e65743a535441544507e78d04"
}

Compare the results:
The value AgAAAABpBAAAAAAAAAACAAAAAGsEAAAAAAAAAAIAAAAAdgIAAAABaQoAAAAAAAAAAgAAAAFrCgAAAAAAAAACAAAAAXZaAAAAEwAAAHByaWNlb3JhY2xlLnRlc3RuZXQAAAAlpAAKi8oiBAAAAAAA returned from the RPC call is missing within Fast-Near's response.

Also i've tried to supply the iterator param in order to fetch the next page but the response is 1:1 the same as without iterator(it seems like the response is cached but not refetched).

https://rpc.web4.testnet.page/account/priceoracle.testnet/data/STATE?encoding=base64&iterator=02643a70726963656f7261636c652e746573746e65743a535441544507e78d04

fastnear / fast-near Goto Github PK

fast-near's Introduction

What is fast-near?

Why

How to

Run directly from npm:

Build and run via yarn:

Build and run with docker:

Pull data into Redis

Pull data into LMDB (experimental)

Load data only for your app's smart contracts

Exclude some undesired accounts

Different data sink options

Load data from a local node

CLI options

Environment variables

HTTP API

Public endpoints

Call view method

POST

Examples

GET

Examples

String parameters:

JSON parameters:

Number parameters (passed as JSON):

Get account info

GET

Example

Get access key info

GET

Example

Download contract WASM code

GET

Example

Get contract methods list

GET

Example

Roadmap

fast-near's People

Contributors

Stargazers

Watchers

Forkers

fast-near's Issues

Recommend Projects

Recommend Topics

Recommend Org