codechain-io / codechain-indexer Goto Github PK

View Code? Open in Web Editor NEW

21.0 7.0 15.0 2.29 MB

A blockchain data indexing tool for CodeChain

License: Apache License 2.0

TypeScript 80.75% Dockerfile 0.09% Shell 0.20% JavaScript 18.96%

codechain indexer

codechain-indexer's Introduction

CodeChain Indexer

A blockchain data indexing tool for CodeChain

Install

Requirements

The software dependencies required to install and run CodeChain-indexer are:

Latest version of the CodeChain
PostgreSQL v11.*
Nodejs higher than version 10

Download

Download CodeChain-indexer code from the GitHub repository

git clone [email protected]:kodebox-io/codechain-indexer.git
cd codechain-indexer

Install packages

Use yarn package manager to install packages

yarn install

Before start

Dependency

Get CodeChain ready with the CodeChain RPC server
Get PostgreSQL database ready for indexing block data

Create the database and user on the PostgreSQL

Update configuration

Update ./create_user_and_db.sql and config/* files before creating database schema. If you want to create a production datatabase, please create config/production.json by copying config/dev.json

Mac

# Download postgresql with Homebrew
brew install postgresql
brew services start postgresql

# Create users and databases
psql postgres -f create_user_and_db.sql

# Create tables for production
NODE_ENV=production yarn migrate

# Create tables for development
yarn run migrate

Ubuntu

# Download postgresql
sudo apt install postgresql postgresql-contrib

# Create users and databases
sudo -u postgres psql -f create_user_and_db.sql

# Create tables for production
NODE_ENV=production yarn migrate

# Create tables for development
yarn migrate

Run (for development)

yarn run start

# You can change the host of CodeChain and DB host on the config/dev.json

Check CCCChanges

If you run the Indexer with the environment ENABLE_CCC_CHANGES_CHECK variable, the Indexer checks to see if CCCChanges is well calculated. If you want to receive an email when an error is found, please set the SENDGRID_API_KEY and SENDGRID_TO variables. If you want to receive a slack notification, please set the SLACK_WEBHOOK variable.

Run (for production)

yarn build
NODE_ENV=production node ./build/index.js

# You can change the host of CodeChain and DB host on the config/production.json

Test

# Create the test database
NODE_ENV=test yarn run migrate

# Start testing
yarn run test

API document

NODE_ENV=dev yarn run start

# Swagger UI is running at "http://host:port/api-docs/"

Tools

Delete all database data

yarn run reset

codechain-indexer's People

Contributors

Stargazers

Watchers

Forkers

zeniel-oroi happyprg sgkim126 majecty hoongee cubismic joojis foriequal0 raccoonrat msntcs sambacha junha1 greatdao

codechain-indexer's Issues

Add an index to the tables

Add a failover logic when an RPC is unavailable

The RPC server can be temporarily unavailable. In that case, the indexer should be able to resume.

Asset image API returns 404

https://corgi.codechain.io/explorer/tx/0xb42cf0be151933f74e6706e29bd9d01c9cd8420c182d471eda3cad260fbb22e1

Add a new endpoint for the chart on the explorer

I need some API to draw the chart on the explorer's main page.

/log/count
params

filter ( ex block, tx, assetTransfer, assetMint, setRegularKey ... (tx types))
date ( ex 2018-09-20)

returns

number ( ex 10)

SequelizeDatabaseError: relation "SetShardUsers" does not exist

Handle the approver and administrator of the transaction

Use Postgresql insteand of Elasticsearch

Add the orderHash and shardId field to the UTXODoc type

Add an assetScheme to the AggsUTXODoc response

Add Order to the transaction

Check a transaction invoice before creating UTXO

Save the asset image when changing an asset scheme

Update the asset scheme after ChangeAssetScheme or IncreaseAssetSupply transaction

Add an API rate limit to the unauthorized client

Chagne the docker-compose, dockerfile script to use PostgreSQL

Add Elasticsearch for searching with the asset name

Add remaining test cases

Ensure database consistency

Ensure data integrity when terminated during indexing

SequelizeDatabaseError: invalid input syntax for type numeric: "0186a0"

This error occurred at 1b68ea1

{ SequelizeDatabaseError: invalid input syntax for type numeric: "0186a0"                                                                                                                                                                
    at Query.formatError (/home/ubuntu/indexer/node_modules/sequelize/lib/dialects/postgres/query.js:363:16)                                                                                                                             
    at query.catch.err (/home/ubuntu/indexer/node_modules/sequelize/lib/dialects/postgres/query.js:86:18)                                                                                                                                
    at tryCatcher (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/util.js:16:23)                                                                                                                           
    at Promise._settlePromiseFromHandler (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/promise.js:512:31)                                                                                                
    at Promise._settlePromise (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/promise.js:569:18)                                                                                                           
    at Promise._settlePromise0 (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/promise.js:614:10)                                                                                                          
    at Promise._settlePromises (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/promise.js:690:18)                                                                                                          
    at _drainQueueStep (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/async.js:138:12)                                                                                                                    
    at _drainQueue (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/async.js:131:9)                                                                                                                         
    at Async._drainQueues (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/async.js:147:5)                                                                                                                  
    at Immediate.Async.drainQueues [as _onImmediate] (/home/ubuntu/indexer/node_modules/sequelize/node_modules/bluebird/js/release/async.js:17:14)                                                                                       
    at runCallback (timers.js:705:18)                                                                                                                                                                                                    
    at tryOnImmediate (timers.js:676:5)                                                                                                                                                                                                  
    at processImmediate (timers.js:658:5)                                                                                                                                                                                                
    at process.topLevelDomainCallback (domain.js:120:23)                                                                                                                                                                                 
  name: 'SequelizeDatabaseError',                                                                                                                                                                                                        
  parent:                                                                                                                                                                                                                                
   { error: invalid input syntax for type numeric: "0186a0"                                                                                                                                                                              
       at Connection.parseE (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:554:11)                                                                                                                                              
       at Connection.parseMessage (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:379:19)                                                                                                                                        
       at Socket.<anonymous> (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:119:22)                                                                                                                                             
       at Socket.emit (events.js:182:13)                                                                                                                                                                                                 
       at Socket.EventEmitter.emit (domain.js:441:20)                                                                                                                                                                                    
       at addChunk (_stream_readable.js:283:12)                                                                                                                                                                                          
       at readableAddChunk (_stream_readable.js:264:11)                                                                                                                                                                                  
       at Socket.Readable.push (_stream_readable.js:219:10)                                                                                                                                                                              
       at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)                                                                                                                                                           
     name: 'error',                                                                                                                                                                                                                      
     length: 108,                                                                                                                                                                                                                        
     severity: 'ERROR',                                                                                                                                                                                                                  
     code: '22P02',                                                                                                                                                                                                                      
     detail: undefined,                                                                                                                                                                                                                  
     hint: undefined,
     position: '207',
     internalPosition: undefined,
     internalQuery: undefined,
     where: undefined,
     schema: undefined,
     table: undefined,
     column: undefined,
     dataType: undefined,
     constraint: undefined,
     file: 'numeric.c',
     line: '621',
     routine: 'numeric_in',
     sql:
      'INSERT INTO "Pays" ("transactionHash","receiver","quantity","createdAt","updatedAt") VALUES (\'bb27d1caed7ad9acf82e30810fea166d432d8aa145ce6579f1b06a74d98994be\',\'wccq8hpt3p988eu5ch62zr92wmd5fjfs7rp354vjgay\',\'0186a0\',\'2019-02-01 09:44:00.218 +00:00\',\'2019-02-01 09:44:00.218 +00:00\') RETURNING *;' },
  original:
   { error: invalid input syntax for type numeric: "0186a0"
       at Connection.parseE (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:554:11)
       at Connection.parseMessage (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:379:19)
       at Socket.<anonymous> (/home/ubuntu/indexer/node_modules/pg/lib/connection.js:119:22)
       at Socket.emit (events.js:182:13)
       at Socket.EventEmitter.emit (domain.js:441:20)
       at addChunk (_stream_readable.js:283:12)
       at readableAddChunk (_stream_readable.js:264:11)
       at Socket.Readable.push (_stream_readable.js:219:10)
       at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
     name: 'error',
     length: 108,
     severity: 'ERROR',
     code: '22P02',
     detail: undefined,
     hint: undefined,
     position: '207',
     internalPosition: undefined,
     internalQuery: undefined,
     where: undefined,
     schema: undefined,
     table: undefined,
     column: undefined,
     dataType: undefined,
     constraint: undefined,
     file: 'numeric.c',
     line: '621',
     routine: 'numeric_in',
     sql:
      'INSERT INTO "Pays" ("transactionHash","receiver","quantity","createdAt","updatedAt") VALUES (\'bb27d1caed7ad9acf82e30810fea166d432d8aa145ce6579f1b06a74d98994be\',\'wccq8hpt3p988eu5ch62zr92wmd5fjfs7rp354vjgay\',\'0186a0\',\'2019-02-01 09:44:00.218 +00:00\',\'2019-02-01 09:44:00.218 +00:00\') RETURNING *;' },
  sql:
   'INSERT INTO "Pays" ("transactionHash","receiver","quantity","createdAt","updatedAt") VALUES (\'bb27d1caed7ad9acf82e30810fea166d432d8aa145ce6579f1b06a74d98994be\',\'wccq8hpt3p988eu5ch62zr92wmd5fjfs7rp354vjgay\',\'0186a0\',\'2019-02-01 09:44:00.218 +00:00\',\'2019-02-01 09:44:00.218 +00:00\') RETURNING *;' }
{ code: 103, message: 'DBError' }

Add the `users` field to CreateShardDoc

The CreateShard spec has been changed.

The assetSchemeDoc doesn't have a "Approvals" field

Remove '0x' from the lockScriptHash on the asset mint transaction

Use Sequelize.ENUM for action

Update README.md

Use "keyword" type for the amount of the asset in the Elasticsearch

Yarn run test fails if the instructions are followed

FAIL test/pending-tx.spec.ts
FAIL test/db-block-interface.spec.ts
FAIL test/mint-transfer.spec.ts
FAIL test/snapshot.spec.ts (31.591s)
FAIL test/wrap-unwrap.spec.ts (5.398s)
FAIL test/store-and-remove.spec.ts (6.713s)
FAIL test/log.spec.ts (13.904s)

Test Suites: 7 failed, 4 passed, 11 total
Tests: 21 failed, 19 passed, 40 total
Snapshots: 0 total
Time: 96.003s
Ran all test suites.

Tested in
Ubuntu 18.04.1 LTS
nodejs v10.15.1
psql (PostgreSQL) 11.1 (Ubuntu 11.1-3.pgdg18.04+1)
codechain commit 7e689aa86d95f100cc121837cb076b4e77e4957f ( Feb 7, 2019 )
index commit e0f5542 ( Feb 3, 2019 )

for more information, please check the link

Add a validation tool like Joi to the API routes

https://github.com/hapijs/joi

Error: createShard is not an expected transaction type

Update the docker file and docker-compose file

Support store/remove actions for Log API

Add a sync parameter to the API

Now the worker synchronizes the CodeChain and DB on every 10sec. Add waitForRefresh parameter so the user can get the latest data.

Support increaseAssetSupply transaction

The "/tx" API filtered with the assetType doesn't return the changeAssetScheme

increaseAssetSupply too.

Get transactions with order owner or order assetType is not available.

I temporarily removed that feature in fd0d34c to fix bug.

For more information about the bug, please check this link

InvalidTransaction Error

{ code: 104, message: 'InvalidTransaction' }

Currently, the indexer fails to get an invoice in some cases. (In the corgi network, the TransferAsset tx in the block 1410)

I did some debug, and, I found that the tracker values(the SDK one and the CodeChain one) do match but the hash values don't.

CodeChain log:

#815501000 2019-02-08 22:01:38  http.worker0 INFO tx  Cannot apply InnerTx(0x11f2…9ebe): AssetNotFound(0x410000005ddc08b053c6c943a020f7aae2cd1618f23f2992d4836c855c7c8d0f)
parcel_address_entries ParcelHash: 0x82c1626da1289f38e2f05a7529d3422a2c0aa5980825a443cad6b572f110a3c2
parcel_address_entries ParcelAddress: ParcelAddress { block_hash: 0xd164a4628e0267d4610d4552d43ffd908d35021ce3400e4a1352ffa1334b4a22, index: 0 }

indexer log:

tx hash (ParcelHash):
a983019920bb5da87e76cac900558c8af02d6c6349eb46b7ea41303087c12167
tx tracker (InnerTx):
11f21e6077e2545ef7278ec15c4d5c176d89249e4b78c797345644b9c5279ebe

I need to find why the hash values don't match. It may be an SDK problem.

server log

SequelizeDatabaseError: column transferAsset->orders.owner does not exist

The assetType can't be a primary key

The same assetType with a different scheme can exist across shards.

Test indexing all types of transactions

MintAsset
TrransferAsset
ChangeAssetScheme
IncreaseAssetSupply
WrapCCC
UnwrapCCC
Pay
SetRegularKey
CreateShard
SetShardOwners
SetShardUsers
Store
Remove
Custom