0xpolygon / polygon-edge Goto Github PK

View Code? Open in Web Editor NEW

991.0 991.0 511.0 156.09 MB

A Framework for Building Ethereum-compatible Blockchain Networks

Home Page: https://polygon.technology/solutions/polygon-edge/

License: Apache License 2.0

Makefile 0.09% Go 97.09% Solidity 2.27% Shell 0.40% Dockerfile 0.01% JavaScript 0.13%

blockchain ethereum protocol

polygon-edge's Introduction

Update: Edge v1.0.0 is here!

Developers at Polygon Labs have been hard at work gathering and incorporating community feedback into the Edge client and a new version of Edge, v1.0.0, is here with several new features! Check out the Release Notes included with the release to find out more!

Polygon Edge

Polygon Edge is a modular and extensible framework for building Ethereum-compatible blockchain networks.

To find out more about Polygon, visit the official website.

WARNING: This is a work in progress so architectural changes may happen in the future. The code is still being audited, so please contact the Polygon team if you would like to use it in production.

Documentation 📝

If you'd like to learn more about the Polygon Edge, how it works and how you can use it for your project, please check out the Polygon Supernets Documentation.

Disclaimer

As this project evolves, the Polygon Labs developer team will focus on the latest version of the Edge client and does not plan to support Edge 0.6 or lower. It is highly recommended that you upgrade to the newest version with the most up-to-date features and fixes. Users that want to stay on 0.6 or below, can continue to do so. The repo will continue to exist and users can fork it and do with it as they please, subject to applicable open-source license terms.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

polygon-edge's People

Contributors

Stargazers

Watchers

Forkers

tenderly shard-labs metaver5o stavrosd slvc nicovogelaar dimajoyti tuanha1305 baki91-3663 i1101111000 alvarlaigna heiho1 kbnorth mitchelltesla p7-33 tai-bcw cybertronchain teslow defcon201 song0987 mad0perator bow0628 genego-dev ny030897 khaiphan11022 rlr-github rfarrey astro44 devilla dankostiuk oursin mrwillis enki-y mkos11 cryptonightcanada skalarfeilds79 thewebchap doraemon93 ocashlab casualuser nungning093 sx-network forkkit hecod jhfnetboy kennashka marshmell0w fave01 bindbob78 chuwt askdaddy bwhour bsix5 dariuszo koiplur mitrabineka1 maticnetwork fairhopeweb oxanayar88 katerega inderashah007 jbuttz m01k0 iias antimonyiq abrdaa settlemint wartstone cfwhisperer lwandsyj blunt-enterprises ross1219 suryatmodulus jackvoorham alvaro061118 johan-kok udaragangabadage sojiritesh2003 diadetedio thebojda omarm123 diegofornalha nhlanhlahasane menacingdummy569 exonetfoundation cryptorites-scotti bitzlato akshar fgimenez verde888 richardsonjf pearchain simond110 developerayo big-run-studios git-yongge varshans2 etm-proterozoic omkar-mohanty tumtum23

polygon-edge's Issues

Panic using `--libp2p 0.0.0.0:port` option

Panic using `--libp2p` option

Description

Running polygon-sdk server --libp2p 0.0.0.0:10001 fails without passing a --nat

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 96be904
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Run polygon-sdk server --libp2p 0.0.0.0:10001 with a genesis file
Where the issue is, if you know
AddrInfoToString seems to not like when there are two addresses in the libp2p address info. It's giving me two addresses,

// AddrInfoToString converts an AddrInfo into a string representation that can be dialed from another node
func AddrInfoToString(addr *peer.AddrInfo) string {
	fmt.Print(addr)
	if len(addr.Addrs) != 1 {
		panic("Not supported")
	}
	return addr.Addrs[0].String() + "/p2p/" + addr.ID.String()
}

Gives

{16Uiu2HAm7dBcv1avSV8TVFZQV4kUNYi5N1jHu7C4xH1gV58McZx2: [/ip4/192.168.1.248/tcp/10002 /ip4/127.0.0.1/tcp/10002]}

Logs

panic: Not supported

goroutine 1 [running]:
github.com/0xPolygon/polygon-sdk/network.AddrInfoToString(0xc0005a04e0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/network/server.go:595 +0x1f7
github.com/0xPolygon/polygon-sdk/network.NewServer(0x1582748, 0xc0002f4cb0, 0xc000100eb0, 0x0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/network/server.go:138 +0xe71
github.com/0xPolygon/polygon-sdk/server.NewServer(0x1582748, 0xc0002f4b60, 0xc0002f8f80, 0x0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/server/server.go:92 +0x5ef
github.com/0xPolygon/polygon-sdk/command/server.(*ServerCommand).Run(0xc00000ed08, 0xc000032260, 0x4, 0x4, 0x0)
	/home/julian/sx-network/polygon-sdk/command/server/server_command.go:187 +0x3de
github.com/mitchellh/cli.(*CLI).Run(0xc00013f7c0, 0x0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/vendor/github.com/mitchellh/cli/cli.go:255 +0xb7d
main.Run(0xc000032250, 0x5, 0x5, 0x0)
	/home/julian/sx-network/polygon-sdk/main.go:25 +0x151
main.main()
	/home/julian/sx-network/polygon-sdk/main.go:12 +0x7c

Proposed solution

If you pass a --nat, this error disappears. I feel like this should not be a panic.

`eth_getTransactionCount` is failing on non initialized accounts

Description

Calling eth_getTransactionCount on a non-initialized account (hasn't done any txs) in the state tree returns

{
    "code": -32603,
    "message": "internal error"
}

Your environment

OS and version Ubuntu 20
version of the Polygon SDK b8659d1
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Run polygon-sdk in dev mode and POST the json rpc endpoint with the below payload in the logs
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

{
    "jsonrpc": "2.0",
    "id": 5482,
    "result": "0x0"
}

Actual behaviour

{"code":-32603,"message":"internal error"}

Logs

2021-08-18T13:11:04.947-0400 [DEBUG] polygon-dev.jsonrpc: handle: request="{
  "jsonrpc" : "2.0",
  "id" : 5482,
  "method" : "eth_getTransactionCount",
  "params" : ["0x321395A4822f109Ae167987a6b0566cF91126F7e"]
}"
2021-08-18T13:11:04.947-0400 [DEBUG] polygon-dev.dispatcher: request: method=eth_getTransactionCount id=5482
2021-08-18T13:11:04.947-0400 [ERROR] polygon-dev.dispatcher: failed to dispatch: method=eth_getTransactionCount err="given root and slot not found in storage"

Proposed solution

It seems like getState in server.go always complains if it can't find the storage slot. Perhaps returning an error all the time is not the best approach as this is a valid request and should give a valid response (of 0). Or maybe we can switch on the error type?

Error `polygon.consensus.syncer: failed to notify` after adding + removing peers/validators

Description

I've seen many occurrences of this new error after adding and removing peers and validators to the original validator set:

2021-09-10T16:32:00.056Z [ERROR] polygon.consensus.syncer: failed to notify: err="rpc error: code = Unavailable desc = connection error: desc = "transport: failed to write client preface: stream reset""

It seems as if the active nodes may somehow still maintain a reference to old (removed) peers and/or validators even so they are no longer visible when running the SDK commands:

./main peers list

and

./main ibft snapshot

I've made sure to run ./main ibft propose --vote drop on greater than 50% of active validators (removing it from the list that ibft snapshot returns, as expected) but it seems I only see an even greater frequency of the same error -- for reference below, validators were removed at 12:08 and 12:14:

Your environment

OS and version OSX Big Sur 11.5.2 (20G95)
version of the Polygon SDK 0b2cc26
branch that causes this issue develop

Steps to reproduce

Run ibft server for 4 bootnodes
Add peer new_node1 using peers add
Vote in new validator new_node1 using ibft propose --vote auth
Vote in new validator new_node2 using ibft propose --vote auth
Stop validators new_node1, new_node2
Drop both new validators new_node1, new_node2 using ibft propose --vote drop
Verify that peers list shows 3 peers for each node (as expected for 4 bootnodes)
Verify that ibft snapshot shows 4 validator addresses total (as expected for 4 bootnodes)
See new errors in logs

Expected behaviour

I shouldn't any of these errors.

Actual behaviour

I see many of these errors in the logs on all 4 bootnodes.

I initially suspected that the validators were stopped prematurely without being dropped by the active validators but when attempting to remove them from the validator list, it seems that I only see more of the same error in the logs.

Contract correct deployed but contract functions gives error

Contract deploy

Contract is correct compiled and deployed but call contract functions is with error.

// SPDX-License-Identifier: GPL-3.0
pragma solidity ^0.8.4;

contract Epicpic2{
mapping (uint256 => string) pic;
uint256 x=0;
address owner;
constructor() {
owner = msg.sender;
}

  function save(string memory s) public{
        require(msg.sender == owner);
        pic[x]=s;
        x++;
    }
    
function getpic(uint256 i)  public view returns (string memory){
        return pic[i];
    }

 function getlastx() public view returns (uint256){
        return x;
    }

}

Description

Like above, maybe problem with strings ?

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

`eth_call` without `from` field fails during nextNonce calculation in decodeTxn

Description

When making an http request to the eth_call endpoint and the from field isn't provided, we now override the from address with a ZeroAddress after recent changes last week (see 3865718).

Further downstream in dispatcher.decodeTxn however, we attempt to call getNextNonce with the new from address (which is now a 0x0 address) and as a result encounter the "given root and slot not found in storage" error after failing to get the snapshot of the 0x0 address.

Your environment

OS and version OSX 11.4
version of the Polygon SDK 3a72f15
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Make request to eth_call with from missing from the params, for example:

{"id":"28eee10a-18db-40ce-a924-37d2f9f69631","jsonrpc":"2.0","method":"eth_call","params":[{"to":"0x011e4e51998e97bd74d0a8ae08b428f51b58eaba","data":"0x70a0823100000000000000000000000061a197c895ff5ceaacf7f800746541f99da4fe79"},"0xec"]}

(this was found using a local instance of https://metamask.github.io/test-dapp/ and attempting to add token to wallet)

Expected behaviour

We should not see a "given root and slot not found in storage" error message and a {"code":-32603,"message":"internal error"} response when trying to add a token to MetaMask (calling eth_call)

Actual behaviour

We see a "given root and slot not found in storage" error message and a {"code":-32603,"message":"internal error"} response.

Proposed solution

We probably need to default to a nonce value of 0 within the decodeTxn function so that we can reach the ApplyTxn logic.

panic on concurrent write to JSON-RPC websocket

Description

If writes are concurrent to the gorilla/websocket writer, the writer panics presumably due to not allowing multiple concurrent writes.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK ad7b97e
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue

Start in dev mode
Where the issue is, if you know

Panic is here in conn.go in gorilla/websocket:

// Write the buffers to the connection with best-effort detection of
	// concurrent writes. See the concurrency section in the package
	// documentation for more info.

	if c.isWriting {
		panic("concurrent write to websocket connection")
	}
	c.isWriting = true

	err := c.write(w.frameType, c.writeDeadline, c.writeBuf[framePos:w.pos], extra)

	if !c.isWriting {
		panic("concurrent write to websocket connection")
	}
	c.isWriting = false

	if err != nil {
		return w.endMessage(err)
	}

	if final {
		w.endMessage(errWriteClosed)
		return nil
	}

Which commands triggered the issue, if any
polygon-sdk dev. Then start a request-heavy client, like a block explorer and point to ws://localhost:8545/ws. You'll eventually hit the error

Expected behaviour

SDK should not crash

Actual behaviour

SDK crashes

Logs

Exception has occurred: panic
"concurrent write to websocket connection"
Stack:
	2  0x0000000000eb88f6 in github.com/gorilla/websocket.(*messageWriter).flushFrame
	    at /home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:610
	3  0x0000000000eb9d65 in github.com/gorilla/websocket.(*Conn).WriteMessage
	    at /home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:763
	4  0x0000000001010b49 in github.com/0xPolygon/minimal/jsonrpc.(*wsWrapper).WriteMessage
	    at /home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:116
	5  0x000000000101a0f9 in github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handleWs.func3
	    at /home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:186

	    panic: concurrent write to websocket connection

goroutine 111 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0003a7d78, 0xc000040201, 0xc0000402d0, 0x0, 0x0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:610 +0xad6
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc000130840, 0x1, 0xc0000402d0, 0x0, 0x0, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:763 +0x2e5
github.com/0xPolygon/minimal/jsonrpc.(*wsWrapper).WriteMessage(0xc0006ae980, 0x1, 0xc0000402d0, 0x50, 0x50, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:116 +0x89
github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handleWs.func3(0xc000693170, 0xc000542e00, 0x4f, 0x200, 0xc0006ae980, 0x1)
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:186 +0x499
created by github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handleWs
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:176 +0x80f
panic: concurrent write to websocket connection

goroutine 166 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc000557d78, 0xc00048c301, 0xc00048c411, 0x0, 0xf, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:610 +0xad6
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc000130840, 0x1, 0xc00048c411, 0x0, 0xf, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/vendor/github.com/gorilla/websocket/conn.go:763 +0x2e5
github.com/0xPolygon/minimal/jsonrpc.(*wsWrapper).WriteMessage(0xc0006ae980, 0x1, 0xc00048c3c0, 0x51, 0x60, 0x0, 0x0)
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:116 +0x89
github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handleWs.func3(0xc000693170, 0xc000488600, 0x50, 0x200, 0xc0006ae980, 0x1)
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:186 +0x499
created by github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handleWs
	/home/julian/sx-network/polygon-sdk/jsonrpc/jsonrpc.go:176 +0x80f

Proposed solution

So there's a note in the code here about concurrency options but it doesn't look it will do much. I've implemented this

gorilla/websocket#119

here

#114

which solves the issue. There doesn't seem to be any performance impact since this is just writing the data to the writer (processing is done way before this).

Need I ethereum node local working to use ? - go run main.go peers list

go run main.go peers list

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8545: connect: connection refused"
exit status 1

Ubuntu

Could not parse NAT IP address

Description

Describe your issue in as much detail as possible here

Your environment

tried it on macOS 11.4 (M1) and "5.4.0-1047-azure #49-Ubuntu" 20.04 LTS
go version mac: go version go1.16.4 darwin/amd64
go version ubuntu: go version go1.13.8 linux/amd64
same error on both machines
sdk version: 0.1.0-dev
develop branch

Steps to reproduce

When I try to start the network after following the docs with:
go run main.go server --data-dir ./test-chain-1 --chain genesis.json --grpc :10000 --libp2p :10001 --jsonrpc :10002 --seal
i get the error message:
Could not parse NAT IP address
exit status 1

Expected behaviour

The nodes should start

Actual behaviour

Could not parse NAT IP address
exit status 1

Logs

My genesis.json (with x-outed Node IDs):
{
"name": "test-chain-",
"genesis": {
"nonce": "0x0000000000000000",
"timestamp": "0x0",
"extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f858f85494ead46e992b0aae3260b3962f90b7fc5aab4e909e948636bfa3b6c05e9f6297aa6a9112999ef0a91b1694b768bc1f50755432b3e439c240cd86c1c0b8d3de94f1ddce413a857320ac088089831860b330e9e60980c0",
"gasLimit": "0x1388",
"difficulty": "0x1",
"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"coinbase": "0x0000000000000000000000000000000000000000",
"alloc": {},
"number": "0x0",
"gasUsed": "0x0",
"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
},
"params": {
"forks": {
"homestead": 0,
"byzantium": 0,
"constantinople": 0,
"petersburg": 0,
"istanbul": 0,
"EIP150": 0,
"EIP158": 0,
"EIP155": 0
},
"chainID": 100,
"engine": {
"ibft": {}
}
},
"bootnodes": [
"/ip4/127.0.0.1/tcp/10001/p2p/16Uiu2mxxxxxxxxxxxxxx",
"/ip4/127.0.0.1/tcp/20001/p2p/16Uiu2xxxxxxxxxxxx",
"/ip4/127.0.0.1/tcp/30001/p2p/16Uiu2xxxxxxxxx",
"/ip4/127.0.0.1/tcp/40001/p2p/16Uiu2xxxxxxxxxxxx"
]
}

Proposed solution

Most likely the error message comes from the server/config.go line 90-92. Seems the IP cannot be parsed.

where is Polygon SDK Documentation github link

hi, where is Polygon SDK Documentation github link so people can participate in the contribution.

big amount in Metamask wallet makes problem

[Subject of the issue]

Big value in premine genesis .... 0x115EEC47F6CF7E35000000 (21000000 whole number in Metamask )

Description

After creation new network with premine amount like above in address it give big number in Metamask connected by rpc to network. When made real transaction from Metamask ( small value like 1 or 5 ) it is still pending. Not this problem if starting amount on account is default ( 1000 ).

Your environment

OS and version - Ubuntu Fossa
version of the Polygon SDK develop today
branch that causes this issue

Steps to reproduce

like above

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it
Maybe too big value from Metamask this starting point and problems with number in javascript?

too many arguments - looks like problem when arguments add to subcommand

[Subject of the issue]

Description

go run main.go peers list --grpc-address localhost:30000

works correct

go run main.go ibft propose --add 0x2a70a391e65a59e03bf3661502c8a79a1d267ab3 --grpc-address localhost:30000

too many arguments

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Limit to MaxPeers the incoming connections

[Subject of the issue]

There is a MaxPeers value that limits the current number of peers we are connected or pending to be connected (in handshake). However, this limit is only respected for outbound connections. We should limit the inbound connections too, this is, if a new connection arrives and we have MaxPeers we should not accept the connection.

Description

Describe your issue in as much detail as possible here

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Bootnode validator not able to sync up to latest block

Description

It seems our 5-validator testnet got stuck and encountered a state where some of our node's latest block numbers fell out of sync, preventing the network from generating new blocks (as 4 validators are required).

node-4 in our case crashed at block 0xb60d1 and couldn't sync up to 0xb60d7, the latest block on node-1.

While investigating, I tried to completely wipe everything within the data-dir of node-4 hoping that it'll be able to sync up to the latest block on node-1 (0xb60d7) from a clean slate:

rm -rf blockchain keystore trie consensus/metadata consensus/snapshots

However upon doing so and re-running server on node-4, the logs show the node to keep falling back into RoundChangeState:

Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.362Z [INFO]  polygon.blockchain: genesis: hash=0x24d80debadd21a42ae8bcc3b28d39a0e2e2cbf9d7fee3507c1e38a33995fc3f5
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.363Z [INFO]  polygon: GRPC server running: addr=0.0.0.0:10000
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.363Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:10002
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [DEBUG] polygon.consensus.ibft: state change: new=SyncState
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [DEBUG] polygon.consensus.ibft: current sequence: sequence=1
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [DEBUG] polygon.consensus.ibft: state change: new=AcceptState
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [DEBUG] polygon.consensus.ibft: cycle: state=AcceptState sequence=1 round=0
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=1
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
Sep 22 18:18:57 ip-172-31-32-129 main[1131978]: 2021-09-22T18:18:57.377Z [INFO]  polygon.consensus.ibft: proposer calculated: proposer=0x4FAC0232243D2C863EF7787e0cb9c9118048d60e block=1
Sep 22 18:19:02 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:02.350Z [DEBUG] polygon.network: dial: local=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW addr="{16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh: [/ip4/3.215.13.226/tcp/10001]}"
Sep 22 18:19:02 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:02.384Z [DEBUG] polygon.network: dial: local=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW addr="{16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW: [/ip4/204.236.222.207/tcp/10001]}"
Sep 22 18:19:02 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:02.384Z [DEBUG] polygon.network: dial: local=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW addr="{16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x: [/ip4/34.207.228.138/tcp/10001]}"
Sep 22 18:19:02 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:02.390Z [DEBUG] polygon.network: dial: local=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW addr="{16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU: [/ip4/3.84.231.38/tcp/10001]}"
Sep 22 18:19:02 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:02.395Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
Sep 22 18:19:07 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:07.353Z [DEBUG] polygon.network: dial: local=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW addr="{16Uiu2HAkwhNtm8iXMoWR7hmVCK3on6Rhz2DM8V6VhQm9bkdUwq1d: [/ip4/54.167.213.167/tcp/10001]}"
Sep 22 18:19:07 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:07.378Z [DEBUG] polygon.consensus.ibft: state change: new=RoundChangeState
Sep 22 18:19:07 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:07.378Z [DEBUG] polygon.consensus.ibft: cycle: state=RoundChangeState sequence=1 round=0
Sep 22 18:19:07 ip-172-31-32-129 main[1131978]: 2021-09-22T18:19:07.378Z [DEBUG] polygon.consensus.ibft: local round change: round=1

I then attempted to reset our non-bootnode (node-5) back to block 0 to test out whether or not the bug exists in the actual syncing and upon restarting, I could immediately see node-5 attempt to sync up to node-1:

Sep 22 18:13:04 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:04.993Z [INFO]  polygon.blockchain: genesis: hash=0x24d80debadd21a42ae8bcc3b28d39a0e2e2cbf9d7fee3507c1e38a33995fc3f5
Sep 22 18:13:04 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:04.994Z [INFO]  polygon: GRPC server running: addr=0.0.0.0:10000
Sep 22 18:13:04 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:04.995Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:10002
Sep 22 18:13:10 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:10.005Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
Sep 22 18:13:10 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:10.014Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
Sep 22 18:13:11 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:11.012Z [INFO]  polygon.blockchain: write blocks: num=10 from=1 to=10 parent=0x24d80debadd21a42ae8bcc3b28d39a0e2e2cbf9d7fee3507c1e38a33995fc3f5
Sep 22 18:13:11 ip-172-31-86-180 main[77785]: 2021-09-22T18:13:11.031Z [INFO]  polygon.blockchain: new block: number=1 hash=0x0b0278e54850e679e866e14dd7e238208c16e4abed2b8f0d5ea5c82a29ce9ff9 txns=0 generation_time_in_sec=1630687894

I'm not sure why node-4 crashed in the first place but based on what I observed above, it seems that there is in issue when trying to sync up nodes that are within the genesis.json.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 4afea94
branch that causes this issue develop

Steps to reproduce

Have 5 nodes running as validators (4 bootnodes + 1 non-bootnode)
Kill one of the 4 bootnodes (e.g. node-4)
Vote off the non-bootnode so that it is no longer a validator
Start up node-4 and observe that it cannot sync up to the latest block
Attempt to clear the history of node-4 and attempt to sync, observing the same issue

Expected behaviour

Nodes (within or outside the bootnode validator set) should be able to sync up to the latest block when they fall behind (due to a crash).

Actual behaviour

One of the 4 bootnode validators could not sync up to the latest block (it remained in RoundChangeState).

Logs

See Description

Panic due to libp2p addresses conversion causes node crash

Description

After running a node on a testchain for a while, randomly the node panics with the same error as in #160

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 18e9317
branch that causes this issue fix/sync-issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs



2021-09-27T21:28:24.149Z | Sep 27 21:28:22 ip-172-31-44-61 web: 2021-09-27T21:28:22.058Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=816768 hash=0xc565431ff24d2860a3752b65a3ff06520cc1c7396b87939891d784cdfffe6212
-- | --
  | 2021-09-27T21:28:24.149Z | Sep 27 21:28:24 ip-172-31-44-61 web: 2021-09-27T21:28:24.021Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=816769 hash=0x1754444b09e71fc569b62a960a08d38ea7d05847ff5be90b20caaba4a6543e6d
  | 2021-09-27T21:28:24.149Z | Sep 27 21:28:24 ip-172-31-44-61 web: 2021-09-27T21:28:24.023Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=816769 hash=0x1754444b09e71fc569b62a960a08d38ea7d05847ff5be90b20caaba4a6543e6d
  | 2021-09-27T21:28:24.149Z | Sep 27 21:28:24 ip-172-31-44-61 web: 2021-09-27T21:28:24.023Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=816769 hash=0x1754444b09e71fc569b62a960a08d38ea7d05847ff5be90b20caaba4a6543e6d
  | 2021-09-27T21:28:25.399Z | Sep 27 21:28:24 ip-172-31-44-61 web: 2021-09-27T21:28:24.025Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=816769 hash=0x1754444b09e71fc569b62a960a08d38ea7d05847ff5be90b20caaba4a6543e6d
  | 2021-09-27T21:28:25.399Z | Sep 27 21:28:25 ip-172-31-44-61 web: 2021-09-27T21:28:25.332Z [DEBUG] polygon.jsonrpc: handle: request={"method":"eth_blockNumber","params":[],"id":55,"jsonrpc":"2.0"}
  | 2021-09-27T21:28:25.399Z | Sep 27 21:28:25 ip-172-31-44-61 web: 2021-09-27T21:28:25.332Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=55
  | 2021-09-27T21:28:26.150Z | Sep 27 21:28:25 ip-172-31-44-61 web: 2021-09-27T21:28:25.332Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":55,"result":"0xc6ee6"}
  | 2021-09-27T21:28:26.150Z | Sep 27 21:28:26 ip-172-31-44-61 web: 2021-09-27T21:28:26.022Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=816770 hash=0x515634a2ab038cbd6228e1dd23cec46ae57e207905dc5c906e5c618bfd5635b0
  | 2021-09-27T21:28:26.150Z | Sep 27 21:28:26 ip-172-31-44-61 web: 2021-09-27T21:28:26.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=816770 hash=0x515634a2ab038cbd6228e1dd23cec46ae57e207905dc5c906e5c618bfd5635b0
  | 2021-09-27T21:28:26.150Z | Sep 27 21:28:26 ip-172-31-44-61 web: 2021-09-27T21:28:26.039Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=816770 hash=0x515634a2ab038cbd6228e1dd23cec46ae57e207905dc5c906e5c618bfd5635b0
  | 2021-09-27T21:28:27.400Z | Sep 27 21:28:26 ip-172-31-44-61 web: 2021-09-27T21:28:26.040Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=816770 hash=0x515634a2ab038cbd6228e1dd23cec46ae57e207905dc5c906e5c618bfd5635b0
  | 2021-09-27T21:28:27.400Z | Sep 27 21:28:27 ip-172-31-44-61 web: 2021-09-27T21:28:27.204Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232328811,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T21:28:27.400Z | Sep 27 21:28:27 ip-172-31-44-61 web: 2021-09-27T21:28:27.204Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232328811e+15
  | 2021-09-27T21:28:28.151Z | Sep 27 21:28:27 ip-172-31-44-61 web: 2021-09-27T21:28:27.204Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232328811,"result":"0xc6ee6"}
  | 2021-09-27T21:28:28.151Z | Sep 27 21:28:28 ip-172-31-44-61 web: 2021-09-27T21:28:28.019Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=816771 hash=0xb37a2cb263b2a3d902bba22a2bc6b68704b9d59900609e3f97a3d4086363457c
  | 2021-09-27T21:28:28.151Z | Sep 27 21:28:28 ip-172-31-44-61 web: 2021-09-27T21:28:28.020Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=816771 hash=0xb37a2cb263b2a3d902bba22a2bc6b68704b9d59900609e3f97a3d4086363457c
  | 2021-09-27T21:28:28.151Z | Sep 27 21:28:28 ip-172-31-44-61 web: 2021-09-27T21:28:28.022Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=816771 hash=0xb37a2cb263b2a3d902bba22a2bc6b68704b9d59900609e3f97a3d4086363457c
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:28 ip-172-31-44-61 web: 2021-09-27T21:28:28.026Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=816771 hash=0xb37a2cb263b2a3d902bba22a2bc6b68704b9d59900609e3f97a3d4086363457c
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: panic: Not supported
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: goroutine 23514 [running]:
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: github.com/0xPolygon/polygon-sdk/network.AddrInfoToString(0xc003d65a98)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/network/server.go:594 +0xa7
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: github.com/0xPolygon/polygon-sdk/network.(*discovery).FindPeers(0xc0004a4300, {0x10f0fb0, 0xc0043c9800}, 0xc0034d2280)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/network/discovery.go:255 +0x30d
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: github.com/0xPolygon/polygon-sdk/network/proto._Discovery_FindPeers_Handler.func1({0x10f0fb0, 0xc0043c9800}, {0xd399c0, 0xc0034d2280})
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/network/proto/discovery_grpc.pb.go:82 +0x78
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: github.com/0xPolygon/polygon-sdk/network/grpc.interceptor({0x10f0d80, 0xc0036f3950}, {0xd399c0, 0xc0034d2280}, 0xc003d65bb8, 0xc0042cf200)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/network/grpc/grpc.go:53 +0xef
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: github.com/0xPolygon/polygon-sdk/network/proto._Discovery_FindPeers_Handler({0xd513e0, 0xc0004a4300}, {0x10f0d80, 0xc0036f3950}, 0xc004c33c20, 0x1006ed8)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/network/proto/discovery_grpc.pb.go:84 +0x138
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003508c0, {0x10ff4f0, 0xc00013b800}, 0xc001c3af00, 0xc00048b800, 0x197d360, 0x0)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/vendor/google.golang.org/grpc/server.go:1217 +0xc8f
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: google.golang.org/grpc.(*Server).handleStream(0xc0003508c0, {0x10ff4f0, 0xc00013b800}, 0xc001c3af00, 0x0)
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/vendor/google.golang.org/grpc/server.go:1540 +0xa2a
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: google.golang.org/grpc.(*Server).serveStreams.func1.2()
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/vendor/google.golang.org/grpc/server.go:878 +0x98
  | 2021-09-27T21:28:30.404Z | Sep 27 21:28:30 ip-172-31-44-61 web: created by google.golang.org/grpc.(*Server).serveStreams.func1
  | 2021-09-27T21:28:30.654Z | Sep 27 21:28:30 ip-172-31-44-61 web: /var/app/staging/polygon-sdk/vendor/google.golang.org/grpc/server.go:876 +0x294

Proposed solution

I found this same error in the linked issue. Perhaps a node is somehow managing to set two addresses or something?

Bulk syncing failing due to fork or incorrectly sent data

Description

When syncing a node from a fresh state, it seems that it does not know how to handle forks correctly. Block 929697 is when the fork starts. The block hash for 929697 on the canonical chain is 0x107a2916feb361e77ab753db582c8d206c34bc7429370b64396b3eebd48842ac but 929698 is expecting 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d which doesn't exist

Is it possible the node it is syncing from is storing an incorrect chain?

Your environment

OS and version Debian 11
version of the Polygon SDK 335e781
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
WriteBlocks() in BulkSyncWithPeer()
Which commands triggered the issue, if any

/home/$USER/validator/node/main server --data-dir /home/$USER/validator/node/mynode --chain /home/$USER/validator/node/genesis.json --grpc 0.0.0.0:10000 --libp2p 0.0.0.0:10001 --jsonrpc 0.0.0.0:10002 --nat $IP --seal

Expected behaviour

The node should sync normally up to the current block

Actual behaviour

The node gets stuck on block 929697

Logs

Node 1 trying to sync

Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.925Z [INFO]  polygon.blockchain: new block: number=929682 hash=0x273ee8e6b14a410796ca579efd41fcef8730a2ab4755dc02f1fe43c4c67ab67b txns=1 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.926Z [INFO]  polygon.blockchain: new block: number=929683 hash=0x0a84b7292f2c578481a1ba60b4f3573f44bf069bedaf4fc6b4f45659a72eb435 txns=0 generation_time_in_sec=10
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.926Z [INFO]  polygon.blockchain: new block: number=929684 hash=0xcacaada2216dc16a2cba2684725e6bd65f73f728e360196e3cd46e79d743959b txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.927Z [INFO]  polygon.blockchain: new block: number=929685 hash=0x5aea6156f8f1c50ae2cee8807b77cdf35c3d84370eaf3ec1c6f80bcf425b4823 txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.928Z [INFO]  polygon.blockchain: new block: number=929686 hash=0x744e81c4227c8eb185857701fa98d7e99234085111daae5162c1835ed87c4dcd txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.928Z [INFO]  polygon.blockchain: new block: number=929687 hash=0x6f7863e883ccc53d69ba1b590936146308188351bbacde35523b3787e3712b90 txns=0 generation_time_in_sec=10
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.928Z [INFO]  polygon.blockchain: new head: hash=0x6f7863e883ccc53d69ba1b590936146308188351bbacde35523b3787e3712b90 number=929687
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.928Z [INFO]  polygon.blockchain: write blocks: num=10 from=929688 to=929697 parent=0x6f7863e883ccc53d69ba1b590936146308188351bbacde35523b3787e3712b90
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.946Z [INFO]  polygon.blockchain: new block: number=929688 hash=0x6e10907711cce5a5bbf958ffe09e3d80a9b31324649af46d2999f1d218ea425d txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.946Z [INFO]  polygon.blockchain: new block: number=929689 hash=0x4005052dee7146a1a0e36c615935cc3d0ae9db7dcb20d5edf4949803431fe3a4 txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.947Z [INFO]  polygon.blockchain: new block: number=929690 hash=0xa92ac10b4fd7e9c20cfcf4e9d4a5b1eb3579f424c153d209ae7d5891082a5971 txns=1 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.948Z [INFO]  polygon.blockchain: new block: number=929691 hash=0xdb20e30540903461c0531c364d374668b6c76d2a476cd70d0159f190fbcb0ff3 txns=0 generation_time_in_sec=10
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.948Z [INFO]  polygon.blockchain: new block: number=929692 hash=0x808e93329dafa8ef47724e999c99d1e7934328b5b60f4e0b10f2e155c62365ec txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.949Z [INFO]  polygon.blockchain: new block: number=929693 hash=0x1bfcbe5debfd343f358d0e98fa61734fee8b135b14a40771c88df35049fc01a3 txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.949Z [INFO]  polygon.blockchain: new block: number=929694 hash=0xc18950225162eedb6cd6389096f78f1f7ab00be8c0d69ef1eb54f041c8b8e448 txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.950Z [INFO]  polygon.blockchain: new block: number=929695 hash=0xdd0a374030ea24430e32b7a3d1b826db2aac78728c1e6078c0ea3ae6a3b638e8 txns=0 generation_time_in_sec=10
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.950Z [INFO]  polygon.blockchain: new block: number=929696 hash=0xd943d00fffa9d2c84c336db0b1362f624234c8d4716ff587d0c0f69cc43e8f32 txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.951Z [INFO]  polygon.blockchain: new block: number=929697 hash=0x107a2916feb361e77ab753db582c8d206c34bc7429370b64396b3eebd48842ac txns=0 generation_time_in_sec=2
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.951Z [INFO]  polygon.blockchain: new head: hash=0x107a2916feb361e77ab753db582c8d206c34bc7429370b64396b3eebd48842ac number=929697
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.951Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:18 instance-1 main[6539]: 2021-10-13T17:52:18.951Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:19 instance-1 main[6539]: 2021-10-13T17:52:19.867Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:19 instance-1 main[6539]: 2021-10-13T17:52:19.867Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:20 instance-1 main[6539]: 2021-10-13T17:52:20.774Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:20 instance-1 main[6539]: 2021-10-13T17:52:20.774Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:21 instance-1 main[6539]: 2021-10-13T17:52:21.680Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:21 instance-1 main[6539]: 2021-10-13T17:52:21.680Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:22 instance-1 main[6539]: 2021-10-13T17:52:22.585Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:22 instance-1 main[6539]: 2021-10-13T17:52:22.586Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:23 instance-1 main[6539]: 2021-10-13T17:52:23.493Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:23 instance-1 main[6539]: 2021-10-13T17:52:23.493Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:24 instance-1 main[6539]: 2021-10-13T17:52:24.407Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:24 instance-1 main[6539]: 2021-10-13T17:52:24.408Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:25 instance-1 main[6539]: 2021-10-13T17:52:25.317Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:25 instance-1 main[6539]: 2021-10-13T17:52:25.317Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:26 instance-1 main[6539]: 2021-10-13T17:52:26.224Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:26 instance-1 main[6539]: 2021-10-13T17:52:26.224Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"
Oct 13 17:52:27 instance-1 main[6539]: 2021-10-13T17:52:27.134Z [INFO]  polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d
Oct 13 17:52:27 instance-1 main[6539]: 2021-10-13T17:52:27.134Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"

Node 2 trying to sync (detects a fork)

2
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.254Z [INFO] polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"}
3
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.255Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err=\"failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d\""}
4
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.273Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=929697"}
5
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.273Z [DEBUG] polygon.consensus.syncer: sync up to block: from=929698 to=1093038"}
6
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.283Z [INFO] polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"}
7
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.284Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err=\"failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d\""}
8
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.305Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=929697"}
9
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.305Z [DEBUG] polygon.consensus.syncer: sync up to block: from=929698 to=1093038"}
10
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.316Z [INFO] polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"}
11
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.316Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err=\"failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d\""}
12
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.333Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=929697"}
13
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.333Z [DEBUG] polygon.consensus.syncer: sync up to block: from=929698 to=1093038"}
14
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.350Z [INFO] polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"}
15
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.350Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err=\"failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d\""}
16
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.368Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=929697"}
17
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.368Z [DEBUG] polygon.consensus.syncer: sync up to block: from=929698 to=1093038"}
18
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.379Z [INFO] polygon.blockchain: write blocks: num=10 from=929698 to=929707 parent=0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d"}
19
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.379Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err=\"failed to write bulk sync blocks: parent of 0xa58235450c120fbad063b0b34a9c0c866bc1289d332390d135a39194c2861e22 (929698) not found: 0x1492502b1455bb8f53def60027b69ff0b82e4538d7d157d0170952ac1470a95d\""}
20
2021-10-12T11:12:39.495-07:00
{"message":"2021-10-12T18:12:39.398Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=929697"}

Add geth's `debug_traceTransaction`

Add `debug_traceTransaction`

Description

debug_traceTransaction is used a lot in different block explorers to trace transactions and provide more metadata on the tx itself. Example:

https://blockscout.com/xdai/mainnet/tx/0x29bc60b639d1f504110637a4f0b5d3e87edc6466797f4332dcd08c5e41677316/internal-transactions

It's a standard geth endpoint and provides similar functionality to parity's trace module. We would be happy to tackle it if given some direction on how to get started!

Block writing stopping but gossiping continuing on non-sealing nodes

Block writing stopping but gossiping continuing on non-sealing nodes after bulk sync

Description

We've had several instances of blocks stopping writing, some during the sync process, and some during operation. We can see in both cases blocks are still be gossiped on the channel, but writing is stopped.

All of these cases are on non-sealing nodes, i.e., not participating in consensus.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 18e9317
branch that causes this issue fix/sync-issue

Steps to reproduce

Tell us how to reproduce this issue

This seems to only happen in non-sealing nodes, or we've only seen it in non-sealing nodes, maybe because they don't participate in the IBFT scheme. Currently have a 5 node cluster participating in consensus. It seems to be the most reproducible when a node starts and enters bulk sync for a bit to catch up.
Where the issue is, if you know

It seems that after bulk sync completes, WatchSyncWithPeer is waiting indefinitely on peer.popBlock() or the WatchSyncWithPeer handler is returning false and existing the loop, but based on the logs in Case 1 it doesn't look like it gets past popBlock().

If you're a validator/sealing node however, it does seem to skip this step b/c of this line https://github.com/0xPolygon/polygon-sdk/blob/develop/consensus/ibft/ibft.go#L321

Probably why we only see this issue on non-sealing nodes

We've added extra logging in WatchSyncWithPeer, BulkSyncWithPeer and enqueueBlock for Log 1 context. See below:

// enqueueBlock adds the specific block to the peerID queue
func (s *Syncer) enqueueBlock(peerID peer.ID, b *types.Block) {
	s.logger.Debug("enqueue block", "peer", peerID, "number", b.Number(), "hash", b.Hash())
	p, ok := s.peers[peerID]
	s.logger.Debug("enqueue block", "ok", ok, "peer", p.peer)
	if ok {
		p.appendBlock(b)
	}
}

func (s *Syncer) WatchSyncWithPeer(p *syncPeer, handler func(b *types.Block) bool) {
	s.logger.Debug("Watch sync with peer")
	// purge from the cache of broadcasted blocks all the ones we have written so far
	header := s.blockchain.Header()
	s.logger.Debug("Header number", header.Number)
	p.purgeBlocks(header.Hash)
	s.logger.Debug("Blocks purged")

	// listen and enqueue the messages
	for {
		b := p.popBlock()

		if err := s.blockchain.WriteBlocks([]*types.Block{b}); err != nil {
			s.logger.Error("failed to write block", "err", err)
			break
		}
		if !handler(b) {
			break
		}
	}
}

func (s *Syncer) BulkSyncWithPeer(p *syncPeer) error {
	// find the common ancestor
	ancestor, fork, err := s.findCommonAncestor(p.client, p.status)
	if err != nil {
		return err
	}

	// find in batches
	s.logger.Debug("fork found", "ancestor", ancestor.Number)

	startBlock := fork

	var lastTarget uint64

	// sync up to the current known header
	for {
		// update target
		target := p.status.Number
		if target == lastTarget {
			s.logger.Debug("No more changes to pull!", "target", target, "lastTarget", lastTarget, "Breaking.")
			// there are no more changes to pull for now
			break
		}

		for {
			s.logger.Debug("sync up to block", "from", startBlock.Number, "to", target)

			// start to synchronize with it
			sk := &skeleton{
				span: 10,
				num:  5,
			}

			if err := sk.build(p.client, startBlock.Hash); err != nil {
				return fmt.Errorf("failed to build skeleton: %v", err)
			}

			// fill skeleton
			for indx := range sk.slots {
				sk.fillSlot(uint64(indx), p.client) //nolint
			}

			// sync the data
			for _, slot := range sk.slots {
				if err := s.blockchain.WriteBlocks(slot.blocks); err != nil {
					return fmt.Errorf("failed to write bulk sync blocks: %v", err)
				}
			}

			// try to get the next block
			startBlock = sk.LastHeader()

			if startBlock.Number >= uint64(target) {
				s.logger.Debug("startBlock.Number", startBlock.Number, "target", target, "Breaking.")
				break
			}
		}

		lastTarget = target
	}
	return nil
}

// appendBlock adds a new block to the block queue
func (s *syncPeer) appendBlock(b *types.Block) {
	s.enqueueLock.Lock()
	defer s.enqueueLock.Unlock()
	fmt.Println(("Got enqueue lock"))

	if len(s.enqueue) == maxEnqueueSize {
		// pop first element
		s.enqueue = s.enqueue[1:]
	}
	// append to the end
	s.enqueue = append(s.enqueue, b)

	// update the number and hash
	s.number = b.Number()
	s.hash = b.Hash()

	fmt.Println("New sync peer number", s.number)
	fmt.Println("New sync peer hash", s.hash)

	select {
	case s.enqueueCh <- struct{}{}:
	default:
	}
}

Which commands triggered the issue, if any
./main server --chain genesis.json --jsonrpc 0.0.0.0:5000 --libp2p 0.0.0.0:1478 --log-level debug --nat $IP

Expected behaviour

Gossip'd blocks should continue to be written to state and processed normally after bulk sync

Actual behaviour

Block writing suddenly stops, causing eth_blockNumber to return incorrect values and have all JSON-RPC endpoints return stale data.

Logs

Case 1 (with added logs from above)

2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.161Z [INFO] polygon: Data dir: path=./test-chain
-- | --
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.163Z [INFO] polygon.network: LibP2P server running: addr=/ip4/18.232.150.56/tcp/1478/p2p/16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.196Z [INFO] polygon.consensus.ibft: validator key: addr=0xc256390D56BEa528C273Ac318B58022c19798445
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.197Z [INFO] polygon.blockchain: Current header: hash=0x7a771b5d2e3665a50bb857a431ff5dd4375019b4385a0741057b6837a914d890 number=872890
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.197Z [INFO] polygon.blockchain: genesis: hash=0x24d80debadd21a42ae8bcc3b28d39a0e2e2cbf9d7fee3507c1e38a33995fc3f5
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.198Z [INFO] polygon: GRPC server running: addr=127.0.0.1:9632
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.198Z [INFO] polygon.jsonrpc: http server started: addr=0.0.0.0:5000
  | 2021-09-30T13:56:29.341Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.198Z [INFO] polygon.consensus.ibft: snapshot was not found, restore snapshot at beginning of current epoch: current epoch=8
  | 2021-09-30T13:56:33.887Z | Sep 30 13:56:29 ip-172-31-24-192 web: 2021-09-30T13:56:29.199Z [INFO] polygon.consensus.ibft: syncing past snapshots: from=800000 to=872890
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.176Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh: [/ip4/3.215.13.226/tcp/10001]}"
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.186Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW: [/ip4/204.236.222.207/tcp/10001]}"
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.190Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.197Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x: [/ip4/34.207.228.138/tcp/10001]}"
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.201Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.207Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU: [/ip4/3.84.231.38/tcp/10001]}"
  | 2021-09-30T13:56:34.343Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.211Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:34 ip-172-31-24-192 web: 2021-09-30T13:56:34.221Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872894 hash=0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.036Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer number 872894
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer hash 0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.037Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872894 hash=0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.038Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer number 872894
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer hash 0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.039Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872894 hash=0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.039Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer number 872894
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer hash 0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.041Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872894 hash=0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: 2021-09-30T13:56:35.041Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:35.093Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer number 872894
  | 2021-09-30T13:56:37.094Z | Sep 30 13:56:35 ip-172-31-24-192 web: New sync peer hash 0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8
  | 2021-09-30T13:56:37.094Z | Sep 30 13:56:36 ip-172-31-24-192 web: 2021-09-30T13:56:36.989Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmPSdFWFTjKCyWSYUpi5jBTK1LBSbApU3SGwXP4ip6UNY2
  | 2021-09-30T13:56:39.345Z | Sep 30 13:56:36 ip-172-31-24-192 web: 2021-09-30T13:56:36.989Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmPSdFWFTjKCyWSYUpi5jBTK1LBSbApU3SGwXP4ip6UNY2: [/ip4/52.90.232.81/tcp/10001]}"
  | 2021-09-30T13:56:43.887Z | Sep 30 13:56:39 ip-172-31-24-192 web: 2021-09-30T13:56:39.180Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAm3VD32VRQPVVU8QsGfLqmYu3bXzyPgvAhhm2nmQX3XKep: [/ip4/54.234.87.242/tcp/1478]}"
  | 2021-09-30T13:56:45.097Z | Sep 30 13:56:44 ip-172-31-24-192 web: 2021-09-30T13:56:44.181Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAkz7raWw7SMDQrySCWWSxBa4mbsC8tbRZazw6cdtXnqUZD: [/ip4/54.160.193.245/tcp/1478]}"
  | 2021-09-30T13:56:45.097Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.096Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872895 hash=0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.097Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.096Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:45.097Z | Sep 30 13:56:45 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer number 872895
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer hash 0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.097Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872895 hash=0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.098Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer number 872895
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer hash 0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.100Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872895 hash=0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.101Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer number 872895
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer hash 0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.102Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872895 hash=0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: 2021-09-30T13:56:45.102Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:45.347Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer number 872895
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:45 ip-172-31-24-192 web: New sync peer hash 0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.019Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872896 hash=0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.019Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer number 872896
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer hash 0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.023Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872896 hash=0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.023Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer number 872896
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer hash 0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.023Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872896 hash=0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.023Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer number 872896
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer hash 0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.027Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872896 hash=0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.028Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:47.098Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer number 872896
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: New sync peer hash 0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.299Z [DEBUG] polygon.jsonrpc: handle: request={"method":"eth_blockNumber","params":[],"id":16913,"jsonrpc":"2.0"}
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.299Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=16913
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.300Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":16913,"result":"0xd51ba"}
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.320Z [DEBUG] polygon.jsonrpc: handle: request={"method":"eth_sendRawTransaction","params":["0xf86c823f6982096d8252089482d4a18c982495594a10311276b02c03df679575871c6bf52634000080820532a08b1a1407f572c49e6744502933d5ba93eb1eb86cbc1b406a445b804e4b59610ea044c7913130326a7cf1b20270cf9b147116231d9e2fffaac6bd18006d7ca75cd8"],"id":16914,"jsonrpc":"2.0"}
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.320Z [DEBUG] polygon.dispatcher: request: method=eth_sendRawTransaction id=16914
  | 2021-09-30T13:56:47.348Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.322Z [DEBUG] polygon.txpool: add txn: ctx=addTxn hash=0x1b010ae4942a22b553ea8cb0ad16dac67218b3a5c058b9a8f3dd0970e7c027ff from=0x8b15973e6f557030FE01D02f09A7cF4f36f3b43F
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:47 ip-172-31-24-192 web: 2021-09-30T13:56:47.322Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":16914,"result":"0x1b010ae4942a22b553ea8cb0ad16dac67218b3a5c058b9a8f3dd0970e7c027ff"}
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.026Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872897 hash=0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.026Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer number 872897
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer hash 0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.029Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872897 hash=0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.029Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer number 872897
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer hash 0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.030Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872897 hash=0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.030Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer number 872897
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer hash 0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.031Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872897 hash=0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.032Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:49.099Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer number 872897
  | 2021-09-30T13:56:49.349Z | Sep 30 13:56:49 ip-172-31-24-192 web: New sync peer hash 0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:49 ip-172-31-24-192 web: 2021-09-30T13:56:49.181Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmLqdTjUjjqtibhoN3ZYpnyzVvyhFgUUHtLiTQJJ3uym9h: [/ip4/18.215.250.99/tcp/1478]}"
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.026Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872898 hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.027Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer number 872898
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer hash 0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.028Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872898 hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.028Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer number 872898
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer hash 0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.029Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872898 hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.029Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer number 872898
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer hash 0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.030Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872898 hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.031Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:56:51.100Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer number 872898
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: New sync peer hash 0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.153Z [DEBUG] polygon.consensus.ibft: state change: new=SyncState
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.153Z [DEBUG] polygon.consensus.ibft: current sequence: sequence=872891
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.182Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=872890
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.182Z [DEBUG] polygon.consensus.syncer: sync up to block: from=872891 to=872894
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.184Z [INFO] polygon.blockchain: write blocks: num=8 from=872891 to=872898 parent=0x7a771b5d2e3665a50bb857a431ff5dd4375019b4385a0741057b6837a914d890
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.195Z [INFO] polygon.blockchain: new block: number=872891 hash=0x443d161ff8c77005e8644179487b7b93fd5bccd6b31d04b04cae2e524e8d72d6 txns=0 generation_time_in_sec=10
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.196Z [INFO] polygon.blockchain: new block: number=872892 hash=0x8fbaaed124578b20cee06374eb2f685a554c8108336d76528f7496f2d2d2d288 txns=0 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.196Z [INFO] polygon.blockchain: new block: number=872893 hash=0x9f71ab83413b748ee8bdeae03264ac169dc81f12e4566c582ae6852d73985926 txns=0 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.197Z [INFO] polygon.blockchain: new block: number=872894 hash=0x2841bdc7524be0ca56a2252af42dc45355ccd2971e7dd87fe960021fd58c7ac8 txns=0 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.197Z [INFO] polygon.blockchain: new block: number=872895 hash=0x56b4f17f45ac9bec6aaebc54691d2711c963f10f089e578b72b19b867e66d046 txns=0 generation_time_in_sec=10
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.198Z [INFO] polygon.blockchain: new block: number=872896 hash=0x706a67534fd2873b512bf730de0c8c4f3042410cc3e9e9d740b87dee7cd666bd txns=0 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.198Z [INFO] polygon.blockchain: new block: number=872897 hash=0x7d6922cf708c0f5277ba65eb3c83ce8177c3747f266af78b747471a233b18406 txns=0 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [INFO] polygon.blockchain: new block: number=872898 hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821 txns=1 generation_time_in_sec=2
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [INFO] polygon.blockchain: new head: hash=0x0b545fea1ef2c7b72f47ad34c13f1a8e9a4b7cac5faf72ea0917b8750fb03821 number=872898
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [DEBUG] polygon.consensus.syncer: startBlock.Number: %!s(uint64=872898)=target %!s(uint64=872894)=Breaking.
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [DEBUG] polygon.consensus.syncer: No more changes to pull!: target=872894 lastTarget=872894 EXTRA_VALUE_AT_END=Breaking.
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [DEBUG] polygon.consensus.syncer: Watch sync with peer
  | 2021-09-30T13:56:51.350Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [DEBUG] polygon.consensus.syncer: Header number: EXTRA_VALUE_AT_END=872898
  | 2021-09-30T13:56:54.351Z | Sep 30 13:56:51 ip-172-31-24-192 web: 2021-09-30T13:56:51.199Z [DEBUG] polygon.consensus.syncer: Blocks purged
  | 2021-09-30T13:56:54.351Z | Sep 30 13:56:54 ip-172-31-24-192 web: 2021-09-30T13:56:54.183Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmM4GFptxAkpQrNBJrdud9yTKg3LAnNQid22Lh7j4ac4bW: [/ip4/184.72.207.241/tcp/1478]}"
  | 2021-09-30T13:56:58.887Z | Sep 30 13:56:54 ip-172-31-24-192 web: 2021-09-30T13:56:54.185Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAm5pstFUEigRQB983B1vpWZ1hAwVHqP7dQHtPzK1yG1dbH: [/ip4/3.81.5.248/tcp/1478]}"
  | 2021-09-30T13:57:01.104Z | Sep 30 13:56:59 ip-172-31-24-192 web: 2021-09-30T13:56:59.186Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAkyRQYsYXUNP5BNnMHEh7pfmDz3tt3esyZ5dv5ALxtyphM: [/ip4/3.81.5.248/tcp/1478]}"
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.065Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872899 hash=0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.065Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer number 872899
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer hash 0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.097Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872899 hash=0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.097Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:01.104Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer number 872899
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer hash 0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.128Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872899 hash=0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.128Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer number 872899
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer hash 0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.139Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872899 hash=0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: 2021-09-30T13:57:01.139Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:01.355Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer number 872899
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:01 ip-172-31-24-192 web: New sync peer hash 0x227dcb55a19a45d661f4cd5b71d74222363b82ed331da189b8f5edef14c43850
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.172Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872900 hash=0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.172Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer number 872900
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer hash 0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.174Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872900 hash=0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.174Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer number 872900
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer hash 0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.218Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872900 hash=0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.218Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer number 872900
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer hash 0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.320Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872900 hash=0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: 2021-09-30T13:57:03.320Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:03.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer number 872900
  | 2021-09-30T13:57:04.356Z | Sep 30 13:57:03 ip-172-31-24-192 web: New sync peer hash 0x43d897b1493b2939a6532dbbd63fcfe6807d566a502df5842780451d414f9118
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:04 ip-172-31-24-192 web: 2021-09-30T13:57:04.188Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmNkhfoMDCEqZ21AxaYNWvUMx6Lg21UkMS4GiLrcK2SWD5: [/ip4/3.81.5.248/tcp/10001]}"
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.025Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872901 hash=0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.026Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer number 872901
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer hash 0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.027Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872901 hash=0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.027Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer number 872901
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer hash 0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.028Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872901 hash=0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.028Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer number 872901
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer hash 0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.030Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872901 hash=0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: 2021-09-30T13:57:05.030Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:05.106Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer number 872901
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:05 ip-172-31-24-192 web: New sync peer hash 0x7e2753bd98ce6f0686e170cbe9537c31396e1e36719eb804acebd67e94902a1a
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.027Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872902 hash=0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.027Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer number 872902
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer hash 0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.028Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872902 hash=0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.028Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer number 872902
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer hash 0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.029Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872902 hash=0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.029Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer number 872902
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer hash 0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.030Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872902 hash=0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: 2021-09-30T13:57:07.030Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:07.107Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer number 872902
  | 2021-09-30T13:57:09.376Z | Sep 30 13:57:07 ip-172-31-24-192 web: New sync peer hash 0x61d93388e964b8ae0945f41f62fc73d42ed14b79d0b13a6810387363ee4a0edd
  | 2021-09-30T13:57:13.887Z | Sep 30 13:57:09 ip-172-31-24-192 web: 2021-09-30T13:57:09.188Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAm52bbB91oJSbn4R3gsiRi8Jpx4BfZ8cTHzUtw8qfpeSUP: [/ip4/3.84.246.251/tcp/1478]}"
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:14 ip-172-31-24-192 web: 2021-09-30T13:57:14.189Z [DEBUG] polygon.network: dial: local=16Uiu2HAm7iiWLhJShny9TG5RFaazXa4zbW4qaprnfbbrpkiE7w7C addr="{16Uiu2HAmEiQZyVpQBPE3o8B4gL3bithtkYZ2Wz5sziFyL162EptF: [/ip4/35.173.248.101/tcp/1478]}"
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.069Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=872903 hash=0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.069Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer number 872903
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer hash 0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.071Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=872903 hash=0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.071Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer number 872903
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer hash 0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.072Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=872903 hash=0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.072Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer number 872903
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer hash 0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.073Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872903 hash=0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: 2021-09-30T13:57:17.073Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:17.112Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer number 872903
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:17 ip-172-31-24-192 web: New sync peer hash 0x8366fff73fb36f607d3c93ca63a3c7cc1af513a1645ca17dc13716170616a9ff
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:19 ip-172-31-24-192 web: 2021-09-30T13:57:19.022Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=872904 hash=0xfe1a16327a64babb1ca496353e2c926724377157ce83577888f9925d59548b5e
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:19 ip-172-31-24-192 web: 2021-09-30T13:57:19.022Z [DEBUG] polygon.consensus.syncer: enqueue block: ok=true peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:19 ip-172-31-24-192 web: Got enqueue lock
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:19 ip-172-31-24-192 web: New sync peer number 872904
  | 2021-09-30T13:57:19.113Z | Sep 30 13:57:19 ip-172-31-24-192 web: New sync peer hash 0xfe1a16327a64babb1ca496353e2c926724377157ce83577888f9925d59548b5e

Case 2:



 | 2021-09-28T00:28:56.131Z | Sep 28 00:28:56 ip-172-31-92-108 web: 2021-09-28T00:28:56.058Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819452 hash=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db
  | 2021-09-28T00:28:56.131Z | Sep 28 00:28:56 ip-172-31-92-108 web: 2021-09-28T00:28:56.059Z [INFO] polygon.blockchain: new block: number=819452 hash=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db txns=0 generation_time_in_sec=10
  | 2021-09-28T00:28:56.131Z | Sep 28 00:28:56 ip-172-31-92-108 web: 2021-09-28T00:28:56.059Z [INFO] polygon.blockchain: new head: hash=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db number=819452
  | 2021-09-28T00:28:56.131Z | Sep 28 00:28:56 ip-172-31-92-108 web: 2021-09-28T00:28:56.063Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819452 hash=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:56 ip-172-31-92-108 web: 2021-09-28T00:28:56.064Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819452 hash=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.023Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=819453 hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.023Z [INFO] polygon.blockchain: write block: num=819453 parent=0x723e15e0cde997de538ab7e68cd93abf67d7d0ae625d705c295a46f0b25e72db
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.025Z [INFO] polygon.blockchain: new block: number=819453 hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938 txns=0 generation_time_in_sec=2
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.025Z [INFO] polygon.blockchain: new head: hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938 number=819453
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819453 hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938
  | 2021-09-28T00:28:58.131Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.037Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819453 hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938
  | 2021-09-28T00:29:00.132Z | Sep 28 00:28:58 ip-172-31-92-108 web: 2021-09-28T00:28:58.038Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819453 hash=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.033Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819454 hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.068Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819454 hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.073Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819454 hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.073Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=819454 hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.073Z [INFO] polygon.blockchain: write block: num=819454 parent=0x48cca157251acd08ba1d0109227ca5be1c13a46a3471e52110ce07a2eab96938
  | 2021-09-28T00:29:00.132Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.075Z [INFO] polygon.blockchain: new block: number=819454 hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9 txns=0 generation_time_in_sec=2
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:00 ip-172-31-92-108 web: 2021-09-28T00:29:00.075Z [INFO] polygon.blockchain: new head: hash=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9 number=819454
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.454Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=819455 hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1c0050210
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.454Z [INFO] polygon.blockchain: write block: num=819455 parent=0xe0e12c0443b5b53bfdb8934e25ea060798b9274305e31c0aad6d1d039e7f4bb9
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.455Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819455 hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1c0050210
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.455Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819455 hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1c0050210
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.456Z [INFO] polygon.blockchain: new block: number=819455 hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1c0050210 txns=0 generation_time_in_sec=2
  | 2021-09-28T00:29:02.633Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.456Z [INFO] polygon.blockchain: new head: hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1c0050210 number=819455
  | 2021-09-28T00:29:04.634Z | Sep 28 00:29:02 ip-172-31-92-108 web: 2021-09-28T00:29:02.540Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819455 hash=0xe0dc0fbad05b4cb4fe002f5d05161a5c0bad0752fd4f438283c638e1
2021-09-28T00:29:09.386Z | Sep 28 00:29:07 ip-172-31-92-108 web: 2021-09-28T00:29:07.556Z [INFO] polygon.network: Peer disconnected: id=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x
2021-09-28T00:29:12.638Z | Sep 28 00:29:12 ip-172-31-92-108 web: 2021-09-28T00:29:12.564Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819456 hash=0xd0f9df30b69f0fc9b2cef7eafee027e92d08933565e354952b4e9e1d1ff809cf
  | 2021-09-28T00:29:12.638Z | Sep 28 00:29:12 ip-172-31-92-108 web: 2021-09-28T00:29:12.565Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819456 hash=0xd0f9df30b69f0fc9b2cef7eafee027e92d08933565e354952b4e9e1d1ff809cf
  | 2021-09-28T00:29:14.138Z | Sep 28 00:29:12 ip-172-31-92-108 web: 2021-09-28T00:29:12.566Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819456 hash=0xd0f9df30b69f0fc9b2cef7eafee027e92d08933565e354952b4e9e1d1ff809cf
  | 2021-09-28T00:29:14.138Z | Sep 28 00:29:14 ip-172-31-92-108 web: 2021-09-28T00:29:14.033Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819457 hash=0xe53d1e50303e835bf260a81bfd7975d437166f8e2f2ffea1dc81aea547c30451
  | 2021-09-28T00:29:14.138Z | Sep 28 00:29:14 ip-172-31-92-108 web: 2021-09-28T00:29:14.034Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819457 hash=0xe53d1e50303e835bf260a81bfd7975d437166f8e2f2ffea1dc81aea547c30451
  | 2021-09-28T00:29:15.139Z | Sep 28 00:29:14 ip-172-31-92-108 web: 2021-09-28T00:29:14.035Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819457 hash=0xe53d1e50303e835bf260a81bfd7975d437166f8e2f2ffea1dc81aea5
  2021-09-28T00:29:24.143Z | Sep 28 00:29:23 ip-172-31-92-108 web: 2021-09-28T00:29:23.987Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=819458 hash=0xe7c9d4335dc80d2b2962d7aeb198cad43d6c4f92a7c73a33180ea7de2dc7c4cd
  | 2021-09-28T00:29:24.143Z | Sep 28 00:29:23 ip-172-31-92-108 web: 2021-09-28T00:29:23.987Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=819458 hash=0xe7c9d4335dc80d2b2962d7aeb198cad43d6c4f92a7c73a33180ea7de2dc7c4cd
  | 2021-09-28T00:29:24.643Z | Sep 28 00:29:23 ip-172-31-92-108 web: 2021-09-28T00:29:23.988Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=819458 hash=0xe7c9d4335dc80d2b2962d7aeb198cad43d6c4f92a7c73a33180ea7de2d

The enqueue blocks pretty much go on forever, so the node is receiving blocks from other nodes but it has stopped writing for some reason.

Case 3:

During bulk syncing:



2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.145Z [INFO] polygon.blockchain: write blocks: num=10 from=813284 to=813293 parent=0x45483aa4b58a1382d426eeba89182975b58060b946349dac53c993f1f99b9dfe
-- | --
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.159Z [INFO] polygon.blockchain: new block: number=813284 hash=0xac72f926be0b0ce1814c5d2e12a93864eb75106cb0603fa0b8a3bea85ddfcef4 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.159Z [INFO] polygon.blockchain: new block: number=813285 hash=0xb7c9a7d5bf786cbae9d214147c21b54d7fcbb0b253560e03f2e5a62a9ee5625f txns=0 generation_time_in_sec=10
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.160Z [INFO] polygon.blockchain: new block: number=813286 hash=0x507e712b76cd8751f8d0c7917d12a199f632b49c9714238b1a060cfc87efed67 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.160Z [INFO] polygon.blockchain: new block: number=813287 hash=0x9e720843460c16015d5100a37d785efa3bdb0ef78607223c8370f381efcbad27 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.161Z [INFO] polygon.blockchain: new block: number=813288 hash=0x8421e5cd66d1d0aef47a69506228c83bdcb50a721d3f1c116a4ff8659e4a9269 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.161Z [INFO] polygon.blockchain: new block: number=813289 hash=0xde05602d6067bf8f7d7f1a92443c6824999cf78b4ef69101907e9e9780c38633 txns=0 generation_time_in_sec=10
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.162Z [INFO] polygon.blockchain: new block: number=813290 hash=0x85e7d471f9faa2de65a5f7aed689ef04f8be32b9b2f742f3b426cfd76c6edb1b txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.162Z [INFO] polygon.blockchain: new block: number=813291 hash=0x457ace51c136a33541139a26e35047ceb9063892b89efe082f822836bcef9aa8 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.163Z [INFO] polygon.blockchain: new block: number=813292 hash=0xdb102b5c45e1adfb7cfca88c3770ae2a2154aab1034a2289b24557b6a34c2094 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.163Z [INFO] polygon.blockchain: new block: number=813293 hash=0x79b15bc986bf2428c9dd575f0024695356fa1d27849c94786b5266a3273f46bf txns=0 generation_time_in_sec=10
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.163Z [INFO] polygon.blockchain: new head: hash=0x79b15bc986bf2428c9dd575f0024695356fa1d27849c94786b5266a3273f46bf number=813293
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.163Z [INFO] polygon.blockchain: write blocks: num=10 from=813294 to=813303 parent=0x79b15bc986bf2428c9dd575f0024695356fa1d27849c94786b5266a3273f46bf
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.177Z [INFO] polygon.blockchain: new block: number=813294 hash=0x6115db20b4ad1da834db8bc30bfcc83e51dd632884f79c70ef3f7e9506e7589e txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.178Z [INFO] polygon.blockchain: new block: number=813295 hash=0xf2a58399318c5984551a83d76b13211e80e7e5ab0ddfd936f4bf18b79efcf88a txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.178Z [INFO] polygon.blockchain: new block: number=813296 hash=0x5fb09cc9c05bcdde41bd81cac6f4d6013e1bc2c4292ff46ae473feebdcf368e2 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.179Z [INFO] polygon.blockchain: new block: number=813297 hash=0xb6c81b70302bb0fcd81c04bda2926a9938c11ba9aefad63425c440a00fb1b20c txns=0 generation_time_in_sec=10
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.179Z [INFO] polygon.blockchain: new block: number=813298 hash=0xe3da01b8598b031fe092ba6368751cec1a0eead36e0d55f98ea7c6caff6cf8b5 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.179Z [INFO] polygon.blockchain: new block: number=813299 hash=0x85ecf5ed95a9870dac66f348f761a40bbf435e9e060ccc1298ccfc901188e403 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.180Z [INFO] polygon.blockchain: new block: number=813300 hash=0x1a744cd3e40abf83bcb79a06b5472625573b12575ac99404a3ff790e9cbda7dd txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.180Z [INFO] polygon.blockchain: new block: number=813301 hash=0xada0fe41041d9fc352ce832019c3ab667f0680e2fcbef5785d4a51d4b19399d3 txns=0 generation_time_in_sec=10
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.181Z [INFO] polygon.blockchain: new block: number=813302 hash=0xc81a2921080a7c5f6525777627224bcf5b86fb00232f6eca09e9a147dd14596a txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.181Z [INFO] polygon.blockchain: new block: number=813303 hash=0x97561b5b1cd2c1ab03af28f6bde68848ebc0d561a2ec6daf0eb81a07e402743b txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.181Z [INFO] polygon.blockchain: new head: hash=0x97561b5b1cd2c1ab03af28f6bde68848ebc0d561a2ec6daf0eb81a07e402743b number=813303
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.181Z [DEBUG] polygon.consensus.syncer: sync up to block: from=813303 to=814788
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.195Z [INFO] polygon.blockchain: write blocks: num=10 from=813303 to=813312 parent=0xc81a2921080a7c5f6525777627224bcf5b86fb00232f6eca09e9a147dd14596a
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.225Z [INFO] polygon.blockchain: new block: number=813303 hash=0x97561b5b1cd2c1ab03af28f6bde68848ebc0d561a2ec6daf0eb81a07e402743b txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:33.395Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.226Z [INFO] polygon.blockchain: new block: number=813304 hash=0x1ef274e97e8e4a0f1a90d125157dcf8b4aa0a8abfb5aed1d26b796aff5a795e7 txns=0 generation_time_in_sec=2
  | 2021-09-27T19:10:37.693Z | Sep 27 19:10:33 ip-172-31-44-61 web: 2021-09-27T19:10:33.226Z [INFO] polygon.blockchain: new block: number=813305 hash=0x8f6b9553e65cce02fa07284438b6c3857509a9e23b6af4e696d55f70fc4db52c txns=0 generation_time_in_sec=10
  | 2021-09-27T19:11:01.162Z | Sep 27 19:11:01 ip-172-31-44-61 web: 2021-09-27T19:11:01.025Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815256 hash=0xc621bd58aa0987a1b36b2c37787f5933badc251b2806fbb3d8f3c7bfa4feedd8
  | 2021-09-27T19:11:01.162Z | Sep 27 19:11:01 ip-172-31-44-61 web: 2021-09-27T19:11:01.025Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815256 hash=0xc621bd58aa0987a1b36b2c37787f5933badc251b2806fbb3d8f3c7bfa4feedd8
  | 2021-09-27T19:11:01.162Z | Sep 27 19:11:01 ip-172-31-44-61 web: 2021-09-27T19:11:01.026Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815256 hash=0xc621bd58aa0987a1b36b2c37787f5933badc251b2806fbb3d8f3c7bfa4feedd8
  | 2021-09-27T19:11:03.663Z | Sep 27 19:11:01 ip-172-31-44-61 web: 2021-09-27T19:11:01.027Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815256 hash=0xc621bd58aa0987a1b36b2c37787f5933badc251b2806fbb3d8f3c7bfa4feedd8
  | 2021-09-27T19:11:03.663Z | Sep 27 19:11:03 ip-172-31-44-61 web: 2021-09-27T19:11:03.649Z [DEBUG] polygon.jsonrpc: handle: request={"id":"ec24f224-c8bf-49d7-bd40-7070a0121024","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:03.663Z | Sep 27 19:11:03 ip-172-31-44-61 web: 2021-09-27T19:11:03.649Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=ec24f224-c8bf-49d7-bd40-7070a0121024
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:03 ip-172-31-44-61 web: 2021-09-27T19:11:03.649Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":"ec24f224-c8bf-49d7-bd40-7070a0121024","result":"0xc6ee6"}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.723Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327712,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.723Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327712e+15
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.723Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327712,"result":"0xc6ee6"}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.789Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854836,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x6aeca9a564c210bab9d40bd6c8a41683d8d9cdba","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.789Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854836e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.790Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.790Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854836,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.793Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854838,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x37a09fa7afbbb48537bf94d5609d3e2fe04fa791","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854838e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854838,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854844,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x3ed6bef5274f9f5556c5cb30ff75a45d2882bfa6","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854844e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854844,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.794Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854842,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x9005d7e31f165e1dd7f443974340baa3d45c0e55","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854842e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854842,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.793Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854840,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7976bc88d6177d65f37a0e1fc55652f928103bd1","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.8047474685484e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.795Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854840,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.797Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854850,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x68200aa9f88d9f185e7bb7e0ea20dd3d773d1a3f","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.797Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.8047474685485e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854850,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854852,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2adddfff3f64be2cec1af49cf90c60c8a0f51c20","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854852e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854852,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854848,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xaf99678f49fbb9c15ce995d6ba6ad0dbee856304","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854848e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854848,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854846,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x51d4b51f9251912c85af18af6e526ecb976ad810","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854846e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854846,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854856,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xf0d6de16e2c14e0598a18c2b43716d7d0672690c","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.798Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854856e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.799Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.799Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854856,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.800Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854854,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2079a994e9bbc88496df03a59a0cfa2c9e2af473","0xc7098"]}
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.800Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854854e+14
  | 2021-09-27T19:11:04.914Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.801Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815256 header"
  | 2021-09-27T19:11:07.165Z | Sep 27 19:11:04 ip-172-31-44-61 web: 2021-09-27T19:11:04.801Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854854,"error":{"code":-32600,"message":"Error fetching block number 815256 header"}}"
  | 2021-09-27T19:11:07.165Z | Sep 27 19:11:07 ip-172-31-44-61 web: 2021-09-27T19:11:07.100Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327713,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:07.165Z | Sep 27 19:11:07 ip-172-31-44-61 web: 2021-09-27T19:11:07.100Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327713e+15
  | 2021-09-27T19:11:11.167Z | Sep 27 19:11:07 ip-172-31-44-61 web: 2021-09-27T19:11:07.100Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327713,"result":"0xc6ee6"}
  | 2021-09-27T19:11:11.167Z | Sep 27 19:11:11 ip-172-31-44-61 web: 2021-09-27T19:11:11.055Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815257 hash=0x241a7e1fdda874adce4a207c3ac3a51b93f15868b39f3bb6341ff16b5067dca3
  | 2021-09-27T19:11:11.167Z | Sep 27 19:11:11 ip-172-31-44-61 web: 2021-09-27T19:11:11.057Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815257 hash=0x241a7e1fdda874adce4a207c3ac3a51b93f15868b39f3bb6341ff16b5067dca3
  | 2021-09-27T19:11:11.167Z | Sep 27 19:11:11 ip-172-31-44-61 web: 2021-09-27T19:11:11.058Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815257 hash=0x241a7e1fdda874adce4a207c3ac3a51b93f15868b39f3bb6341ff16b5067dca3
  | 2021-09-27T19:11:13.169Z | Sep 27 19:11:11 ip-172-31-44-61 web: 2021-09-27T19:11:11.059Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815257 hash=0x241a7e1fdda874adce4a207c3ac3a51b93f15868b39f3bb6341ff16b5067dca3
  | 2021-09-27T19:11:13.169Z | Sep 27 19:11:13 ip-172-31-44-61 web: 2021-09-27T19:11:13.039Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815258 hash=0x8b0c6e253e485558d62b1148b776495e7ac91af84ad3c4f1604e4545ad018dc6
  | 2021-09-27T19:11:13.169Z | Sep 27 19:11:13 ip-172-31-44-61 web: 2021-09-27T19:11:13.040Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815258 hash=0x8b0c6e253e485558d62b1148b776495e7ac91af84ad3c4f1604e4545ad018dc6
  | 2021-09-27T19:11:13.169Z | Sep 27 19:11:13 ip-172-31-44-61 web: 2021-09-27T19:11:13.041Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815258 hash=0x8b0c6e253e485558d62b1148b776495e7ac91af84ad3c4f1604e4545ad018dc6
  | 2021-09-27T19:11:15.170Z | Sep 27 19:11:13 ip-172-31-44-61 web: 2021-09-27T19:11:13.042Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815258 hash=0x8b0c6e253e485558d62b1148b776495e7ac91af84ad3c4f1604e4545ad018dc6
  | 2021-09-27T19:11:15.170Z | Sep 27 19:11:15 ip-172-31-44-61 web: 2021-09-27T19:11:15.035Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815259 hash=0xcd149b62e4c04feef6814d9cc9be453d92f2b4f7baf728fbc9580a7f5c1100c0
  | 2021-09-27T19:11:15.170Z | Sep 27 19:11:15 ip-172-31-44-61 web: 2021-09-27T19:11:15.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815259 hash=0xcd149b62e4c04feef6814d9cc9be453d92f2b4f7baf728fbc9580a7f5c1100c0
  | 2021-09-27T19:11:15.170Z | Sep 27 19:11:15 ip-172-31-44-61 web: 2021-09-27T19:11:15.037Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815259 hash=0xcd149b62e4c04feef6814d9cc9be453d92f2b4f7baf728fbc9580a7f5c1100c0
  | 2021-09-27T19:11:16.421Z | Sep 27 19:11:15 ip-172-31-44-61 web: 2021-09-27T19:11:15.038Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815259 hash=0xcd149b62e4c04feef6814d9cc9be453d92f2b4f7baf728fbc9580a7f5c1100c0
  | 2021-09-27T19:11:16.421Z | Sep 27 19:11:16 ip-172-31-44-61 web: 2021-09-27T19:11:16.409Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327714,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:16.421Z | Sep 27 19:11:16 ip-172-31-44-61 web: 2021-09-27T19:11:16.409Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327714e+15
  | 2021-09-27T19:11:17.172Z | Sep 27 19:11:16 ip-172-31-44-61 web: 2021-09-27T19:11:16.409Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327714,"result":"0xc6ee6"}
  | 2021-09-27T19:11:17.172Z | Sep 27 19:11:17 ip-172-31-44-61 web: 2021-09-27T19:11:17.021Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815260 hash=0xfdfd284be797a50a77268024acbcc779739335654e7c78f040477ffa59836760
  | 2021-09-27T19:11:17.172Z | Sep 27 19:11:17 ip-172-31-44-61 web: 2021-09-27T19:11:17.022Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815260 hash=0xfdfd284be797a50a77268024acbcc779739335654e7c78f040477ffa59836760
  | 2021-09-27T19:11:17.172Z | Sep 27 19:11:17 ip-172-31-44-61 web: 2021-09-27T19:11:17.035Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815260 hash=0xfdfd284be797a50a77268024acbcc779739335654e7c78f040477ffa59836760
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:17 ip-172-31-44-61 web: 2021-09-27T19:11:17.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815260 hash=0xfdfd284be797a50a77268024acbcc779739335654e7c78f040477ffa59836760
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.449Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327716,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.449Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327716e+15
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.449Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327716,"result":"0xc6ee6"}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854863,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7976bc88d6177d65f37a0e1fc55652f928103bd1","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854859,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x6aeca9a564c210bab9d40bd6c8a41683d8d9cdba","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854859e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854859,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854863e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854863,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.525Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854861,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x37a09fa7afbbb48537bf94d5609d3e2fe04fa791","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854861e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854861,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854867,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x3ed6bef5274f9f5556c5cb30ff75a45d2882bfa6","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854867e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854865,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x9005d7e31f165e1dd7f443974340baa3d45c0e55","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854865e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854865,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.526Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854867,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.529Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854869,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x51d4b51f9251912c85af18af6e526ecb976ad810","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.529Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854869e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854869,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.529Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854871,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xaf99678f49fbb9c15ce995d6ba6ad0dbee856304","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854871e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854871,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854873,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x68200aa9f88d9f185e7bb7e0ea20dd3d773d1a3f","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854873e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854873,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854875,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2adddfff3f64be2cec1af49cf90c60c8a0f51c20","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854875e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.530Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854875,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854877,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2079a994e9bbc88496df03a59a0cfa2c9e2af473","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854877e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854877,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854879,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xf0d6de16e2c14e0598a18c2b43716d7d0672690c","0xc709c"]}
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854879e+14
  | 2021-09-27T19:11:19.673Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815260 header"
  | 2021-09-27T19:11:21.674Z | Sep 27 19:11:19 ip-172-31-44-61 web: 2021-09-27T19:11:19.531Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854879,"error":{"code":-32600,"message":"Error fetching block number 815260 header"}}"
  | 2021-09-27T19:11:21.674Z | Sep 27 19:11:21 ip-172-31-44-61 web: 2021-09-27T19:11:21.642Z [DEBUG] polygon.jsonrpc: handle: request={"id":"b3f689b8-f6bb-45e2-a724-274fdf49dd56","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:21.674Z | Sep 27 19:11:21 ip-172-31-44-61 web: 2021-09-27T19:11:21.642Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=b3f689b8-f6bb-45e2-a724-274fdf49dd56
  | 2021-09-27T19:11:24.925Z | Sep 27 19:11:21 ip-172-31-44-61 web: 2021-09-27T19:11:21.642Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":"b3f689b8-f6bb-45e2-a724-274fdf49dd56","result":"0xc6ee6"}
  | 2021-09-27T19:11:24.925Z | Sep 27 19:11:24 ip-172-31-44-61 web: 2021-09-27T19:11:24.762Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327718,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:24.925Z | Sep 27 19:11:24 ip-172-31-44-61 web: 2021-09-27T19:11:24.762Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327718e+15
  | 2021-09-27T19:11:27.176Z | Sep 27 19:11:24 ip-172-31-44-61 web: 2021-09-27T19:11:24.762Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327718,"result":"0xc6ee6"}
  | 2021-09-27T19:11:27.176Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.061Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815261 hash=0x9d782c32a089792e5e88a8a3c2e2f41c5141c8ec83e01743e917e10b56a90017
  | 2021-09-27T19:11:27.176Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.073Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815261 hash=0x9d782c32a089792e5e88a8a3c2e2f41c5141c8ec83e01743e917e10b56a90017
  | 2021-09-27T19:11:27.176Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.080Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815261 hash=0x9d782c32a089792e5e88a8a3c2e2f41c5141c8ec83e01743e917e10b56a90017
  | 2021-09-27T19:11:27.426Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.080Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815261 hash=0x9d782c32a089792e5e88a8a3c2e2f41c5141c8ec83e01743e917e10b56a90017
  | 2021-09-27T19:11:27.426Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.186Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854881,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xc709d",false]}
  | 2021-09-27T19:11:27.426Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.186Z [DEBUG] polygon.dispatcher: request: method=eth_getBlockByNumber id=1.80474746854881e+14
  | 2021-09-27T19:11:29.177Z | Sep 27 19:11:27 ip-172-31-44-61 web: 2021-09-27T19:11:27.186Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":180474746854881,"result":null}
  | 2021-09-27T19:11:29.177Z | Sep 27 19:11:29 ip-172-31-44-61 web: 2021-09-27T19:11:29.018Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815262 hash=0x13ca0b8f3f1a40e6841ea9d46175390a1c7398546baf1be51c6434417071a78d
  | 2021-09-27T19:11:29.177Z | Sep 27 19:11:29 ip-172-31-44-61 web: 2021-09-27T19:11:29.034Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815262 hash=0x13ca0b8f3f1a40e6841ea9d46175390a1c7398546baf1be51c6434417071a78d
  | 2021-09-27T19:11:29.177Z | Sep 27 19:11:29 ip-172-31-44-61 web: 2021-09-27T19:11:29.035Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815262 hash=0x13ca0b8f3f1a40e6841ea9d46175390a1c7398546baf1be51c6434417071a78d
  | 2021-09-27T19:11:31.178Z | Sep 27 19:11:29 ip-172-31-44-61 web: 2021-09-27T19:11:29.036Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815262 hash=0x13ca0b8f3f1a40e6841ea9d46175390a1c7398546baf1be51c6434417071a78d
  | 2021-09-27T19:11:31.178Z | Sep 27 19:11:31 ip-172-31-44-61 web: 2021-09-27T19:11:31.020Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815263 hash=0x059865df300285b231df3e29528ec73403c4d3778007986385f089b9686ff36b
  | 2021-09-27T19:11:31.178Z | Sep 27 19:11:31 ip-172-31-44-61 web: 2021-09-27T19:11:31.021Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815263 hash=0x059865df300285b231df3e29528ec73403c4d3778007986385f089b9686ff36b
  | 2021-09-27T19:11:31.178Z | Sep 27 19:11:31 ip-172-31-44-61 web: 2021-09-27T19:11:31.040Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815263 hash=0x059865df300285b231df3e29528ec73403c4d3778007986385f089b9686ff36b
  | 2021-09-27T19:11:33.179Z | Sep 27 19:11:31 ip-172-31-44-61 web: 2021-09-27T19:11:31.044Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815263 hash=0x059865df300285b231df3e29528ec73403c4d3778007986385f089b9686ff36b
  | 2021-09-27T19:11:33.179Z | Sep 27 19:11:33 ip-172-31-44-61 web: 2021-09-27T19:11:33.024Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815264 hash=0x1dd43d65c280abc183069b309216b04dbdcff71ef7c24d604447f47971cce11d
  | 2021-09-27T19:11:33.179Z | Sep 27 19:11:33 ip-172-31-44-61 web: 2021-09-27T19:11:33.024Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815264 hash=0x1dd43d65c280abc183069b309216b04dbdcff71ef7c24d604447f47971cce11d
  | 2021-09-27T19:11:33.179Z | Sep 27 19:11:33 ip-172-31-44-61 web: 2021-09-27T19:11:33.033Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815264 hash=0x1dd43d65c280abc183069b309216b04dbdcff71ef7c24d604447f47971cce11d
  | 2021-09-27T19:11:36.681Z | Sep 27 19:11:33 ip-172-31-44-61 web: 2021-09-27T19:11:33.035Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815264 hash=0x1dd43d65c280abc183069b309216b04dbdcff71ef7c24d604447f47971cce11d
  | 2021-09-27T19:11:36.681Z | Sep 27 19:11:36 ip-172-31-44-61 web: 2021-09-27T19:11:36.456Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327720,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:36.681Z | Sep 27 19:11:36 ip-172-31-44-61 web: 2021-09-27T19:11:36.456Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.68181123232772e+15
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:36 ip-172-31-44-61 web: 2021-09-27T19:11:36.456Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327720,"result":"0xc6ee6"}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.495Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327722,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.495Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327722e+15
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.495Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327722,"result":"0xc6ee6"}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.569Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854883,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x6aeca9a564c210bab9d40bd6c8a41683d8d9cdba","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.569Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854883e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.569Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.569Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854883,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854885,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x37a09fa7afbbb48537bf94d5609d3e2fe04fa791","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854885e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854885,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854887,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7976bc88d6177d65f37a0e1fc55652f928103bd1","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854887e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.573Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854887,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854893,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x51d4b51f9251912c85af18af6e526ecb976ad810","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854893e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854893,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854889,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x9005d7e31f165e1dd7f443974340baa3d45c0e55","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854889e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854889,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854891,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x3ed6bef5274f9f5556c5cb30ff75a45d2882bfa6","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854891e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.574Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854891,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854895,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xaf99678f49fbb9c15ce995d6ba6ad0dbee856304","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854895e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854895,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854899,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2adddfff3f64be2cec1af49cf90c60c8a0f51c20","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854899e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854901,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2079a994e9bbc88496df03a59a0cfa2c9e2af473","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854901e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854899,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854901,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854897,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x68200aa9f88d9f185e7bb7e0ea20dd3d773d1a3f","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.578Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854897e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.579Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.579Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854897,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.580Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854903,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xf0d6de16e2c14e0598a18c2b43716d7d0672690c","0xc70a0"]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.580Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854903e+14
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.580Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815264 header"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.580Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854903,"error":{"code":-32600,"message":"Error fetching block number 815264 header"}}"
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.640Z [DEBUG] polygon.jsonrpc: handle: request={"id":"7b3c3899-1fea-4439-aa23-371244e373d2","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:39.683Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.640Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=7b3c3899-1fea-4439-aa23-371244e373d2
  | 2021-09-27T19:11:43.184Z | Sep 27 19:11:39 ip-172-31-44-61 web: 2021-09-27T19:11:39.640Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":"7b3c3899-1fea-4439-aa23-371244e373d2","result":"0xc6ee6"}
  | 2021-09-27T19:11:43.184Z | Sep 27 19:11:43 ip-172-31-44-61 web: 2021-09-27T19:11:43.074Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmJzMKm6c3HUAVzH1oQ6aV3QRzMrEvz5TVtcbvuY2oH54x number=815265 hash=0xc6c0206d8c31ae98391cfbca5e0f9a61224b71268b7747c3ef5080f4a63dc009
  | 2021-09-27T19:11:43.184Z | Sep 27 19:11:43 ip-172-31-44-61 web: 2021-09-27T19:11:43.097Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=815265 hash=0xc6c0206d8c31ae98391cfbca5e0f9a61224b71268b7747c3ef5080f4a63dc009
  | 2021-09-27T19:11:43.184Z | Sep 27 19:11:43 ip-172-31-44-61 web: 2021-09-27T19:11:43.098Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm9SX9R9oqi1afyw5Qg1JgNvZfbPwbSEQH5NCBuBNWk8Lh number=815265 hash=0xc6c0206d8c31ae98391cfbca5e0f9a61224b71268b7747c3ef5080f4a63dc009
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:43 ip-172-31-44-61 web: 2021-09-27T19:11:43.098Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmGGueWQo4oudvL6uWGXGMorEHvT7GyUtYN4t2yNi6rujU number=815265 hash=0xc6c0206d8c31ae98391cfbca5e0f9a61224b71268b7747c3ef5080f4a63dc009
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.835Z [DEBUG] polygon.jsonrpc: handle: request={"id":2681811232327724,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.835Z [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=2.681811232327724e+15
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.835Z [DEBUG] polygon.jsonrpc: handle: response={"jsonrpc":"2.0","id":2681811232327724,"result":"0xc6ee6"}
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.913Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854906,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x6aeca9a564c210bab9d40bd6c8a41683d8d9cdba","0xc70a1"]}
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.913Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854906e+14
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.913Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815265 header"
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.913Z [DEBUG] polygon.jsonrpc: handle: response="{"jsonrpc":"2.0","id":180474746854906,"error":{"code":-32600,"message":"Error fetching block number 815265 header"}}"
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.919Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854910,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7976bc88d6177d65f37a0e1fc55652f928103bd1","0xc70a1"]}
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.919Z [DEBUG] polygon.jsonrpc: handle: request={"id":180474746854908,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x37a09fa7afbbb48537bf94d5609d3e2fe04fa791","0xc70a1"]}
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.919Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.8047474685491e+14
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.919Z [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=1.80474746854908e+14
  | 2021-09-27T19:11:44.935Z | Sep 27 19:11:44 ip-172-31-44-61 web: 2021-09-27T19:11:44.919Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getBalance err="Error fetching block number 815265 header"

Missing call to close LevelDB connection for /trie dir on close

Description

When stopping the process gracefully, the Close() function in server.go closes the blockchain layer which closes the levelDB session for the /blockchain folder. It looks like the leveldb.DB for the stateStorage (/trie) stays open.

For example when stopping the process at 18:41, one can see in the LOG for /blockchain:

18:26:25.559234 table@remove removed @10296
18:26:25.559429 table@remove removed @10297
18:41:23.591281 db@close closing
18:41:23.591489 db@close done T·206.365µs
=============== Sep 29, 2021 (UTC) ===============
18:42:14.243688 log@legend F·NumFile S·FileSize N·Entry C·BadEntry B·BadBlock Ke·KeyError D·DroppedEntry L·Level Q·SeqNum T·TimeElapsed
18:42:14.244604 version@stat F·[0 60 224] S·545MiB[0B 98MiB 447MiB] Sc·[0.00 0.98 0.45]
18:42:14.244632 db@open opening
18:42:14.245225 journal@recovery F·1
18:42:14.245469 journal@recovery recovering @25440
18:42:14.280050 memdb@flush created L0@25508 N·1675 S·172KiB "b\x00k..\b\x9a\x05,v6098864":"r\xffK..\xc5\xedW,v6099554"
18:42:14.280296 version@stat F·[1 60 224] S·545MiB[172KiB 98MiB 447MiB] Sc·[0.25 0.98 0.45]
18:42:14.312486 db@janitor F·288 G·1
18:42:14.312506 db@janitor removing table-25439
18:42:14.312570 db@open done T·67.929489ms

/trie/LOG however contains:

18:26:23.146333 db@janitor F·15 G·0
18:26:23.146362 db@open done T·76.615875ms
=============== Sep 29, 2021 (UTC) ===============
18:42:14.203082 log@legend F·NumFile S·FileSize N·Entry C·BadEntry B·BadBlock Ke·KeyError D·DroppedEntry L·Level Q·SeqNum T·TimeElapsed
18:42:14.203297 version@stat F·[2 11] S·22MiB[85KiB 22MiB] Sc·[0.50 0.22]
18:42:14.203325 db@open opening
18:42:14.203414 journal@recovery F·1
18:42:14.203984 journal@recovery recovering @167
18:42:14.235378 memdb@flush created L0@169 N·248 S·48KiB "\x03\xbe\xc2..h\x1a>,v112665":"\xfdO\x16..\x1dٚ,v112616"
18:42:14.235692 version@stat F·[3 11] S·22MiB[134KiB 22MiB] Sc·[0.75 0.22]
18:42:14.243432 db@janitor F·16 G·0
18:42:14.243482 db@open done T·40.147145ms

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 96be904
branch that causes this issue develop

Steps to reproduce

Stopped process running server
Attempted to make a backup of data-dir by uploading a .tar.gz to AWS S3
Downloaded the .tar.gz and saw error while attempting to extract /trie directory
Observed db@close in /blockchain/LOG
Did not observe db@close in /trie/LOG

Expected behaviour

The trie db session should release its lock and close gracefully.

Actual behaviour

The trie db session doesn't appear to close.

Restart with failure for the last node

[Subject of the issue]

Restarted like the last client sever for doc example after stop all nodes. The three other without problem restarted.
go run main.go server --data-dir ./test-chain-4 --chain genesis.json --grpc :40000 --libp2p :40001 --jsonrpc :40002 --seal

Description

2021-06-05T16:32:54.961+0200 [INFO] polygon.jsonrpc: http server started: addr=127.0.0.1:40002
2021-06-05T16:32:54.961+0200 [INFO] polygon.consensus.ibft: syncing past snapshots: from=0 to=4162
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc8ad97]

goroutine 1 [running]:
github.com/0xPolygon/minimal/consensus/ibft.(*Snapshot).Copy(0x0, 0x0)
/home/robert/Documents/polygonsdk/polygon-sdk/consensus/ibft/snapshot.go:367 +0x37
github.com/

Describe your issue in as much detail as possible here

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Getting error for getCode

Below is my error:
failed to dispatch: method=eth_getCode err="given root and slot not found in storage"

This is based on the docs provided. I am trying to use Metamask to send transaction.

Cannot send txs using chainId other than 100 when running `server`

Description

When trying to send txs using MetaMask (eth_sendRawTransaction), tx signature encoding fails when using a genesis file created with --chainid 647. I observed the error message below after trying to do a simple transaction (sending the premined account 100eth from itself):

2021-08-17T20:07:47.398Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_sendRawTransaction err="invalid sender"

Upon testing again using chainId 100, I no longer saw this error.

It looks like this is related to an issue reported earlier (see #99).

Your environment

OS and version OSX 11.4
version of the Polygon SDK d8b80b9
branch that causes this issue develop

Steps to reproduce

Set chainId in genesis file to be 647.
Spin up 4 validator nodes using server, with validator-1 being the premined account.
Using MetaMask, connect to the RPC URL, inputting 647 for the chainId.
Import premined account privateKey and see that I have the starting balance in my account.
Try sending 100ETH to the same account, observe a brief MetaMask error and errors in the logs indicating an "invalid sender".

Expected behaviour

The transaction should have went through without any issues when using a chainId other than 100.

Actual behaviour

The transaction failed to go through.

Proposed solution

Looking at #99, I see that using chainId 100 will avoid this issue however we'd like to use a number other than 100 when making our chain public.

Add custom operator protocols on the modules

Right now, minimal defines all the grpc protocols for all the modules in system.proto (peers, blockchain, txpool...). We should pass to each module the reference to the grpc server and let each module define his own system.proto spec.

`from` is required in the SDK but optional in the spec causing `eth_call` to often fail

Description

In the spec https://eth.wiki/json-rpc/API#eth_call and in the geth docs https://geth.ethereum.org/docs/rpc/ns-eth, the from parameter is marked optional, presumably for methods where the caller's data is not required (for example, retrieving a token name, retrieving a token's decimal count), but it is required here https://github.com/0xPolygon/polygon-sdk/blob/develop/jsonrpc/dispatcher.go#L405

This causes calls like this (from metamask)

2021-07-29T14:40:40.962-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request={"id":"109b6e95-32ed-4c01-a842-3ea8ee1e5897","jsonrpc":"2.0","method":"eth_call","params":[{"to":"0xDA4e8F6A6af5909dDC6E38eA6e490099d88c0139","data":"0x95d89b41"},"0x1e"]}

to fail, which is actually the call to retrieve the token symbol:

  /**
     * @dev Returns the symbol of the token, usually a shorter version of the
     * name.
     */
    function symbol() public view returns (string memory) {
        return _symbol;
    }

SDK logs:

    2021-07-29T18:15:33.538Z [DEBUG] polygon.dispatcher: request: method=eth_call id=0b198c7b-9cf9-4373-8ae8-552805ec3d91
2021-07-29T18:15:33.538Z [ERROR] polygon.dispatcher: failed to dispatch: method=eth_call err="from is empty"

Your environment

OS and version Ubuntu 20
version of the Polygon SDK a0618c4
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know

https://github.com/0xPolygon/polygon-sdk/blob/develop/jsonrpc/dispatcher.go#L405 should be marked optional. Maybe a new decodeTxn is needed for eth_call specifically? Not sure how this affects the tx execution down the line.
Which commands triggered the issue, if any
Going into metamask and adding a token deployed on the polygon SDK in dev mode. It'll error when trying to auto populate the token name and decimals

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Add e2e test to check the premined balance

Small e2e test to validate that the balance endpoint works.

Steps:

Create a testserver with some dummy premined account.
Use the GetBalance method to query the account.

Segmentation violation when adding txn with gRPC client

[Segmentation violation when adding txn with gRPC client]

Description

Hi, I am currently working on a TypeScript gRPC client for this SDK (available here).

I was adding the support for the Txn Pool Operator, but when I was testing my addTxn function a SIGSEGV happened.

Your environment

OS: Pop OS 21.04
Polygon SDK Version: 0.1.0
Branch: default

Steps to reproduce

First, I started a dev server using the command provided here.

go run main.go genesis --premine 0x1010101010101010101010101010101010101010
go run main.go server --chain genesis.json --dev --log-level debug --grpc :9632

I generated the protobuf files for my client using the following command and the txpool/proto/operator.proto file of the polygon-sdk repository on branch develop:

yarn proto-loader-gen-types --grpcLib=@grpc/grpc-js --outDir=src/proto proto/*.proto

Then using my client I ran the following test with the command yarn test:

const client = new PolygonTxnPoolOperatorClient('localhost', 9632);

await client.addTxn(null, '0x1010101010101010101010101010101010101010');

where PolygonTxnPoolOperatorClient is the following:

class PolygonTxnPoolOperatorClient {
    private readonly address: string;
    private client: TxnPoolOperatorClient;

    constructor(hostname: string, port: number) {
        this.address = `${hostname}:${port}`;
	this.client = new grpcObj.v1.TxnPoolOperator(this.address, credentials.createInsecure());
    }

    addTxn(raw: Any | null, from: string): Promise<Empty__Output> {
        return new Promise((resolve, reject) => {
            const addTxnReq: AddTxnReq = {
                raw,
		from,
            };

	    this.client.addTxn(addTxnReq, (error?: ServiceError, result?: Empty__Output) => {
	        if (error) {
		    return reject(error);
		}
	        if (result) {
		    return resolve(result);
	        }
		throw new Error('unknown error: no error and status could be retrieved from the request');
	    });
        });
    }
}

Expected behaviour

I was actually trying to understand the behaviour of this method ^^.

Actual behaviour

The polygon server crashed because of a segmentation violation.

Logs

➜  polygon-sdk git:(develop) ✗ go run main.go server --chain genesis.json --dev --log-level debug --grpc :9632
2021-08-01T20:11:06.750+0200 [INFO]  polygon: Data dir: path=./test-chain
2021-08-01T20:11:06.751+0200 [INFO]  polygon.network: LibP2P server running: addr=/ip4/127.0.0.1/tcp/43345/p2p/16Uiu2HAmF3obRnYGsdzThy2irJRAD3DoEiDgsrHbJR8QnhGKVW7x
2021-08-01T20:11:06.759+0200 [INFO]  polygon.blockchain: Current header: hash=0xb66970dc536ae4a18cc6c9b760f17afa6e31cf03d579a77cbf86ce276e8f602c number=0
2021-08-01T20:11:06.759+0200 [INFO]  polygon.blockchain: genesis: hash=0xb66970dc536ae4a18cc6c9b760f17afa6e31cf03d579a77cbf86ce276e8f602c
2021-08-01T20:11:06.759+0200 [INFO]  polygon: GRPC server running: addr=127.0.0.1:9632
2021-08-01T20:11:06.759+0200 [INFO]  polygon.jsonrpc: http server started: addr=127.0.0.1:8545
2021-08-01T20:11:06.759+0200 [INFO]  polygon.consensus.dev: consensus started
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xc68455]

goroutine 182 [running]:
github.com/0xPolygon/minimal/txpool.(*TxPool).AddTxn(0xc000596230, 0x1246360, 0xc0016f8c90, 0xc0016f5bc0, 0xc000596230, 0xc0016f8c90, 0xc0004c5ba0)
        /home/ptitluca/GolandProjects/polygon-sdk/txpool/operator.go:23 +0x55
github.com/0xPolygon/minimal/txpool/proto._TxnPoolOperator_AddTxn_Handler(0xec1180, 0xc000596230, 0x1246360, 0xc0016f8c90, 0xc0016fac60, 0x0, 0x1246360, 0xc0016f8c90, 0xc00003a5a0, 0x2c)
        /home/ptitluca/GolandProjects/polygon-sdk/txpool/proto/operator_grpc.pb.go:151 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00033e540, 0x124e798, 0xc000a70780, 0xc0004b2100, 0xc000516660, 0x1a37db8, 0x0, 0x0, 0x0)
        /home/ptitluca/GolandProjects/polygon-sdk/vendor/google.golang.org/grpc/server.go:1217 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00033e540, 0x124e798, 0xc000a70780, 0xc0004b2100, 0x0)
        /home/ptitluca/GolandProjects/polygon-sdk/vendor/google.golang.org/grpc/server.go:1540 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc001f20720, 0xc00033e540, 0x124e798, 0xc000a70780, 0xc0004b2100)
        /home/ptitluca/GolandProjects/polygon-sdk/vendor/google.golang.org/grpc/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/ptitluca/GolandProjects/polygon-sdk/vendor/google.golang.org/grpc/server.go:876 +0x1fd
exit status 2

Proposed solution

The error that caused the segmentation violation is self explaining : I sent a null raw field in the transaction and the server is trying to access it's content in the txpool/operator.go file, in the function AddTxn on line 23 :

// AddTxn adds a local transaction to the pool
21 func (t *TxPool) AddTxn(ctx context.Context, raw *proto.AddTxnReq) (*empty.Empty, error) {
22	txn := new(types.Transaction)
23	if err := txn.UnmarshalRLP(raw.Raw.Value); err != nil {
24		return nil, err
25	}
26
	if raw.From != "" {
		from := types.Address{}
		if err := from.UnmarshalText([]byte(raw.From)); err != nil {
			return nil, err
		}
		txn.From = from
	}

	if err := t.AddTx(txn); err != nil {
		return nil, err
	}

	return &empty.Empty{}, nil
}

Should you check the content of the raw parameter, and return an error if it is actually nil ?
Like this:

// AddTxn adds a local transaction to the pool
func (t *TxPool) AddTxn(ctx context.Context, raw *proto.AddTxnReq) (*empty.Empty, error) {
	if raw.Raw == nil {
		return nil, fmt.Errorf("transaction's field raw is empty")
	}

	txn := new(types.Transaction)
...

use web3.eth.getBlock the return data transactions is empty

Description

first step: use sendTransaction add txs:

web3.eth.sendTransaction({from:"0x3eCf5195c8cf2166198C62c677C379975e0304aB",to:"0x8844B6D45fCE2fd6D030B446893f61aA01Ea672E",value:web3.utils.toWei("100","ether")})

{ root:
'0x0000000000000000000000000000000000000000000000000000000000000000',
cumulativeGasUsed: 21000,
logsBloom:
'0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
logs: [],
status: true,
transactionHash:
'0x17c93192ab09357c63a7b31d329d836e0a7a0eb5d6aaa0db8d0c9bda24f86c53',
transactionIndex: 0,
blockHash:
'0x9f7bb58565b5d587752e3828ee34e0de15921e6346af277f918cd1ae30be8fa4',
blockNumber: 196,
gasUsed: 21000,
contractAddress: '0x0000000000000000000000000000000000000000',
from: '0x3eCf5195c8cf2166198C62c677C379975e0304aB',
to: '0x8844B6D45fCE2fd6D030B446893f61aA01Ea672E' }'

the blockNumber this 196

second step: use getBlock(196):

web3.eth.getBlock(196)

{ parentHash:
'0x702f7dfea12d7bd5a875dec995d67f097cb5eedba3332ddfad7eeb94f3fd119f',
sha3Uncles:
'0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
miner: '0x0000000000000000000000000000000000000000',
stateRoot:
'0xa86d1466efd9ad7e54e602494ad1bf2e0c0964604de1c270cf292cfb9d349f2e',
transactionsRoot:
'0x7029a401649226aca7aa865a6c5cac10246b70a59b22e0b2ad66d28871ca1ab5',
receiptsRoot:
'0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2',
logsBloom:
'0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
difficulty: '196',
totalDifficulty: '196',
size: 0,
number: 196,
gasLimit: '0x5f5e100',
gasUsed: '0x5208',
timestamp: 1628758675,
extraData:
'0x0000000000000000000000000000000000000000000000000000000000000000f90164f8549401e8a1a5b973121230f3a1f67d3def3351c1f99194991f9bbb04113002c4678e50a199966e396b4246940c0816ddc9d48c01ee00d56054966f6925291c71945556e89b52249b7dfdb7e36f814b0a55d89ab574b8411278b54a0943d45dd5f80bcbe6897e8bc4cf196aeda7c14314bdda3dd3a0aeab2ad8aca4bce271f87c65cbb8137a0c96bcae9826353d8e5f4e871c06b9340e4f01f8c9b841d2581b1fa90d16d53755375af40141917b4d6edc7f46bd48a86daa0bc4eeb2eb091085e016c57b61cd4271afec0560398ebaaaba4c76b4dabb0f614e6c184e8a01b841f0499bd11870daee784a79ef8747be136d54fbf91944103ea0746173b53c82a8727ed627b5ba85a858207a9b35870e43787b09e6ab3602d31d7b3ac7e8bd514d00b841d561196c9d70c63947a8e7c84908fe5498b539d8b275443312b2ad5a49cfaf6576a8b796193d33680710055102e9c75456babc4772062a5822a06e95ca769dea01',
mixHash:
'0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365',
nonce: '0x0000000000000000',
hash:
'0x9f7bb58565b5d587752e3828ee34e0de15921e6346af277f918cd1ae30be8fa4',
transactions: [],
uncles: [] }

But, the transactions is empty ? this is a bug?

Your environment

OS and version : MacOS Big Sur
version of the Polygon SDK : 0.1.0
branch that causes this issue: develop , The code is up to date

"invalid json request" for eth_blockNumber and eth_getBlockByNumber requests

Invalid json request for eth_blockNumber and eth_getBlockByNumber requests

Description

It seems like the sdk sometimes doesn't like eth_blockNumber and eth_getBlockByNumber calls.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 2f50a194f21d972b542ccd1e1571ccba40c5c810
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue

Modify jsonrpc.go to add some logging:

unc (j *JSONRPC) handle(w http.ResponseWriter, req *http.Request) {
	w.Header().Set("Access-Control-Allow-Origin", "*")
	w.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS")
	w.Header().Set("Access-Control-Allow-Headers", "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")

	if (*req).Method == "OPTIONS" {
		return
	}

	buf := new(bytes.Buffer)
    buf.ReadFrom(req.Body)
    j.logger.Debug("handle request", "request body", buf.String())

	handleErr := func(err error) {
		j.logger.Debug("handle request", "error", err.Error())
		w.Write([]byte(err.Error()))
	}
	if req.Method == "GET" {
		w.Write([]byte("PolygonSDK JSON-RPC"))
		return
	}
	if req.Method != "POST" {
		w.Write([]byte("method " + req.Method + " not allowed"))
		return
	}
	data, err := ioutil.ReadAll(req.Body)
	if err != nil {
		handleErr(err)
		return
	}
	resp, err := j.dispatcher.Handle(data)
	if err != nil {
		handleErr(err)
		return
	}
	j.logger.Debug("handle request", "response", string(resp))

	w.Write(resp)
}

go build main.go
./main dev --log-level DEBUG --premine [YOUR_METAMASK_ADDRESS]

Open up metamask, connect to localhost:8545. You'll see logs like this

./main dev --log-level DEBUG --premine 0x4343B00319a766DeF5b2039581840EB5ea691e1b
2021-07-22T18:53:27.879-0400 [INFO]  polygon-dev: Data dir: path=./test-chain
2021-07-22T18:53:27.881-0400 [INFO]  polygon-dev.network: LibP2P server running: addr=/ip4/127.0.0.1/tcp/33961/p2p/16Uiu2HAmHbD5Reoo7NSK4kq5tMNAyowHaqy1ozjFkDpKQcwvNM5c
2021-07-22T18:53:27.899-0400 [INFO]  polygon-dev.blockchain: Current header: hash=0x25c197e231a2b456a6ffc44a625e277e4f7ddede8b30e39ca1fd0b8adf75b642 number=0
2021-07-22T18:53:27.899-0400 [INFO]  polygon-dev.blockchain: genesis: hash=0x25c197e231a2b456a6ffc44a625e277e4f7ddede8b30e39ca1fd0b8adf75b642
2021-07-22T18:53:27.900-0400 [INFO]  polygon-dev: GRPC server running: addr=127.0.0.1:9632
2021-07-22T18:53:27.900-0400 [INFO]  polygon-dev.jsonrpc: http server started: addr=127.0.0.1:8545
2021-07-22T18:53:27.900-0400 [INFO]  polygon-dev.consensus.dev: consensus started
2021-07-22T18:53:28.714-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":5704491462363075,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:28.714-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:28.719-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":2599990714646,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:53:28.719-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:28.721-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":"a037547b-13d1-4db3-98ad-95e735c437f9","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:28.721-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:29.982-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":7984741462524145,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:29.982-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:29.986-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253605,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:53:29.986-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:48.718-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":5704491462363076,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:48.718-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:48.720-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":2599990714647,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:53:48.720-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:48.721-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":"ed421703-562e-45e7-b60f-cfecd568ae86","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:48.721-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:49.987-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":7984741462524146,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:49.987-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:49.991-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253606,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:53:49.991-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:53:49.993-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":"bc5e2986-4199-4c2d-a1bc-cc801f3900fc","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:53:49.993-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:08.722-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":5704491462363077,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:54:08.722-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:08.726-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":2599990714648,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:54:08.726-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:08.726-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":"87145f41-8bb1-4ce7-9e18-5fc8128391f7","jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:54:08.726-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:09.992-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":7984741462524147,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}
2021-07-22T18:54:09.992-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:09.997-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253607,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[null,false]}
2021-07-22T18:54:09.997-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:09.998-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253608,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x6f75ba6c90e3da79b7acafc0fb9cf3968aa4ee39",null]}
2021-07-22T18:54:09.998-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253609,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x0a6b4bc17fc45f6bde245038cb01550e24d70d88",null]}
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253610,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x926be1443b9bc2c948f3cc4a4ccf62e7c1abb232",null]}
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253611,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x20d1c859808518388d4022f761d0ab762ea43cd6",null]}
2021-07-22T18:54:10.006-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.007-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253612,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x5b79a0eb0485c207993cf346cca5faa737637f3b",null]}
2021-07-22T18:54:10.007-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.007-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253613,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7ace2d63c8bd45309afb7e92b9dcd9a36ebe2b50",null]}
2021-07-22T18:54:10.007-0400 [DEBUG] polygon-dev.jsonrpc: handle request: error="{"code":-32600,"message":"invalid json request"}"
2021-07-22T18:54:10.008-0400 [DEBUG] polygon-dev.jsonrpc: handle request: request body={"id":820926693253614,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x025b9fd9f1ed818d688e19c5f43c500bf44cea47",null]}

Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

It should not return invalid response for these requests.

Actual behaviour

Returns invalid response

Logs

Targeting a real cluster (with blocks and not in dev mode), we get similar errors (these are elixir errors from blockscout). Can confirm these blocks exist.

2021-07-22T19:05:50.638 application=indexer fetcher=block_catchup first_block_number=9620 last_block_number=9611 [error] ** (FunctionClauseError) no function clause matching in EthereumJSONRPC.HTTP.standardize_response/1
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc/http.ex:136: EthereumJSONRPC.HTTP.standardize_response(%{"code" => -32600, "message" => "invalid json request"})
    (elixir 1.11.4) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc/http.ex:45: EthereumJSONRPC.HTTP.chunked_json_rpc/3
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc/request_coordinator.ex:87: anonymous fn/3 in EthereumJSONRPC.RequestCoordinator.perform/4
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc/request_coordinator.ex:107: EthereumJSONRPC.RequestCoordinator.trace_request/2
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc.ex:482: EthereumJSONRPC.fetch_blocks_by_params/3
    (indexer 0.1.0) lib/indexer/block/fetcher.ex:131: Indexer.Block.Fetcher.fetch_and_import_range/2
    (indexer 0.1.0) lib/indexer/block/catchup/fetcher.ex:194: Indexer.Block.Catchup.Fetcher.fetch_and_import_range_from_sequence/3
    (elixir 1.11.4) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.4) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3


Retrying.
[
  %{
    id: 0,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2530", true]
  },
  %{
    id: 1,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252F", true]
  },
  %{
    id: 2,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252E", true]
  },
  %{
    id: 3,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252D", true]
  },
  %{
    id: 4,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252C", true]
  },
  %{
    id: 5,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252B", true]
  },
  %{
    id: 6,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252A", true]
  },
  %{
    id: 7,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2529", true]
  },
  %{
    id: 8,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2528", true]
  },
  %{
    id: 9,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2527", true]
  }
]
[
  %{
    id: 0,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2530", true]
  },
  %{
    id: 1,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252F", true]
  },
  %{
    id: 2,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252E", true]
  },
  %{
    id: 3,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252D", true]
  },
  %{
    id: 4,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252C", true]
  },
  %{
    id: 5,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252B", true]
  },
  %{
    id: 6,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x252A", true]
  },
  %{
    id: 7,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2529", true]
  },
  %{
    id: 8,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2528", true]
  },
  %{
    id: 9,
    jsonrpc: "2.0",
    method: "eth_getBlockByNumber",
    params: ["0x2527", true]
  }
]

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Remove unnecessary blockchain interface methods in Jsonrpc

The JsonRPC module uses State and BeginTx methods https://github.com/0xPolygon/polygon-sdk/blob/develop/jsonrpc/blockchain.go#L41 to be used during gas estimation. However, this methods can be replaced by the methods GetAccount and ApplyTxn.

New node `fail to write block` error when syncing near most recent block

Description

When a new peer is syncing, it looks like after the last batch sync of blocks, we sometimes get a failure:

`[ERROR] polygon.consensus.syncer: failed to write block: err="parent of 0x5dda26ea7e3157baa4fa07746be00c43c2f084a946a2fb171634650f34efbc27 (853608) not found: 0xc2cbd6ac4ed7b7c340f6070bc2e13caef6ec582e14acffca41f49f9ae50760c9"`

See larger log snippet:

Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.442Z [INFO]  polygon.blockchain: write blocks: num=10 from=852347 to=852356 parent=0xcb750c6af46655acc5d8a2ef0ff555da7cace894c761da985a99a3d3862dbeb5
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.479Z [INFO]  polygon.blockchain: new block: number=852347 hash=0x12039ac1d933d9a0c4761ae84ecbb227a7e8232b2b590d0b90965037e6e9fe51 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.480Z [INFO]  polygon.blockchain: new block: number=852348 hash=0xe550e56c73dd0695a218cb509e993b18a0c7229e0a6923b2caee328c76b60029 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.482Z [INFO]  polygon.blockchain: new block: number=852349 hash=0xd1464576099a359200620cfa5e8c468b8efa61affe8843fef3260528a4b9f2c8 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.482Z [INFO]  polygon.blockchain: new block: number=852350 hash=0x36c9b9114dd72fa7c3d6c6c7d78910e2acec6a038e9860c1571ea1592b072e2a txns=0 generation_time_in_sec=10
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.483Z [INFO]  polygon.blockchain: new block: number=852351 hash=0xf1f1e6c70bf3caedb625e2087b5c1df5075bd1550c6cd2c7e914462e9479933e txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.485Z [INFO]  polygon.blockchain: new block: number=852352 hash=0x2a0f4cef8616c30d40d9dcaa9d421e31545fac74d047b758ae95fe07861714f5 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.486Z [INFO]  polygon.blockchain: new block: number=852353 hash=0x7d0b63698c309d5aab3f28d64bd0cd30ce73356c003151509ca9585b03a3c346 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.486Z [INFO]  polygon.blockchain: new block: number=852354 hash=0x1d83f22145427ccf4861da7ebfc5374ebd03ffd004e28f42669e5ba63b481e99 txns=1 generation_time_in_sec=10
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.489Z [INFO]  polygon.blockchain: new block: number=852355 hash=0x2d3147d271012dd927da8d0fc41f9c421594abe9a86c62ddd147fbbcc61f236e txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.490Z [INFO]  polygon.blockchain: new block: number=852356 hash=0x264a473677e736779edb91cc4753b90cf368d04f85395ea913114583599ee7e8 txns=0 generation_time_in_sec=2
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.490Z [INFO]  polygon.blockchain: new head: hash=0x264a473677e736779edb91cc4753b90cf368d04f85395ea913114583599ee7e8 number=852356
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.492Z [INFO]  polygon.blockchain: write block: num=853608 parent=0xc2cbd6ac4ed7b7c340f6070bc2e13caef6ec582e14acffca41f49f9ae50760c9
Sep 29 15:27:04 ip-172-31-25-125 main[72967]: 2021-09-29T15:27:04.496Z [ERROR] polygon.consensus.syncer: failed to write block: err="parent of 0x5dda26ea7e3157baa4fa07746be00c43c2f084a946a2fb171634650f34efbc27 (853608) not found: 0xc2cbd6ac4ed7b7c340f6070bc2e13caef6ec582e14acffca41f49f9ae50760c9"

Restarting the node fixes the issue.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 96be904
branch that causes this issue develop

Steps to reproduce

Have 4-node validator set writing and validating new blocks
Connect 5th peer and observe polygon.blockchain: new block output that occurs during syncing
When almost synced to the current block, we see the error above

Expected behaviour

There shouldn't be an error that requires us to restart the node.

Actual behaviour

An error stops the node from further syncing, requiring a restart.

ethjs-query formatting query for sendRawTransaction

When using Metamask and connected to the JSON-RPC url(as per demo video), when sending ETH, the metamask did not transfer any ETH as the error of sendRawTransaction failed to format 'undefined'.

Exact error is as below:

Block writing still stops in non-sealing nodes due to nil BestPeer

Block writing stops in non-sealing nodes due to nil BestPeer

Description

It looks like we are still observing the same issue as originally described in #167

After investigating, this seems to be related to some inconsistencies in how we determine the best peer to connect to within BestPeer(). We are seeing that at times, curDiff is always greater than bestTd, resulting in this function always returning nil.

We've adding some extra logging in BestPeer to help investigate:

func (s *Syncer) BestPeer() *syncPeer {
	var bestPeer *syncPeer
	var bestTd *big.Int

	for _, p := range s.peers {
		status := p.status
		if bestPeer == nil || status.Difficulty.Cmp(bestTd) > 0 {
			bestPeer, bestTd = p, status.Difficulty
		}
	}
	if bestPeer == nil {
		return nil
	}
	curDiff := s.blockchain.CurrentTD()
	fmt.Println("BestPeer() curDiff = ", curDiff)
	fmt.Println("BestPeer() bestTd = ", bestTd)
	if bestTd.Cmp(curDiff) <= 0 {
		return nil
	}
	return bestPeer
}

Although the change 5170883 was put in (to broadcast the header's difficulty instead of the block number), it appears that we are actually initializing the header's Difficulty to be the same value as the header's Number, resulting in a broadcast of the same value:

https://github.com/0xPolygon/polygon-sdk/blob/bba205e6ed4aad9522450558ed3dbad67664e723/consensus/ibft/ibft.go#L349-L353

Upon attempting to correct L353 above, we are still seeing the same issue as can be seen in our logs. Below the bestTd is always 464670654277. Furthermore, the difficulty outputted in the enqueue block debug statement has the same value as the block number:

2021-10-07T14:11:43.046Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW number=982519 hash=0xfbda9f200b380eeab2eeb9fb2e80bb25a925929b55c1f2a35d38dbc085f1c831
2021-10-07T14:11:43.046Z [DEBUG] polygon.consensus.syncer: update peer status: peer=16Uiu2HAmBv5UkctkZK1ui2tMip6XqpykvMnrzDuLGNjN38YDztmW latest block number=982519 latest block hash=0xfbda9f200b380eeab2eeb9fb2e80bb25a925929b55c1f2a35d38dbc085f1c831 difficulty=982519
2
BestPeer() curDiff =  482664423817
BestPeer() bestTd =  464670654277

I've put together the difficulties of the 5 peers currently connected:

peer1.status.Difficulty = 982668
peer2.status.Difficulty = 982668
peer3.status.Difficulty = 982668
peer4.status.Difficulty = 982668
peer5.status.Difficulty = 464670654277

In the above scenario, peer5 will always be our 'bestPeer', however its difficulty is always less than the current difficulty of 482664423817 so we will always return nil.

I would assume that if the Difficulty value for peer1-peer4 were sent correctly, we would connect to a better beer whose Difficulty would be greater than our current difficulty.

Your environment

OS and version Ubuntu 20
version of the Polygon SDK bba205e
branch that causes this issue develop

Steps to reproduce

start some non-sealing nodes connected to a cluster of 5 validators
observe that we can never connect to a peer who's difficulty is greater than the current block difficulty
restarting sometimes fixes the isssue

Expected behaviour

Gossip'd blocks should continue to be written to state and processed normally after bulk sync.

Actual behaviour

Although we still enqueue blocks, we are no longer writing them to state when we can't properly determine the best difficulty of a connected peer.

Runtime error after dispatcher fails to dispatch

Description

Running locally, we've seen what appears to be a runtime error likely right after the error: [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getCode err="given root and slot not found in storage".

Your environment

OS and version: OSX 11.4
version of the Polygon SDK: 6906ed9
branch that causes this issue: develop

Steps to reproduce

Tell us how to reproduce this issue

polygon-sdk server run with IBFT consensus of 4 nodes running for about 12 hours.
Where the issue is, if you know
Which commands triggered the issue, if any
polygon-sdk server run..

Expected behaviour

No runtime error.

Actual behaviour

Runtime error (see logs).

Logs

2021-07-16T10:39:20.673-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getTransactionCount err="given root and slot not found in storage"
2021-07-16T10:39:40.661-0400 [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=3.325198870728132e+15
2021-07-16T10:39:49.879-0400 [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=5.813328093453609e+15
2021-07-16T10:39:53.868-0400 [DEBUG] polygon.dispatcher: request: method=eth_gasPrice id=bfadc345-7925-44c1-bbe1-0af437654a53
2021-07-16T10:39:54.655-0400 [DEBUG] polygon.dispatcher: request: method=eth_getCode id=1d6f7e80-dd9b-4c76-afcb-4c404722bb40
2021-07-16T10:39:54.655-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getCode err="given root and slot not found in storage"
2021-07-16T10:39:54.661-0400 [DEBUG] polygon.dispatcher: request: method=eth_estimateGas id=8.258272843514e+12
2021/07/16 10:39:54 http: panic serving 127.0.0.1:63872: runtime error: index out of range [-1]
goroutine 185 [running]:
net/http.(*conn).serve.func1(0xc0006c6460)
    /usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x1a9e820, 0xc002119830)
    /usr/local/go/src/runtime/panic.go:971 +0x499
github.com/0xPolygon/minimal/crypto.RecoverPubkey(0x0, 0x0, 0x0, 0xc000581820, 0x20, 0x20, 0x7a, 0xc00209c200, 0xc000609070)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/crypto/crypto.go:128 +0x1aa
github.com/0xPolygon/minimal/consensus/ibft.ecrecoverImpl(0x0, 0x0, 0x0, 0xc0005817e0, 0x20, 0x20, 0x0, 0x0, 0xc000000000, 0xc0020282b0, ...)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/sign.go:22 +0xb8
github.com/0xPolygon/minimal/consensus/ibft.ecrecoverFromHeader(0xc000147d40, 0x0, 0x0, 0x0, 0xc000616c40, 0x4)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/sign.go:42 +0x119
github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).GetBlockCreator(0xc0006aaf70, 0xc000147d40, 0x0, 0x0, 0x0, 0x0, 0x269e2e0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/ibft.go:921 +0x45
github.com/0xPolygon/minimal/minimal.(*jsonRPCHub).ApplyTxn(0xc0016dec30, 0xc000147d40, 0xc0002d1080, 0xc000147d40, 0x5a5d896ffadb8ffe, 0xc000616bc0, 0xc000616be0, 0xc0006e2580, 0xc0002d0fd0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/minimal/server.go:281 +0x5a
github.com/0xPolygon/minimal/jsonrpc.(*Eth).EstimateGas.func1(0x7ca24, 0xc00212a110, 0x0, 0x0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/eth_endpoint.go:312 +0x7c
github.com/0xPolygon/minimal/jsonrpc.(*Eth).EstimateGas(0xc0005901b8, 0xc002058ac0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/eth_endpoint.go:324 +0x332
reflect.Value.call(0xc000426880, 0xc000590230, 0x13, 0x1b194ca, 0x4, 0xc00211ceb0, 0x3, 0x3, 0x19690a0, 0xc00069b038, ...)
    /usr/local/go/src/reflect/value.go:476 +0x8e7
reflect.Value.Call(0xc000426880, 0xc000590230, 0x13, 0xc00211ceb0, 0x3, 0x3, 0x0, 0xa0, 0xc000430a60)
    /usr/local/go/src/reflect/value.go:337 +0xb9
github.com/0xPolygon/minimal/jsonrpc.(*Dispatcher).handleReq(0xc0006004b0, 0x1994d80, 0xc002103f00, 0xc002103f10, 0xf, 0xc0020141e0, 0x94, 0xa0, 0xc00011a060, 0xc00011a060, ...)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/dispatcher.go:234 +0x4cd
github.com/0xPolygon/minimal/jsonrpc.(*Dispatcher).Handle(0xc0006004b0, 0xc0006d7800, 0xdd, 0x200, 0x200, 0x0, 0x0, 0xc001fa5b90, 0x1335676)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/dispatcher.go:209 +0x13f
github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handle(0xc0016df140, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/jsonrpc.go:149 +0x4b9
net/http.HandlerFunc.ServeHTTP(0xc00063eb70, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2069 +0x44
net/http.(*ServeMux).ServeHTTP(0x266d640, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2448 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00058e540, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0006c6460, 0x1e498c0, 0xc0005c6880)
    /usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3013 +0x39b
2021-07-16T10:39:54.665-0400 [DEBUG] polygon.dispatcher: request: method=eth_estimateGas id=8.258272843514e+12

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Block miner hash value always 0x0 in `eth_getBlockBy*` jsonrpc response

Description

According to https://eth.wiki/json-rpc/API#eth_getBlockByHash, the miner hash should be "the address of the beneficiary to whom the mining rewards were given" (or the address of the block's validator).

Using either the dev or server command - and tested on both eth_getBlockByNumber and eth_getBlockByHash, I always see the miner value being:

0x0000000000000000000000000000000000000000

I expect the miner value to be one of the validator's addresses.

Your environment

OS and version OSX 11.4
version of the Polygon SDK ad7b97e
branch that causes this issue develop

Steps to reproduce

Set up 4 nodes, each running ./main server... and ensure all 4 nodes have visibility of eachother:

2021-08-03T23:54:36.390Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=25530
2021-08-03T23:54:36.390Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
2021-08-03T23:54:36.390Z [INFO]  polygon.consensus.ibft: proposer calculated: proposer=0x40dd53141a759efd606a13d3e8a7c0aa005faf35 block=25530

Send HTTP request {"id":1,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", true]}
Observe that the miner of the JSON-RPC response object is always:

"miner":"0x0000000000000000000000000000000000000000"

Expected behaviour

I expect the miner value to be one of the validator's addresses. In my example above running 4 validators, I expect the miner hash to be the address of one of those 4 validators.

Actual behaviour

I always see the HTTP response containing:

"miner":"0x0000000000000000000000000000000000000000"

which makes it difficult to determine who mined the block.

Stopped nodes and blockchain.

[Subject of the issue]

I stopped it all 4 nodes by sending ethereum contract from remix and Metamask connected by rpc ...
No new blocks .... after restart too.

Description

2021-06-05T19:31:42.099+0200 [INFO] polygon.network: Peer connected: id=16Uiu2HAmJvQpodoNQk6Mq12GAS4GLR4JJqgXTTHFAGdN3ToRKMhV
2021-06-05T19:31:42.107+0200 [INFO] polygon.network: Peer connected: id=16Uiu2HAmPvMtVmPaTbxL92jdq9sSAPpvAgGRieCnT21tMrrXv7Zc
2021-06-05T19:31:47.112+0200 [DEBUG] polygon.consensus.ibft: state change: new=RoundChangeState
2021-06-05T19:31:47.113+0200 [DEBUG] polygon.consensus.ibft: cycle: state=RoundChangeState sequence=1 round=0
2021-06-05T19:31:47.113+0200 [DEBUG] polygon.consensus.ibft: local round change: round=1
2021-06-05T19:31:59.114+0200 [DEBUG] polygon.consensus.ibft: round change timeout
2021-06-05T19:31:59.114+0200 [DEBUG] polygon.consensus.ibft: local round change: round=2
and nothing more

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

JSON-RPC error object should be wrapped within Response object

##JSON-RPC error object should be nested within response object

Description

While adding support for batch JSON-RPC requests (#106), I noticed that the current implementation sends back error responses that aren't being nested within a response object - see https://www.jsonrpc.org/specification#error_object.

For example, the response body of an errored request currently appears as:

{"code":-32603,"message":"internal error"}

when it should actually be something more like:

{"id":1,"jsonrpc":"2.0", "error": {"code":-32603,"message":"internal error"} }

Your environment

OSX 11.4
develop
a0618c4 (latest on develop)

Steps to reproduce

Run ./main dev...
Used postman to send the following request (which errors since we are unable to get block by num 0):

{"id":1,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x0", true]}

Observed the following response:

{"code":-32603,"message":"internal error"}

Expected behaviour

The response should have the error object nested to appear as:

{"id":1,"jsonrpc":"2.0", "error": {"code":-32603,"message":"internal error"} }

Actual behaviour

Observed the following response instead:

{"code":-32603,"message":"internal error"}

Discovery peers e2e tests

Implement a new e2e case for the network discovery of new nodes.

Create a n number of nodes.
Join the nodes in serial: A -> B -> C
Wait for one minute, after that, all the nodes should know each other.

Before we implement the test we need to fix the Peers operator methods (here) to use the new list of connected peers instead.

Add e2e tests for simple transfer transactions

We should add an e2e test that transfers ether between two accounts.

Request Support for BLS12 based signature verification

Hi Team!

Since many protocols are putting BLS12 to use for digital signatures.
I want to verify BLS 12 based signature from the polygon smart contract.

Hoping for some positive inputs/response on this.

The program will exit error： concurrent map read and map write

Description

The program will exit when 200 transitions request together 。

Logs

concurrent map read and map write

goroutine 55 [running]:
runtime.throw(0x1051d9ff3, 0x21)
/opt/homebrew/Cellar/go/1.15.7_1/libexec/src/runtime/panic.go:1117 +0x54 fp=0x14001d47d30 sp=0x14001d47d00 pc=0x104963b54
runtime.mapaccess2(0x1054c4ba0, 0x140022d64b0, 0x1400207ca68, 0x14002aed380, 0x6)
/opt/homebrew/Cellar/go/1.15.7_1/libexec/src/runtime/map.go:469 +0x230 fp=0x14001d47d70 sp=0x14001d47d30 pc=0x10493f2c0
github.com/0xPolygon/polygon-sdk/txpool.(*TxPool).addImpl(0x140003303f0, 0x1050f5eb5, 0x6, 0x1400207c9c0, 0x0, 0x0)
/code/txpool/txpool.go:214 +0x1e4 fp=0x14001d47ea0 sp=0x14001d47d70 pc=0x105048284
github.com/0xPolygon/polygon-sdk/txpool.(*TxPool).handleGossipTxn(0x140003303f0, 0x105535760, 0x14002eb8d50)
/code/txpool/txpool.go:148 +0x180 fp=0x14001d47f10 sp=0x14001d47ea0 pc=0x1050479a0
github.com/0xPolygon/polygon-sdk/txpool.(*TxPool).handleGossipTxn-fm(0x105535760, 0x14002eb8d50)
/code/txpool/txpool.go:138 +0x40 fp=0x14001d47f40 sp=0x14001d47f10 pc=0x10504b190
github.com/0xPolygon/polygon-sdk/network.(*Topic).readLoop(0x140022d65a0, 0x14001cf6100, 0x14000018be0)
/code/network/gossip.go:63 +0x1e4 fp=0x14001d47fc0 sp=0x14001d47f40 pc=0x10503aa14
runtime.goexit()
/opt/homebrew/Cellar/go/1.15.7_1/libexec/src/runtime/asm_arm64.s:1130 +0x4 fp=0x14001d47fc0 sp=0x14001d47fc0 pc=0x104997c34
created by github.com/0xPolygon/polygon-sdk/network.(*Topic).Subscribe
/code/network/gossip.go:39 +0x88

goroutine 1 [chan receive, 8 minutes]:
github.com/0xPolygon/polygon-sdk/command/helper.HandleSignals(0x14000019510, 0x10562e1b8, 0x14000358180, 0x14000350bd0)
/code/command/helper/helper.go:183 +0xbc
github.com/0xPolygon/polygon-sdk/command/server.(*ServerCommand).Run(0x14000128e40, 0x1400011cb80, 0xb, 0xb, 0x140001e74b0)
/code/command/server/server_command.go:201 +0x168
github.com/mitchellh/cli.(*CLI).Run(0x140001697c0, 0x140001697c0, 0x14000000180, 0x300000002)
/code/vendor/github.com/mitchellh/cli/cli.go:255 +0x2c4
main.Run(0x1400011cb70, 0xc, 0xc, 0x14000000180)
/code/main.go:25 +0x88
main.main()
/code/main.go:12 +0x60

goroutine 51 [select, 8 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x14001cbb340)
/code/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:91 +0x74
created by github.com/syndtr/goleveldb/leveldb.openDB
/code/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x30c

goroutine 7 [select, 8 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0x140000dd110)
/code/vendor/github.com/ipfs/go-log/writer/writer.go:71 +0xb4
created by github.com/ipfs/go-log/writer.NewMirrorWriter
/code/vendor/github.com/ipfs/go-log/writer/writer.go:36 +0xac

Tx gasLimit > 100000000 causes gas limit reached error and stuck tx

Description

It seems that setting a tx gasLimit higher than 100000000 results in a pending tx that will never confirm. Any txs after this from the same account will also get stuck with no way of overwriting them.

https://github.com/0xPolygon/polygon-sdk/blob/335e78199b52c768e9b00c3f955e9a42a3ce5ed4/consensus/ibft/ibft.go#L356

Looking at the code above, a block header is initialized with its gasLimit set to 100000000. All tx gasLimits within that same block must not surpass this limit or else this issue occurs.

Aside from these transactions getting 'stuck', I observed many occurrences of the following log messages:

[ERROR] polygon: failed to apply tx: err=\"gas limit reached in the pool\""}

[31mERROR\u001b[0m\tbasichost\tfailed to resolve local interface addresses\t{\"error\": \"route ip+net: netlinkrib: too many open files\"}"}

[ERROR] polygon.network: failed to dial bootnode: err=\"peer rejected; insufficient capacity\""}

Your environment

OS and version Ubuntu 20
version of the Polygon SDK develop
branch that causes this issue 96be904

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Send a tx with gasLimit 5,000,000,000
Observe the gas limit reached in the pool log messages
Observe that subsequent txs from the same address cannot be processed
Attempting to override the nonce of the stuck tx (with a lower gasLimit but higher gasPrice) had no effect

Expected behaviour

I expect there to be some sort of threshold that a tx's gasLimit must be under before it can be accepted. This threshold should be well under the hardcoded header gasLimit (aka Transition.gasPool) of 100000000.

Actual behaviour

A tx with an extremely high gasLimit still gets picked up resulting in indefinite pending txs and error messages.

Memory leak

Description

It seems like there is a memory leak somewhere in the code. Happens after a while of producing blocks leaving us to believe it's something related to block production. CPU usage remains constant.

Your environment

OS and version
AWS linux
version of the Polygon SDK
6906ed9
branch that causes this issue
develop

Steps to reproduce

Tell us how to reproduce this issue

polygon-sdk server run with IBFT consensus of 4 nodes running for about 12 hours.
Where the issue is, if you know
Which commands triggered the issue, if any
polygon-sdk server run..

Expected behaviour

No memory leak

Actual behaviour

Memory leak causing an OOM eerror causing a crash

Logs

9053 2021-07-20T16:42:52.019Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=2210
 9054 2021-07-20T16:42:52.019Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
 9055 2021-07-20T16:42:52.020Z [INFO]  polygon.consensus.ibft: proposer calculated: proposer=0x2d48d27cf3eec87c82f39527934df7bd2124b30a block=2210
 9056 2021-07-20T16:42:52.020Z [DEBUG] polygon.consensus.syncer: enqueue block: peer=16Uiu2HAm1k16mVYWaG1aoG7rZkbuaETqZ6vhV85vB9vtsjXmYbyz number=2209 hash=0x921643448243c439e2db6c1414c7254f2f4e05d78aa7ecb742bec4cd04259a36
 9057 fatal error: runtime: out of memory
 9058
 9059 runtime stack:
 9060 runtime.throw(0x1006f20, 0x16)
 9061         /usr/local/go/src/runtime/panic.go:1117 +0x72
 9062 runtime.sysMap(0xc008000000, 0x4000000, 0x1a90370)
 9063         /usr/local/go/src/runtime/mem_linux.go:169 +0xc6
 9064 runtime.(*mheap).sysAlloc(0x1a76840, 0x400000, 0x42dc97, 0x1a76848)
 9065         /usr/local/go/src/runtime/malloc.go:729 +0x1e5
 9066 runtime.(*mheap).grow(0x1a76840, 0x4, 0x0)
 9067         /usr/local/go/src/runtime/mheap.go:1346 +0x85
 9068 runtime.(*mheap).allocSpan(0x1a76840, 0x4, 0xc007ff0003, 0x7fc0c5fea668)
 9069         /usr/local/go/src/runtime/mheap.go:1173 +0x609
 9070 runtime.(*mheap).allocManual(0x1a76840, 0x4, 0xc007ff6803, 0x4239b2)
 9071         /usr/local/go/src/runtime/mheap.go:943 +0x47
 9072 runtime.getempty.func1()
 9073         /usr/local/go/src/runtime/mgcwork.go:374 +0x45
 9074 runtime.getempty(0xc0055f67c0)
 9075         /usr/local/go/src/runtime/mgcwork.go:373 +0x14e
 9076 runtime.(*stackScanState).addObject(0x7fc0c5feab78, 0xc0055f6758, 0xe1f600)
 9077         /usr/local/go/src/runtime/mgcstack.go:278 +0x113
 9078 runtime.scanframeworker(0x7fc0c5feaa90, 0x7fc0c5feab78, 0xc000045698)
 9079         /usr/local/go/src/runtime/mgcmark.go:945 +0x149
 9080 runtime.scanstack.func1(0x7fc0c5feaa90, 0x0, 0x18ea7c0)
 9081         /usr/local/go/src/runtime/mgcmark.go:746 +0x3d
 9082 runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc0028d4d80, 0x0, 0x0, 0x7fffffff, 0x7fc0c5feacd8, 0x0, 0x0, ...)
 9083         /usr/local/go/src/runtime/traceback.go:322 +0x1135
 9084 runtime.scanstack(0xc0028d4d80, 0xc000045698)
 9085         /usr/local/go/src/runtime/mgcmark.go:749 +0x17c
 9086 runtime.markroot.func1()
 9087         /usr/local/go/src/runtime/mgcmark.go:233 +0xc6
 9088 runtime.markroot(0xc000045698, 0x6fb)
 9089         /usr/local/go/src/runtime/mgcmark.go:206 +0x33e
 9090 runtime.gcDrain(0xc000045698, 0x7)
 9091         /usr/local/go/src/runtime/mgcmark.go:1014 +0x118
 9092 runtime.gcBgMarkWorker.func2()
 9093         /usr/local/go/src/runtime/mgc.go:2003 +0x17e
 9094 runtime.systemstack(0xc004a5d200)
 9095         /usr/local/go/src/runtime/asm_amd64.s:379 +0x66
 9096 runtime.mstart()
 9097         /usr/local/go/src/runtime/proc.go:1246
 9098
 9099 goroutine 6 [GC worker (idle), 2 minutes]:
 9100 runtime.systemstack_switch()
 9101         /usr/local/go/src/runtime/asm_amd64.s:339 fp=0xc000054760 sp=0xc000054758 pc=0x46cd60
 9102 runtime.gcBgMarkWorker()
 9103         /usr/local/go/src/runtime/mgc.go:1967 +0x1c7 fp=0xc0000547e0 sp=0xc000054760 pc=0x41f667
 9104 runtime.goexit()
 9105         /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000547e8 sp=0xc0000547e0 pc=0x46eba1
 9106 created by runtime.gcBgMarkStartWorkers
 9107         /usr/local/go/src/runtime/mgc.go:1835 +0x37
 9108
 9109 goroutine 1 [chan receive, 25 minutes]:
 9110 github.com/0xPolygon/minimal/command/helper.HandleSignals(0xc003179660, 0x1247b98, 0xc00082a8d0, 0xc0007b4bd0)
 9111         /home/ec2-user/polygon-sdk/command/helper/helper.go:181 +0x108
 9112 github.com/0xPolygon/minimal/command/server.(*ServerCommand).Run(0xc00000ecf0, 0xc000032140, 0xf, 0x10, 0xc00061b250)
 9113         /home/ec2-user/polygon-sdk/command/server/server_command.go:201 +0x1ce
 9114 github.com/mitchellh/cli.(*CLI).Run(0xc0000c97c0, 0xc0000c97c0, 0x44299c, 0xc000000300)
 9115         /home/ec2-user/polygon-sdk/vendor/github.com/mitchellh/cli/cli.go:255 +0x41a
 9116 main.Run(0xc000032130, 0x10, 0x11, 0xc00008a058)
 9117         /home/ec2-user/polygon-sdk/main.go:25 +0x96
 9118 main.main()
 9119         /home/ec2-user/polygon-sdk/main.go:12 +0x65
 9120
 9121 goroutine 94 [select, 2 minutes]:
 9122 github.com/libp2p/go-yamux.(*Session).sendLoop(0xc0000ee2a0, 0x0, 0x0)
 9123         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-yamux/session.go:474 +0x1bc
 9124 github.com/libp2p/go-yamux.(*Session).send(0xc0000ee2a0)
 9125         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-yamux/session.go:420 +0x2b
 9126 created by github.com/libp2p/go-yamux.newSession
 9127         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-yamux/session.go:128 +0x3e5
 9128
 9129 goroutine 8 [select, 25 minutes]:
 9130 github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0001250e0)
 9131         /home/ec2-user/polygon-sdk/vendor/github.com/ipfs/go-log/writer/writer.go:71 +0x106
 9132 created by github.com/ipfs/go-log/writer.NewMirrorWriter
 9133         /home/ec2-user/polygon-sdk/vendor/github.com/ipfs/go-log/writer/writer.go:36 +0xb9
 9134
 9135 goroutine 9 [select, 2 minutes]:
 9136 go.opencensus.io/stats/view.(*worker).start(0xc0000f4480)
 9137         /home/ec2-user/polygon-sdk/vendor/go.opencensus.io/stats/view/worker.go:276 +0xcd
 9138 created by go.opencensus.io/stats/view.init.0
 9139         /home/ec2-user/polygon-sdk/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68
 9140
 9141 goroutine 10 [select, 25 minutes]:
 9142 github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc000159600)
 9143         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-peerstore/pstoremem/addr_book.go:93 +0x105
 9144 created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
 9145         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-peerstore/pstoremem/addr_book.go:83 +0x18f
 9146
 9147 goroutine 54 [chan receive, 25 minutes]:
 9148 github.com/0xPolygon/minimal/network.(*Subscription).run(0xc00000fb48)
 9149         /home/ec2-user/polygon-sdk/network/server.go:484 +0x68
 9150 created by github.com/0xPolygon/minimal/network.(*Server).Subscribe
 9151         /home/ec2-user/polygon-sdk/network/server.go:515 +0x145
 9152
 9153 goroutine 12 [select, 2 minutes]:
 9154 github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0xc000132a10, 0x12432d8, 0xc00082eb40)
 9155         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-swarm/swarm_dial.go:126 +0xee
 9156 created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init
 9157         /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-swarm/swarm_dial.go:119 +0x5d

Last 100:

google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00452e480, 0xc0058a13b0, 0x115d858)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_server.go:459 +0x9b
google.golang.org/grpc.(*Server).serveStreams(0xc0007c48c0, 0x124b7b8, 0xc00452e480)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/server.go:862 +0x103
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0007c48c0, 0x124b7b8, 0xc00452e480)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/server.go:821 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/server.go:820 +0x525
goroutine 7988 [chan send, 10 minutes]:
github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout.func1(0x2540be400, 0xc005271c20)
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:848 +0x45
created by github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:846 +0xd6
goroutine 7984 [select, 10 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc005877450, 0x1, 0x0, 0x0, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0043ede00, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:515 +0x1dd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00452e480)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_server.go:291 +0xd7
created by google.golang.org/grpc/internal/transport.newHTTP2Server
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_server.go:288 +0x1110
goroutine 8006 [select, 10 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc005dcfcc0, 0x1, 0x0, 0x0, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004ae9020, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:515 +0x1dd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc004b7da40)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_client.go:391 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_client.go:389 +0x11e7
goroutine 8020 [select, 10 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00452f500)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_server.go:979 +0x265
created by google.golang.org/grpc/internal/transport.newHTTP2Server
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_server.go:299 +0x1135
goroutine 8004 [select, 10 minutes]:
github.com/libp2p/go-ya(0xc0019f38c0, 0xc006850000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-yamux/stream.go:117 +0x197
github.com/libp2p/go-libp2p-yamux.(*stream).Read(0xc0019f38c0, 0xc006850000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-yamux/stream.go:14 +0x4d
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc0057f7e30, 0xc006850000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-swarm/swarm_stream.go:57 +0x62
github.com/multiformats/go-multistream.(*lazyClientConn).Read(0xc0057f7ea0, 0xc006850000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/multiformats/go-multistream/lazyClient.go:68 +0xac
github.com/libp2p/go-libp2p/p2p/host/basic.(*streamWrapper).Read(0xc004ac4540, 0xc006850000, 0x8000, 0x8000, 0x0, 0x800010601, 0x1060100000000)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p/p2p/host/basic/basic_host.go:1008 +0x52
bufio.(*Reader).Read(0xc004ae8e40, 0xc0019c7ed8, 0x9, 0x9, 0x14, 0x7fc0f4bba108, 0x18)
   /usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x122e3c0, 0xc004ae8e40, 0xc0019c7ed8, 0x9, 0x9, 0x9, 0xc006759dd0, 0x45a513, 0x1924c408a7582)
   /usr/local/go/src/io/io.go:328 +0x87
io.ReadFull(...)
   /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader(0xc0019c7ed8, 0x9, 0x9, 0x122e3c0, 0xc004ae8e40, 0x0, 0x0, 0xc035dca7898b8edd, 0xed3cbafc16)
   /home/ec2-user/polygon-sdk/vendor/golang.org/x/net/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0019c7ea0, 0xc003685d40, 0xc003685d40, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc004b7da40)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1330 +0x185
created by google.golang.org/grpc/internal/transport.newHTTP2Client
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/internal/transport/http2_client.go:345 +0x1011
goroutine 8003 [chan receive, 10 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc002a09b80)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/clientconn.go:1213 +0x465
created by google.golang.org/grpc.(*addrConn).connect
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/clientconn.go:843 +0x12a
goroutine 8002 [select, 10 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc004529180)
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xac
created by google.golang.org/grpc.newCCBalancerWrapper
   /home/ec2-user/polygon-sdk/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x172
goroutine 8027 [chan send, 10 minutes]:
github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout.func1(0x2540be400, 0xc00528c7e0)
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:848 +0x45
created by github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:846 +0xd6
goroutine 8025 [chan send, 10 minutes]:
github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout.func1(0x2540be400, 0xc0062bbc20)
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:848 +0x45
created by github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).randomTimeout
   /home/ec2-user/polygon-sdk/consensus/ibft/ibft.go:846 +0xd6
goroutine 8021 [select, 10 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc0019f3970, 0xc006870000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-yamux/stream.go:117 +0x197
github.com/libp2p/go-libp2p-yamux.(*stream).Read(0xc0019f3970, 0xc006870000, 0x8000, 0x8000, 0x2f, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-yamux/stream.go:14 +0x4d
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc0058a42a0, 0xc006870000, 0x8000, 0x8000, 0x2f, 0x0, 0x0)
   /home/ec2-user/polygon-sdk/vendor/github.com/libp2p/go-libp2p-swarm/swarm_stream.go:57 +0x62
   ```

### Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Storing slot not found on sending $ from empty account

Storage slot not found on sending $ from empty account

Description

Currently in a situation where I'm launching with dev mode (no blocks yet created), using the premine account like

./main dev --premine "0x...."

I load up metamask and load up another account (NOT the premine) and try to send a balance around to any account. (You might have to manually adjust gas limit, I made #97 to help) I always seem to get this error as a result from this call

https://github.com/0xPolygon/polygon-sdk/blob/develop/minimal/server.go#L243

It seems this call generates a key (the hash of the account address) that never exists in the storage.

https://github.com/0xPolygon/polygon-sdk/blob/develop/minimal/server.go#L229

Some debug output

Your environment

OS and version Ubuntu 20
version of the Polygon SDK 2f50a19
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue

Set premine to account 1, start up the chain. Load up metamask targeting account 1.
Set premine to account 2, start up the chain. Load up metamask still targeting account 1.
Try to send $ from account 1 to any account

Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Error getting storage snapshot when deploying a smart contract to private blockchain via hardhat

Description

When using hardhat to deploy a copy of the sample Greeter.sol script to a private polygon blockchain, I get the error: error getting storage snapshot in the server log. I am a bit of a noob at this so please correct me if I'm doing something stupid.

Your environment

OS: Manjaro linux 5.10.41
Ver: 0.1.0-dev

Steps to reproduce

To reproduce start a polygon blockchain, exactly copying this guide: https://sdk-docs.polygon.technology/docs/how-tos/howto-set-ibft,
Start a JsonRPC server with go run main.go --chain genesis.json --log-level debug,
Compile the sample script in hardhat with npx hardhat compile in the sample repository,
Your JsonRPC servers ip and port, + an account in the hardhat.config.js config file
Run the provided script to deploy the compiled smart contract with npx hardhat run scripts/sample-script.js --network polygon

Expected behaviour

The simple hello world script should be deployed to the blockchain.

Actual behaviour

In hardhat you get the error HardhatError: HH110: Invalid JSON-RPC response received: {"code":-32603,"message":"internal error"}
In the polygon server you get the error [ERROR] polygon.dispatcher: failed to dispatch: method=eth_estimateGas err="error getting storage snapshot"

Logs

2021-06-10T13:17:22.133+0100 [DEBUG] polygon.dispatcher: request: method=web3_clientVersion id=1
2021-06-10T13:17:22.144+0100 [DEBUG] polygon.dispatcher: request: method=eth_chainId id=2
2021-06-10T13:17:22.154+0100 [DEBUG] polygon.dispatcher: request: method=eth_chainId id=3
2021-06-10T13:17:22.163+0100 [DEBUG] polygon.dispatcher: request: method=eth_estimateGas id=4
2021-06-10T13:17:22.163+0100 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_estimateGas err="error getting storage snapshot"

Proposed solution

No idea how to fix it, but I found that the file giving the error was minimal/server.go

Can't send transactions via metamask in dev mode as `v` in the signature is invalid

Can't send transactions via metamask in dev mode

Description

When running in dev mode and trying to send a transaction, the sdk errors out with "invalid sender" which is actually a result of the error "invalid txn signature". Interestingly, txn.From is always the zero address.

Your environment

OS and version Unbutu 20
version of the Polygon SDK 2f50a19
branch that causes this issue develop

Steps to reproduce

Tell us how to reproduce this issue
Start up dev mode. Example:

dev --log-level DEBUG --premine 0x6F75bA6c90E3da79b7ACAfc0fb9cf3968aa4ee39 --gas-limit 1000000

Where the issue is, if you know

It looks like the signature is being sent as invalid as v because it's greater than 1 here. Is 21 a valid value for v?

https://github.com/0xPolygon/polygon-sdk/blob/develop/crypto/txsigner.go#L183

Which commands triggered the issue, if any

Expected behaviour

It should send the balance fine

Tell us what should happen

Actual behaviour

Metamask errors out

Logs

2021-07-23T16:54:33.469-0400 [ERROR] polygon-dev.dispatcher: failed to dispatch: method=eth_sendRawTransaction err="invalid sender"

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

message ServerStatus { int64 network = 1; string genesis = 2; Block current = 3; string p2pAddr = 4; message Block { int64 number = 1; string hash = 2; } }

[Subject of the issue]

Description

Describe your issue in as much detail as possible here

Your environment

OS and version
version of the Polygon SDK
branch that causes this issue

Steps to reproduce

Tell us how to reproduce this issue
Where the issue is, if you know
Which commands triggered the issue, if any

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Logs

Please paste any logs here that demonstrate the issue, if they exist

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Intrinsic gas too low Error while deploying the contract

intrinsic gas too low Error while deploying the contract

Description

Getting an error using hardhat contract deployment procedure. I followed the steps in Getting Started Guide, and was able to start all nodes without any errors, however, I am not able to deploy a simple contract.

Contract code:

//SPDX-License-Identifier: Unlicense
pragma solidity ^0.8.0;

contract Greeter {
  string greeting;

  constructor(string memory _greeting) {
    greeting = _greeting;
  }

  function greet() public view returns (string memory) {
    return greeting;
  }

  function setGreeting(string memory _greeting) public {
    greeting = _greeting;
  }
}

hardhat.config.js section with network:

module.exports = {
  solidity: "0.8.0",
  defaultNetwork: "polygon",
  networks: {
    polygon: {
      chainId: 100,
      url: 'http://localhost:30002',
      accounts: ['0x${private-key-hardcoded}'],
      gas: "auto",
      gasPrice: "auto",
      timeout: 20000
    }
  }
};

deploy.js script:

async function main() {
  // We get the contract to deploy
  const [deployer] = await ethers.getSigners();

  console.log("Deploying contracts with the account:", deployer.address);

  console.log("Account balance:", (await deployer.getBalance()).toString());

  const Greeter = await ethers.getContractFactory("Greeter");
  const greeter = await Greeter.deploy("Hello, Hardhat!");

  console.log("Greeter deployed to:", greeter.address);
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

Your environment

macOS Catalina
version of the Polygon SDK
current develop branch

Steps to reproduce

Tell us how to reproduce this issue

Follow the steps in Getting Started Guide to configure and run polygon-sdk locally
Configure sample hardhat project (see hardhat.config.js code above)
Create sample contract (see provided source above)
Run npx hardhat run scripts/deploy.js --network polygon

Where the issue is, if you know
Which commands triggered the issue, if any

attempt to deploy the contract

Expected behaviour

Contract deployed and address returned

Actual behaviour

"intrinsic gas too low"

Logs

2021-09-09T16:21:30.670-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_sendRawTransaction err="intrinsic gas too low"
2021-09-09T16:25:22.933-0400 [ERROR] polygon.txpool: Discarding invalid transaction: hash=0x5e171beb08bfe8b6c5d645a5cec28c9f65bd19feda5212e12fd97f167ed04743 err="intrinsic gas too low"
2021-09-09T16:25:22.933-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_sendRawTransaction err="intrinsic gas too low"

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

Fix JsonRPC request conversion

Due to the use of alias methods in the Jsonrpc module, the dispatcher is unable to reflect the Jsonrpc request into Go structs.

Can't verify smart contracts with multiple imports on Polygon scan.

Description

While we were able to successfully deploy all of our solidity smart contracts on to Polygonscan, we ran into some issues verifying each smart contract

We were only able to verify smart contracts that did not have imports. As for the ones that did, the interface made it difficult to import multiple files at once (whenever a new file was added, the preceding one got deleted) making it more difficult to verify the smart contracts. Even after importing some the files, our smart contracts were not verified for this following reason.

We follow the same procedure on Etherscan and it works perfectly.

Steps to reproduce

Simply trying to verify a multi-file solidity smart contract, choose the compile version, and then import the corresponding files for that smart contract, that includes the other smart contracts that the first one is importing and then attempt to verify.

Expected behaviour

Smart-contract should be verified.

Actual behaviour

Smart contract is not verified

Displayed address after 'ibft init' does not match address associated with the validator.key

Description

After running ibft init, one can see the generated Public key and Node ID as seen below:

polygon-sdk ibft init --data-dir validator-1

[IBFT INIT]
Public key (address) = 0xa43a17ccfde80300364118bc709ba492aa7499c1
Node ID              = 16Uiu2HAm2VBsW751CcQx5eJyrSqTT1sByzQeQLoDKx3NzszboDgm

It seems however that this Public key does not match the address derived from validator-1/consensus/validator.key when importing into MetaMask. I was under the impression that these should match...

Your environment

OSX 11.4
Polygon SDK version 0.1.0
Seen on latest on develop (6906ed9)

Steps to reproduce

Run polygon-sdk ibft init --data-dir validator-1 and note the Public key (address) value displayed in the terminal
Import account by specifying private key validator-1/consensus/validator.key in MetaMask
Observe that the Address of the imported account does not resemble what we saw in Step 1.

Expected behaviour

The private key's associated Address should match what was shown under Public key (address) after running the ibft init command.

Actual behaviour

The Public key (address) after running the ibft init is a completely different Address. As a result, one cannot premine that address or send $ to it. Instead, I had to derive the Address first from consensus/validator.key and later use that value when running polygon-sdk genesis --premine 0xa33a17ccfde80300364118bc709ba492aa2499c1.

Delegating to a validator

[How to delegate using SDK]

I'm not able to find any doc related to Staking.

Is there any documentation around that shows how to craft a Delegation tx?
Also any rpc that will return some info about the amount staked etc.?

0xpolygon / polygon-edge Goto Github PK

polygon-edge's Introduction

Update: Edge v1.0.0 is here!

Polygon Edge

Documentation 📝

Disclaimer

polygon-edge's People

Contributors

Stargazers

Watchers

Forkers

polygon-edge's Issues

Panic using --libp2p option

Description

Your environment

Steps to reproduce

Logs

Proposed solution

eth_getTransactionCount is failing on non initialized accounts

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Logs

Proposed solution

Error polygon.consensus.syncer: failed to notify after adding + removing peers/validators

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Contract deploy

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Logs

Proposed solution

eth_call without from field fails during nextNonce calculation in decodeTxn

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Proposed solution

panic on concurrent write to JSON-RPC websocket

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Logs

Proposed solution

Could not parse NAT IP address

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Logs

Proposed solution

[Subject of the issue]

Description

Your environment

Steps to reproduce

[Subject of the issue]

Description

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Logs

Proposed solution

[Subject of the issue]

Description

Your environment

Steps to reproduce

Expected behaviour

Panic using `--libp2p` option

`eth_getTransactionCount` is failing on non initialized accounts

Error `polygon.consensus.syncer: failed to notify` after adding + removing peers/validators

`eth_call` without `from` field fails during nextNonce calculation in decodeTxn

Add `debug_traceTransaction`

Cannot send txs using chainId other than 100 when running `server`