kardiachain / go-kardia Goto Github PK

View Code? Open in Web Editor NEW

33.0 33.0 18.0 38.94 MB

Golang implementation of KardiaChain Decentralize Network

Home Page: https://kardiachain.io

License: GNU Lesser General Public License v3.0

Shell 0.01% Dockerfile 0.01% Go 97.32% Assembly 0.33% Solidity 1.46% Makefile 0.01% C 0.13% JavaScript 0.73%

blockchain kardiachain

go-kardia's People

Contributors

Stargazers

Watchers

Forkers

hadeskers hadesker trungthanhnt 0xtoshi pnkduy trungthaihieu93-dev rstakaiito kevinnguyenai alicandestek thang14 becoswap-dev fweni shin288 rameight kardiachain hovanhoa

go-kardia's Issues

Buggy behaviours of event logs RPC APIs

Current behaviours

blockhash field of event logs is always empty 0x0000000000000000000000000000000000000000.

curl --location --request POST 'https://kai-internal-1.kardiachain.io/' \
--header 'Content-Type: application/json' \
--header 'Cookie: __cfduid=d0de3543a6b9c69cec083b6758e1677f31617347746' \
--data-raw '{
    "jsonrpc": "2.0",
    "method": "kai_getLogs",
    "params": [
        {
            "fromBlock": 1583362,
            "toBlock": 1583475
        }
    ],
    "id": 1
}'

will results like below, the blockHeight is correct but the blockHash fields are always empty.

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "address": "0x4dae614b2ea2faeedde7830a2e7fceddae9f9161",
            "topics": [
                "0xb0d234274aef7a61aa5a2eb44c23881ebf46a068cccbd413c978bcbd555fe17f"
            ],
            "data": "0x0000000000000000000000002dbf5da7f4e3cc72fb3f9cf4413543796c5f3206000000000000000000000000000000000000000000000878678326eac9000000",
            "blockHeight": 1583362,
            "transactionHash": "0x17c8686caea71e8b89b4b443c4b008b29c9e878ac5b48ebec3cd5dfb06d0290d",
            "transactionIndex": 0,
            "blockHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
            "logIndex": 1,
            "removed": false
        },
        {
            "address": "0xf512b1e8acc96baf05b470182f1814bfb14df563",
            "topics": [
                "0x15490ebb4bc37fd11febf8b4a71b7bed47ac2d66d82fbdd5c83da3f202726eed"
            ],
            "data": "0x0000000000000000000000007640b8f0ee079873d8a54cd7e5d95c8dd43073b80000000000000000000000000000000000000000000000000000000000000000",
            "blockHeight": 1583451,
            "transactionHash": "0xba5928d357647e5e983f09a64b89e93827a9d9ebc1d0a6fd89d6193f97b703c6",
            "transactionIndex": 0,
            "blockHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
            "logIndex": 1,
            "removed": false
        },
        {
            "address": "0xeff34b63f55200a9d635b8abbbfcc719b4977864",
            "topics": [
                "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
                "0x0000000000000000000000004102a799b5b87db21f7707e2cc2789330254397f",
                "0x000000000000000000000000bf42a75a362df31b62bb99257f9a66163eb6b962"
            ],
            "data": "0x0000000000000000000000000000000000000000000000000000000001312d01",
            "blockHeight": 1583475,
            "transactionHash": "0x2e37b77519a29769db600fe32dd69f9d476287dea570638d2c1e4d3410a3fd30",
            "transactionIndex": 0,
            "blockHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
            "logIndex": 1,
            "removed": false
        }
    ]
}

kai_getLogs API is crashed with error block info is missing when we try to retrieve logs of old blocks, which didn't include a logsBloom in block info after being applied to state.

curl --location --request POST 'https://kai-internal-1.kardiachain.io/' \
--header 'Content-Type: application/json' \
--header 'Cookie: __cfduid=d0de3543a6b9c69cec083b6758e1677f31617347746' \
--data-raw '{
    "jsonrpc": "2.0",
    "method": "kai_getLogs",
    "params": [
        {
            "fromBlock": 1,
            "toBlock": 2
        }
    ],
    "id": 1
}'

which will response error instead of an empty events list

{
    "jsonrpc": "2.0",
    "id": 1,
    "error": {
        "code": -32000,
        "message": "block info is missing"
    }
}

Expected behaviours

We should include the correct blockHash of the block which these events are emitted in. We also should skip old blocks which don't have a block logsBloom to avoid crashing the whole logs query.

Implement commit signatures

[Propose]Refactor node and chain modules

As we're going to support light node, so I think better if we take a look into our source and how we implement before get started, since our code, imo, quite fragment right now.

Source length

Some source files take too long, for example:

Interfaces

We have some base interface defined in modules:

Move to kai/base package

Split code into small pieces

For example in https://github.com/kardiachain/go-kardiamain/blob/5534fcd2b708ebae183cad06444f4684ce619a11/mainchain/tx_pool/tx_pool.go

We have

which can split into small files.

Reduce duplicate code

Current in our main chain and dual chain both have blockchain and maybe, tx_pool and other modules when we come back to finish dual node, which contain a lot of duplicate code and can't reuse since import cycle. And we might have that issues with light node.

Should we move those modules into same places and extend instead have different modules for each kind of nodes.

Certik review 09/11

Review report

PRE-KardiaChain-09_11_2020.html.zip

Created issues:

[CodeReview] Moves error into dedicated files #76
[CodeReview] Removes code panics #75
[CodeReview] Removes math/rand #74

Tasks

Use of weak random number generator

The code base is using math/rand, the random generation is weak and its not suited for sensitive implementations.
The implementation is part of the tool/ folder and it is part of the testing suite.
This exhibit is more of a warning rather than a critical finding. It serves as a reminder to avoid math.rand

PR #74

Code Panic

The code panics in various places without making sure that running processes are terminated properly.
This exhibit is just a example of the identified issue. The issue exists in various places within the codebase and its amplified by the fact that we are going to be running two main-net chains.

Reply: Code panic at the moments panic while starting node/network, wrong state data or consensus errors
For this milestones, we are sure all the panic code sections well-handed in its case.

Unsafe Pointer Access

Usafe memory access as the code base makes assumption that tx will be never nil.
This exhibit is a example, and while the code base follows a good handling of those issues for the most part of the code, some parts need refactoring for a more secure outcome under any case.
dbd705c

Function Visibility

The function is public and although it does not need to be.
ac32f8a

Redundant constant declaration

We recommend not declaring return variables on the functions signature.

Redundant Code

Redundant variable declaration.
KAR-05: Redundant const declaration > 45f993c
KAR-06-13: 4ed5e4b

Passing lock by value

Function is passing lock by value as BlockChain contains a mutex.
fc477ee

Defer usage

It is recommended to use defer in cases that are non access specific.
KAR-06: Return variable declaration >61c727f

Issues by severity

Critical

ID	Title	Example	Status
KAR-01	Use of weak random number generator	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/tool/tx_generator.go#L284

Major

ID	Title	Example	Status
KAR-02	Code Panic	https://github.com/kardiachain/go-kardiamain/blob/ccf7672814d10ada7158803282dc873bb2b35523/kai/state/statedb.go#L576

Minor

ID	Title	Example	Status
KAR-03	Unsafe Pointer Access	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/mainchain/tx_pool/tx_pool.go#L633

Informational

ID	Title	Example
KAR-04	Function Visibility	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/tool/tx_generator.go#L284
KAR-05	Redundant constant declaration	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/tool/tx_generator.go#L284
KAR-06	Return variable declaration	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/consensus/types/height_vote_set.go#L120
KAR-07	Redundant Code	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/lib/events/events.go#L56
KAR-08	Passing lock by value	https://github.com/kardiachain/go-kardiamain/blob/6b2bec4fa98a80650280aaeb2d913f6c5fef8dad/lib/events/events.go#L56
KAR-09	Defer Usage

Update triedb

Need improvements of get blocks/headers RPC APIs

The format of data field in event logs is not consistent between tx_getTransactionReceipt and other filter APIs. The log data returned by tx_getTransactionReceipt API is prefix with 0x while other filter APIs don't return the logs data with any prefix.
The "latest" header/block/state should not be the CurrentBlock which is processing in consensus manager. For example, the HeaderByNumber is curently implemented as below

func (k *KardiaService) HeaderByNumber(ctx context.Context, number rpc.BlockNumber) *types.Header {
	// Return the latest block if rpc.LatestBlockNumber has been passed in
	if number == rpc.LatestBlockNumber {
		return k.blockchain.CurrentBlock().Header()
	}
	return k.blockchain.GetHeaderByHeight(number.Uint64())
}

The "latest" header should be the header before k.blockchain.CurrentBlock().Header(). That CurrentBlock in KardiaChain network is equal to "pending" block in Ethereum network, which is processing by miners. The same for "latest" block and state.

The block hash field of get blocks/headers by hash RPC APIs should allow "earliest", "pending" and "latest" keyword for convinient.

[CodeReview] Removes math/rand

Function math/rand is not secured. Generator must use crypto.rand that use /urandom.

[CodeReview] Removes code panics

Panics should be completely avoids

Problem with RPC method `kai_getBlockHeaderByNumber`

Method: `kai_getBlockHeaderByNumber`

Params: `[1]`

Current RPC result

On node1: block header of block 1 (correct)
On node2: block header of block 123
On node4: method handler crashed

Expected result

Corect, consistent result between nodes

Clean up unused code, hardcoded genesis address and old/outdated genesis smc

#59 #86 #89

Error log when starting node: Address not found for posHandlerAddress

Splitting from bug #3, when during Genesis Block creation,

INFO [07-02|16:54:46.074]KARDIA Writing custom genesis block 
ERROR[07-02|16:54:46.079]KARDIA StateDB addr not found                   addr=0x0000000000000000000000000000000000000005

Failed address is posHandlerAddress, currently hardcode to be 5 in post_handler.go

This should be one of the fix point to improve PoS in this codebase

Sometime nodes drop peers

When large amount of txs sync over peer channel and/or over exceed msg size cause node sometime drops its peer, need to refactor txs sync and peer handler

Apphash implementation

Memory leak while running

Memory leak while running. On bootstrapping, nodes memory usage is 40MB, but memory usage is increased all the time.

Example, memory usage after 2 days

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
2d0e7b1ee7c4        node1               0.46%               1.866GiB / 3.699GiB   50.43%              9.34GB / 8.39GB     6.57MB / 9.33GB     12

For profiling of go process, we can start node with cmd pump in tool/pump/main.go, with parameters in *.yaml file

Debug:
  Port: ":6000"

Example config file: (change *.txt to *.yaml)

pump.yaml.txt

Then run cmd to get pdf file:

go tool pprof -pdf http://localhost:6000/debug/pprof/heap

Example:
profile001.pdf

"header for hash not found" error encountered if toBlock = "latest" when create filter criteria

Supposed that we created a logs filter with following parameters:

{
    "jsonrpc": "2.0",
    "method": "kai_newFilter",
    "params": [
        {
            "toBlock": "latest",
            "topics": [
                "0xb0d234274aef7a61aa5a2eb44c23881ebf46a068cccbd413c978bcbd555fe17f"
            ]
        }
    ],
    "id": 1
}

After that, when we call kai_getFilterLogs API to retrieve logs, API responses always failed with error header for hash not found.

[Feature Request] Blockchain reactor for fast sync purpose

Blockchain reactor implementation details

The reactor will include a demultiplexing routine which will send each message to each sub routine for independent processing. The fast sync protocol logic is decoupled from IO by using three concurrent threads of execution: a scheduler, a processor, and a demuxer. The demuxRoutine acts as "pacemaker" setting the time in which events are expected to be handled. It is responsible for translating between internal events and network IO messages, and for routing events between components. Both the scheduler and processor are structured as finite state machines with input and output events. Input events are received on an unbounded priority queue, with higher priority for error events. Output events are emitted on a blocking, bounded channel. Network IO is handled by the KardiaChain p2p subsystem, where messages are sent in a non-blocking manner.

// Takes the channel as a parameter to avoid race conditions on r.events.
func (r *BlockchainReactor) demux(events <-chan Event) {
	var (
		scheduleFreq = 20 * time.Millisecond
		doScheduleCh = make(chan struct{}, 1)
		doScheduleTk = time.NewTicker(scheduleFreq)
	)
	defer doScheduleTk.Stop()
        ...
	for {
		select {
			case <-doScheduleTk.C:
			        select {
			        case doScheduleCh <- struct{}{}:
			        default:
			        }
                        case <-doScheduleCh:
			        r.scheduler.send(rTrySchedule{time: time.Now()})

                        // Events from peers. Closing the channel signals event loop termination.
		        case event, ok := <-events:
                        ...

			// Incremental events from scheduler
			case event := <-r.scheduler.next():
			...

			// Incremental events from processor
			case event := <-r.processor.next():
			...

			// Terminal event from processor
			case err := <-r.processor.final():
			...
		}
	}
}

The IO component is responsible for exchanging (sending and receiving) fast sync protocol messages with peers. There is one send and one receive routine per peer.

Life cycle management

A set of routines for individual processes allow processes to run in parallel with clear life cycle management. Start, Stop, and AddPeer hooks currently present in the reactor will delegate to the sub-routines allowing them to manage internal state independent without further coupling to the reactor.

func (r *BlockChainReactor) Start() {
	r.events = make(chan Event, chBufferSize)
	go r.scheduler.start()
	go r.processor.start()
        go r.demux(r.events)
	...
}

func (bcR *BlockchainReactor) Receive(...) {
	...
	r.msgs <- msg
	...
}

func (r *BlockchainReactor) Stop() {
	...
	r.msgs <- stop
	...
}

...

func (r *BlockchainReactor) AddPeer(peer p2p.Peer) {
	...
	r.msgs <- bcAddPeerEv{peer.ID}
	...
}

IO handling

An IO handling routine within the reactor will isolate peer communication. Message going through the ioRoutine will usually be one way, using p2p APIs. In the case in which the p2p API such as trySend return errors, the ioRoutine can funnel those message back to the demuxRoutine for distribution to the other routines. For instance errors from the ioRoutine can be consumed by the scheduler to inform better peer selection implementations.

func (sio *switchIO) sendBlockRequest(peerID p2p.ID, height uint64) error {
	...
	msgBytes, err := EncodeMsg(&bcproto.BlockRequest{Height: height})
	if err != nil {
		return err
	}
	queued := peer.TrySend(BlockchainChannel, msgBytes)
	if !queued {
		return fmt.Errorf("send queue full")
	}
	return nil
}

func (sio *switchIO) sendStatusResponse(base uint64, height uint64, peerID p2p.ID) error {
        ...
	msgBytes, err := EncodeMsg(&bcproto.StatusResponse{Height: height, Base: base})
	if err != nil {
		return err
	}
	if queued := peer.TrySend(BlockchainChannel, msgBytes); !queued {
		return fmt.Errorf("peer queue full")
	}
	return nil
}

func (sio *switchIO) sendBlockToPeer(block *types.Block, peerID p2p.ID) error {
	...
	msgBytes, err := EncodeMsg(&bcproto.BlockResponse{Block: bpb})
	if err != nil {
		return err
	}
	if queued := peer.TrySend(BlockchainChannel, msgBytes); !queued {
		return fmt.Errorf("peer queue full")
	}
	return nil
}

func (sio *switchIO) sendBlockNotFound(height uint64, peerID p2p.ID) error {
	peer := sio.sw.Peers().Get(peerID)
	if peer == nil {
		return fmt.Errorf("peer not found")
	}
	msgBytes, err := EncodeMsg(&bcproto.NoBlockResponse{Height: height})
	if err != nil {
		return err
	}

	if queued := peer.TrySend(BlockchainChannel, msgBytes); !queued {
		return fmt.Errorf("peer queue full")
	}

	return nil
}

func (sio *switchIO) trySwitchToConsensus(state cstate.LastestBlockState, skipWAL bool) bool {
	conR, ok := sio.sw.Reactor("CONSENSUS").(consensusReactor)
	if ok {
		conR.SwitchToConsensus(state, skipWAL)
	}
	return ok
}

func (sio *switchIO) broadcastStatusRequest() error {
	...
	sio.sw.Broadcast(BlockchainChannel, msgBytes)
	return nil
}

Processor

The processor is responsible for ordering, verifying and executing blocks. The Processor will maintain an internal map queue referring to the blocks waiting to be processed. As a set of blocks arrive unordered, the Processor will check if it has height+1 necessary to process the next block. The processor also maintains the interface processorContext in order to verify, apply and save new blocks.

type pcState struct {
	// blocks waiting to be processed
	queue blockQueue

	// draining indicates that the next rProcessBlock event with a queue miss constitutes completion
	draining bool

	// the number of blocks successfully synced by the processor
	blocksSynced int

	// the processorContext which contains the processor dependencies
	context processorContext
}

type processorContext interface {
	applyBlock(blockID types.BlockID, block *types.Block) error
	verifyCommit(chainID string, blockID types.BlockID, height uint64, commit *types.Commit) error
	saveBlock(block *types.Block, blockParts *types.PartSet, seenCommit *types.Commit)
	kaiState() cstate.LastestBlockState
	setState(cstate.LastestBlockState)
}

In Kardia blockchain, the commit for block (signed votes messages) h is contained in block h+1, and thus a node performing fast sync must receive two sequential blocks before it can verify fully the first one. If verification succeeds, the first block is accepted; if it fails, both blocks are rejected, since it is not known which block was faulty. When the node rejects a block, it
suspects the sending peer of being faulty and evicts this peer from the set of peers. The same happens when a peer does not reply within a predefined time interval.

// nextTwo returns the next two unverified blocks
func (state *pcState) nextTwo() (queueItem, queueItem, error) {
	if first, ok := state.queue[state.height()+1]; ok {
		if second, ok := state.queue[state.height()+2]; ok {
			return first, second, nil
		}
	}
	return queueItem{}, queueItem{}, fmt.Errorf("not found")
}

func (state *pcState) handle(event Event) (Event, error) {
        switch event := event.(type) {
        case rProcessBlock:
		// verify if +second+ last commit "confirms" +first+ block
		err = state.context.verifyCommit(kaiState.ChainID, firstID, first.Height(), second.LastCommit())
		if err != nil {
			state.purgePeer(firstItem.peerID)
			if firstItem.peerID != secondItem.peerID {
				state.purgePeer(secondItem.peerID)
			}
			return pcBlockVerificationFailure{
					height: first.Height(), firstPeerID: firstItem.peerID, secondPeerID: secondItem.peerID},
				nil
		}
...

Furthermore, it informs the scheduler whether a block processing was successful (pcBlockProcessed) or it has led to an error (pcBlockVerificationFailure).

Scheduler

The scheduler contains the business logic for tracking peers and determining which block to request from whom. The scheduler needs to maintain state on:

The state blockState of every block seem up to height of maxHeight
The set of peers and their peer state peerState
which peers have which blocks
which blocks have been requested from which peers

type blockState int

const (
	blockStateUnknown   blockState = iota + 1 // no known peer has this block
	blockStateNew                             // indicates that a peer has reported having this block
	blockStatePending                         // indicates that this block has been requested from a peer
	blockStateReceived                        // indicates that this block has been received by a peer
	blockStateProcessed                       // indicates that this block has been applied
)

type scheduler struct {
	initHeight uint64

	// next block that needs to be processed. All blocks with smaller height are
	// in Processed state.
	height uint64

	// lastAdvance tracks the last time a block execution happened.
	// syncTimeout is the maximum time the scheduler waits to advance in the fast sync process before finishing.
	// This covers the cases where there are no peers or all peers have a lower height.
	lastAdvance time.Time
	syncTimeout time.Duration

	// a map of peerID to scheduler specific peer struct `scPeer` used to keep
	// track of peer specific state
	peers       map[p2p.ID]*scPeer
	peerTimeout time.Duration   // maximum response time from a peer otherwise prune
	minRecvRate int64             // minimum receive rate from peer otherwise prune

	// the maximum number of blocks that should be New, Received or Pending at any point
	// in time. This is used to enforce a limit on the blockStates map.
	targetPending int
	// a list of blocks to be scheduled (New), Pending or Received. Its length should be
	// smaller than targetPending.
	blockStates map[uint64]blockState

	// a map of heights to the peer we are waiting a response from
	pendingBlocks map[uint64]p2p.ID

	// the time at which a block was put in blockStatePending
	pendingTime map[uint64]time.Time

	// a map of heights to the peers that put the block in blockStateReceived
	receivedBlocks map[uint64]p2p.ID
}

The scheduler receives relevant protocol messages from peers (for example bcBlockResponse and bcStatusResponse), but also internal events that are the result of the block processing in the processor (the events carry the information of whether
a block was successfully processed or there was an error). The scheduler schedules block requests by emitting internal events (scBlockRequest) and also informs the processor about internal processing, for example, when block response is received (scBlockReceived) or if there is an error in peer behaviour (scPeerError). It is configured to maintain a target n of in flight messages and will use feedback from bcBlockResponse, bcStatusResponse and scPeerError to produce an optimal assignment of rTrySchedule at each doScheduleCh ticker.

func (sc *scheduler) handle(event Event) (Event, error) {
	switch event := event.(type) {
	case bcResetState:
		nextEvent, err := sc.handleResetState(event)
		return nextEvent, err
	case bcStatusResponse:
		nextEvent, err := sc.handleStatusResponse(event)
		return nextEvent, err
	case bcBlockResponse:
		nextEvent, err := sc.handleBlockResponse(event)
		return nextEvent, err
	case bcNoBlockResponse:
		nextEvent, err := sc.handleNoBlockResponse(event)
		return nextEvent, err
	case rTrySchedule:
		nextEvent, err := sc.handleTrySchedule(event)
		return nextEvent, err
	case bcAddNewPeer:
		nextEvent, err := sc.handleAddNewPeer(event)
		return nextEvent, err
	case bcRemovePeer:
		nextEvent, err := sc.handleRemovePeer(event)
		return nextEvent, err
	case rTryPrunePeer:
		nextEvent, err := sc.handleTryPrunePeer(event)
		return nextEvent, err
	case pcBlockProcessed:
		nextEvent, err := sc.handleBlockProcessed(event)
		return nextEvent, err
	case pcBlockVerificationFailure:
		nextEvent, err := sc.handleBlockProcessError(event)
		return nextEvent, err
	default:
		return scSchedulerFail{reason: fmt.Errorf("unknown event %v", event)}, nil
	}
}
...

// lowest block in sc.blockStates with state == blockStateNew or -1 if no new blocks
func (sc *scheduler) nextHeightToSchedule() uint64 {
	var min uint64 = math.MaxUint64
	for height, state := range sc.blockStates {
		if state == blockStateNew && height < min {
			min = height
		}
	}
	if min == math.MaxUint64 {
		min = 0
	}
	return min
}

func (sc *scheduler) handleTrySchedule(event rTrySchedule) (Event, error) {
	if time.Since(sc.lastAdvance) > sc.syncTimeout {
		return scFinishedEv{reason: "timeout, no advance"}, nil
	}

	nextHeight := sc.nextHeightToSchedule()
	if nextHeight == 0 {
		return noOp, nil
	}

	bestPeerID, err := sc.selectPeer(nextHeight)
	if err != nil {
		return scSchedulerFail{reason: err}, nil
	}
	if err := sc.markPending(bestPeerID, nextHeight, event.time); err != nil {
		return scSchedulerFail{reason: err}, nil // XXX: peerError might be more appropriate
	}
	return scBlockRequest{peerID: bestPeerID, height: nextHeight}, nil
}
...

type scPeer struct {
	peerID p2p.ID

	// initialized as New when peer is added, updated to Ready when statusUpdate is received,
	// updated to Removed when peer is removed
	state peerState

	base        uint64 // updated when statusResponse is received
	height      uint64 // updated when statusResponse is received
	lastTouched time.Time
	lastRate    int64 // last receive rate in bytes
}

Once the Fastsync protocol terminates, this is signaled to the KardiaChain consensus component (denoted ConsensusManager) with a trySwitchToConsensus event.

Prerequisites for client to integrate

ValidatorSet at block height API
Commit at block height API
GetProof at block height API
PendingNonceAt

How to map validator node with wallet?

Hi, I try to become a validator which is running full node. It's seem there's a way to register to become a candidate, do I need to running my own node when I was elected enough 12.5M KAI? Or it's just my identity on-chain without operating the node?

Subscribe filter logs return data but cannot get transaction with hash

Subscribe LogFilter with ws

{
  "level": "debug",
  "ts": 1619000339.835479,
  "caller": "handler/pair.go:39",
  "msg": "PairEvent",
  "method": "OnPairEvents",
  "Log": {
    "address": "0xA37888A6FF1B0A5347D8355DE05A044F2981a959",
    "methodName": "Swap",
    "argumentsName": "index_topic_1 address sender, uint256 amount0In, uint256 amount1In, uint256 amount0Out, uint256 amount1Out, index_topic_2 address to",
    "arguments": {
      "amount0In": "10000000000000000000",
      "amount0Out": "0",
      "amount1In": "0",
      "amount1Out": "2388318147512360489",
      "sender": "0x581e5fdb8114023faef21666910b3b9b4e612ddc",
      "to": "0xf64c35a3d5340b8493ce4cd988b3c1e890b2bd68"
    },
    "topics": [
      "0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822",
      "0x000000000000000000000000581e5fdb8114023faef21666910b3b9b4e612ddc",
      "0x000000000000000000000000f64c35a3d5340b8493ce4cd988b3c1e890b2bd68"
    ],
    "data": "0000000000000000000000000000000000000000000000008ac7230489e8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000212502b38917de29",
    "blockHeight": 459708,
    "time": "0001-01-01T00:00:00Z",
    "transactionHash": "0xbf4b6118e2ecd922efddbc049693fd64b7ad43bc3dc49f6d35fd0718de5b3bce",
    "blockHash": "0x97d584e4a97560d3001f5e86bc5194abe579fc08dcb91e919cf488a2fd58abe3",
    "logIndex": 5
  }
}

Sometimes, cannot get txHash which got from event logs

{
  "level": "error",
  "ts": 1619000339.9845388,
  "caller": "handler/pair.go:69",
  "msg": "cannot get tx",
  "method": "OnSwapEvent",
  "error": "tx for hash not found",
  "stacktrace": ""
}

Websocket client crash frequency

There are reports from clients that websocket subscriptions crash frequency without errors
Need to optimize RPC server and requests handler to reduce the workloads of spamming HTTP RPC API

Refactor node_config.yaml to genesis config

Add new config .yaml contains all genesis config

Define list config should be move to new config
Move those config to genesis.yaml
Write unit test

[Certik Report] 2020-11-19 (P2)

Continue update and fix from Certik's report

Certik-20201119.zip

K02: Code panic at this version only panicking while starting node/network, wrong state or consensus errors, we are sure all the panic code sections are well-handed in its case.
K17: fd986c1
K18: b0157ce
K19: 2c9ee05
K20: the preCheck() has been follow up with buyGas(), which also has check for Clause 3 at https://github.com/kardiachain/go-kardiamain/blob/master/mainchain/blockchain/state_processor.go#L198
K21:508cf8e
K22: 6438692
K23: https://github.com/kardiachain/go-kardiamain/blob/master/mainchain/tx_pool/tx_pool_helper.go#L290
K24: #102
K25: 8284ddc Use when create genesis state, so its should panic if vs nil before make this call
K26: ed44c91
K27: fbc446f
K28: 6c14e74

Fixes unit test for dual node eth client

Unit test eth_client is failing after upgrading to new geth library.

go test ./dualnode/eth/eth_client/...

--- FAIL: TestGetMethodAndParams (0.00s)
panic: reflect.StructOf: field 0 has no type [recovered]
	panic: reflect.StructOf: field 0 has no type

goroutine 32 [running]:
testing.tRunner.func1.1(0xe52bc0, 0xc000500f40)
	/snap/go/5830/src/testing/testing.go:940 +0x2f5
testing.tRunner.func1(0xc0004998c0)
	/snap/go/5830/src/testing/testing.go:943 +0x3f9
panic(0xe52bc0, 0xc000500f40)
	/snap/go/5830/src/runtime/panic.go:969 +0x166
reflect.StructOf(0xc0004c9ad0, 0x2, 0x2, 0x0, 0x0)
	/snap/go/5830/src/reflect/type.go:2383 +0x32fa
github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client.makeStruct(0xc000522240, 0x2, 0x4, 0x101, 0xc0004a91e0)
	/home/thien/go/src/github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client/client.go:725 +0x17f
github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client.GenerateInputStruct(0x0, 0x0, 0x0, 0x0, 0x0, 0xc0005207f0, 0xa, 0x0, 0xc000522b40, 0x1, ...)
	/home/thien/go/src/github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client/client.go:708 +0x1b4
github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client.GetMethodAndParams(0x0, 0x0, 0x0, 0x0, 0x0, 0xc0005207f0, 0xa, 0x0, 0xc000522b40, 0x1, ...)
	/home/thien/go/src/github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client/client.go:667 +0xb8
github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client.TestGetMethodAndParams(0xc0004998c0)
	/home/thien/go/src/github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client/client_test.go:54 +0x216
testing.tRunner(0xc0004998c0, 0x1198280)
	/snap/go/5830/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
	/snap/go/5830/src/testing/testing.go:1042 +0x357
FAIL	github.com/kardiachain/go-kardiamain/dualnode/eth/eth_client	0.043s
FAIL

Investigation should start here first:
https://github.com/kardiachain/go-kardiamain/blob/master/dualnode/eth/eth_client/client.go#L721
This was modified after upgrading geth library, not confirm to work 100% commit link

fix mongodb node to write custom db struct and catchup network

P2P emergency update for supporting channel

Evidence pool implementation

Need a abigen tool to bind KardiaChain smart contracts to a golang package

This PR is an initial stab at a web3 equivalent Go binding generator. It creates a abigen command that takes a contract ABI and input and generates a Go package/file as output containing all type safe wrapper code to interact with an instance of the particular contract on the blockchain.

Some of the features it includes:

Wrap all contract methods in type safe Go API functions
Have separate types (also) for read-only (free) and write-only (paid) calls/transactions
Have the possibility to fins tune all call/transaction parameters (e.g. gas, price, nonce)
Implement session types that have the above parameters pre-configured (i.e. simpler API)
By providing the bytecode, contract deployment methods are also generated
By providing the solidity sources, everything is automatically generated in one go

This PR also expands the generated code with new types and methods to support filtering past contract events, and subscribing to a stream of future events!

For every event present in the contract, abigen will create a Go counterpart. E.g. for the Transfer event in the token contract:

event Transfer(address indexed from, address indexed to, uint256 value);

// Krc721Transfer represents a Transfer event raised by the Krc721 contract.
type Krc721Transfer struct {
	From    common.Address
	To      common.Address
	TokenId *big.Int
	Raw     types.Log // Blockchain specific contextual infos
}

For each event, abigen will generate a Filter<event name> method to retrieve past logs. As you can see below, the topics being filtered for are strongly typed Go types, not topic hashes as all other APIs surface. This allows calling code to be meaningfully read and understood without having to guess what some cryptic numbers mean. abigen will generate all the necessary code to convert the user types into topic filter criteria under the hood.

The log filterer method returns an iterator that can be used to iterate over the found logs, each already unpacked from its raw Kardia RLP encoded form (and topic form) into strongly typed Go structs like Krc721Transfer listed above.

// FilterTransfer is a free log retrieval operation binding the contract event 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef.
//
// Solidity: event Transfer(address indexed from, address indexed to, uint256 indexed tokenId)
func (_Krc721 *Krc721Filterer) FilterTransfer(opts *bind.FilterOpts, from []common.Address, to []common.Address, tokenId []*big.Int) (*Krc721TransferIterator, error) {

Similarly, for each event, abigen will generate a Watch<event name> method to subscribe to future logs and similarly to filtering past events, subscribing to future ones can be done via strongly typed criteria.

The method also takes a sink channel as an argument to deliver newly found contract events on, and returns a subscription which can be used to tear down the watcher constructs. As with past event subscriptions, the events streamed in the user-provided channel are strongly typed like Krc721Transfer above.

// WatchTransfer is a free log subscription operation binding the contract event 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef.
//
// Solidity: event Transfer(address indexed from, address indexed to, uint256 indexed tokenId)
func (_Krc721 *Krc721Filterer) WatchTransfer(opts *bind.WatchOpts, sink chan<- *Krc721Transfer, from []common.Address, to []common.Address, tokenId []*big.Int) (event.Subscription, error) {

Caveats

Filtering for past log events returns an iterator for future API stability. The current implementation under the hood runs the entire user filtering in one big go, caches the entire results and then uses the iterator to simulate streaming filtering. This will be replaced eventually, but to avoid breaking the API then, we're enforcing this future mode of operation in user code already now.

Watching for future log events has an optional parameter for specifying the starting block number. abigen will naively be able to use it without API changes.

Evidence pool unit test unstable

Evidence pool test sometimes fails:

thien@ubuntu:~/go/src/github.com/kardiachain/go-kardiamain/types/evidence$ go test ./... -count=1 
ok  	github.com/kardiachain/go-kardiamain/types/evidence	0.920s
thien@ubuntu:~/go/src/github.com/kardiachain/go-kardiamain/types/evidence$ go test ./... -count=1 
ok  	github.com/kardiachain/go-kardiamain/types/evidence	0.902s
thien@ubuntu:~/go/src/github.com/kardiachain/go-kardiamain/types/evidence$ go test ./... -count=1 
--- FAIL: TestEvidencePool (0.47s)
    pool_test.go:100: Fail to add goodEvidence: evidence is not valid: evidence from height 100002 (created at: 1600917576) is too old; min height is 90001 and evidence can not be older than 1601090375 
FAIL
FAIL	github.com/kardiachain/go-kardiamain/types/evidence	0.948s
FAIL

Are there any public nodes for rpc testing?

Implement batch transaction sync

Currently nodes share transaction one by one.

We should implement transaction sync routine that send transactions in small batches.
This routine should handle special case for init sync when node is too far away and potentially send a too big batch.
Similar work: ProtocolManger.syncLoop64

In the future, we can consider high-bandwich broadcast style for propagate transactions, some interesting discussion here EIP-2464

Network stop commit new block

INFO [11-03|13:48:57.869] Ensure peers                             numOutPeers=1 numInPeers=0 numDialing=0 numToDial=9
INFO [11-03|13:48:57.870] We need more addresses. Sending pexRequest to random peer peer="Peer{MConn{127.0.0.1:3001} 7cefc13b6e2aedeedfb7cb6c32457240746baee5 out}"
INFO [11-03|13:48:57.870] No addresses to dial. Falling back to seeds
INFO [11-03|13:48:59.003]KARDIA Timed out                                dur=57s   height=4 round=108 step=RoundStepPropose
INFO [11-03|13:48:59.003] enterPrevote(4/108). Current: 4/108/RoundStepPropose
INFO [11-03|13:48:59.003] enterPrevote: ProposalBlock is nil       height=4 round=108
INFO [11-03|13:48:59.005] Signed and pushed vote                   height=4 round=108 vote="Vote{2:FF3DAC4F04DD 4/108/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000:000000000000 , 5396CCEB2143 @2020-11-03 06:48:59.003793149 +0000 UTC}" err=nil
INFO [11-03|13:48:59.006] Added to prevote                         vote="Vote{2:FF3DAC4F04DD 4/108/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000:000000000000 , 5396CCEB2143 @2020-11-03 06:48:59.003793149 +0000 UTC}" prevotes="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PREVOTE +2/3:<nil>(0.2777777777777778) BA{9:__x______} map[]}"
INFO [11-03|13:48:59.008] Added to prevote                         vote="Vote{0:7CEFC13B6E2A 4/108/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000:000000000000 , 12E082CAEFF0 @2020-11-03 06:48:58.903039454 +0000 UTC}" prevotes="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PREVOTE +2/3:<nil>(0.5555555555555556) BA{9:x_x______} map[]}"
INFO [11-03|13:48:59.209] Added to prevote                         vote="Vote{1:C1FE56E3F58D 4/108/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000:000000000000 , 037344486529 @2020-11-03 06:48:59.00387503 +0000 UTC}" prevotes="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PREVOTE +2/3:000000000000:000000000000(0.8333333333333334) BA{9:xxx______} map[]}"
INFO [11-03|13:48:59.209] enterPrecommit(4/108). Current: 4/108/RoundStepPrevote height=4 round=108
INFO [11-03|13:48:59.209] enterPrecommit: +2/3 prevoted for nil.   height=4 round=108
INFO [11-03|13:48:59.212] Signed and pushed vote                   height=4 round=108 vote="Vote{2:FF3DAC4F04DD 4/108/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000:000000000000 , 1CE1B42CD04A @2020-11-03 06:48:59.20981159 +0000 UTC}" err=nil
INFO [11-03|13:48:59.213] Added to precommit                       vote="Vote{2:FF3DAC4F04DD 4/108/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000:000000000000 , 1CE1B42CD04A @2020-11-03 06:48:59.20981159 +0000 UTC}" precommits="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PRECOMMIT +2/3:<nil>(0.2777777777777778) BA{9:__x______} map[]}"
INFO [11-03|13:48:59.213] Added to precommit                       vote="Vote{0:7CEFC13B6E2A 4/108/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000:000000000000 , FF585ABE523D @2020-11-03 06:48:59.109053918 +0000 UTC}" precommits="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PRECOMMIT +2/3:<nil>(0.5555555555555556) BA{9:x_x______} map[]}"
INFO [11-03|13:48:59.413] Added to precommit                       vote="Vote{1:C1FE56E3F58D 4/108/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000:000000000000 , 272546F25E90 @2020-11-03 06:48:59.209735391 +0000 UTC}" precommits="VoteSet{H:4 R:108 T:SIGNED_MSG_TYPE_PRECOMMIT +2/3:000000000000:000000000000(0.8333333333333334) BA{9:xxx______} map[]}"
INFO [11-03|13:48:59.413] enterPrecommitWait(4/108). Current: 4/108/RoundStepPrecommit height=4 round=108

Mongo node could not started

Preproduce:

Run 3 LevelDB Node:

./cmd --config <path>/ kai_eth_config_1.yaml
./cmd --config <path>/ kai_eth_config_2.yaml
./cmd --config <path>/ kai_eth_config_3.yaml

Then run 1 MongoNode

./cmd --config <path>/ mongo.yaml

With mongo.yaml configuration:

    Type: 1                                  # 0 is leveldb, 1 is mongodb
    Dir: chaindata                           # directory stores leveldb
    Cache: 16                                # cache is used in leveldb
    Handles: 32                              # handles is used in leveldb
    URI: mongodb://127.0.0.1:27017           # URI is used in mongodb
    Name: kardiachain                        # Name is use in mongodb
    Drop: 1

Observed Result:

3 LevelDB node run well. But 1 Mongo Node has error exception(panic) :

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x428a6be]

goroutine 1 [running]:
github.com/kardiachain/go-kardiamain/trie.(*TrieDatabase).Commit(0xc00027e000, 0x544843ad0270935f, 0x67304d28e680ef8b, 0x36c3633e13a146db, 0x891705ff89d61e34, 0x891705ff89d61e01, 0x0, 0x0)
        /Users/phucnguyen/blockchain/go-kardiamain/trie/trie_database.go:614 +0x25e
github.com/kardiachain/go-kardiamain/mainchain/genesis.(*Genesis).ToBlock(0xc0000f6a00, 0x4ff1760, 0xc000033410, 0x4ff21e0, 0xc0002f22e0, 0x47)
        /Users/phucnguyen/blockchain/go-kardiamain/mainchain/genesis/genesis.go:198 +0x532
github.com/kardiachain/go-kardiamain/mainchain/genesis.(*Genesis).Commit(0xc0000f6a00, 0x4ff1760, 0xc000033410, 0x5004e40, 0xc0002f22e0, 0x0, 0x41a91c5, 0xc0000d2028)
        /Users/phucnguyen/blockchain/go-kardiamain/mainchain/genesis/genesis.go:208 +0x88
github.com/kardiachain/go-kardiamain/mainchain/genesis.SetupGenesisBlock(0x4ff1760, 0xc000033410, 0x5004e40, 0xc0002f22e0, 0xc0000f6a00, 0xc00080c580, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/phucnguyen/blockchain/go-kardiamain/mainchain/genesis/genesis.go:104 +0x1b3
github.com/kardiachain/go-kardiamain/mainchain.newKardiaService(0xc000172110, 0xc0000fc780, 0x11, 0x400eb36, 0xc0000333e0)
        /Users/phucnguyen/blockchain/go-kardiamain/mainchain/kardia_service.go:92 +0x20f
github.com/kardiachain/go-kardiamain/mainchain.NewKardiaService(0xc000172110, 0xc0000333b0, 0xc00014fdd0, 0x2, 0x2)
        /Users/phucnguyen/blockchain/go-kardiamain/mainchain/kardia_service.go:191 +0x174
github.com/kardiachain/go-kardiamain/node.(*Node).Start(0xc0002721c0, 0x0, 0x0)
        /Users/phucnguyen/blockchain/go-kardiamain/node/node.go:103 +0x3bf
main.(*Config).Start(0xc000227110)
        /Users/phucnguyen/blockchain/go-kardiamain/cmd/main.go:407 +0x446
main.main()
        /Users/phucnguyen/blockchain/go-kardiamain/cmd/main.go:605 +0xc0

Process finished with exit code 2

[Feature Request] Please provide a way to backup the Mnemonic with the web wallet

Hi!

Please provide a way to backup the Mnemonic with the web wallet.
This is usually a feature of other web wallets but can't be found in yours.
If you have created the wallet with a keystore file you can never switch to the e.g. mobile wallet because you don't have the mnemonic.
Thanks!

Cheers,
BF

[CodeReview] Moves error into dedicated files

Implement gas station oracle

We need a RPC API to suggest gas price based on some most recently transactions, should be kai_gasPrice.

Block height in WAL should be uint64 instead of int64

Intermediate state after applying transaction (root field of receipt) is always empty

Implement BlockPart sync in consensus

We should break Block into parts to sync in consensus instead of sending entire Block.

Upgrade Go lang version to 13+ and use go modules

Currently we use go v12 and use go dep for dependency management.

Go language now officially use Go Modules for dependency management in v13+, we already use Modules in go-kardiachain.

We should migrate go dep to Modules, there is official migraton tool here:
https://blog.golang.org/migrating-to-go-modules

Remove pump tool from repo

Since we are now using external TXs generator and send it over RPC to simulate 'real-life-production' of the network. Consider remove internal Txs pumping tool and all hardcoded address from this repo.

[URGENT] Need a way to get revert reason in case transaction is reverted

[Propose] Emergency support Metamask

Add API Interface to emergency support Metamask #124

Should limit the gas limit of a transaction

Gas limit of this transaction is more than 199 mil

Suggestion: even distribution of kai staked rewards to each validator node

Has the team thought about distributing equal staking rewards to each validator node to incentivize kardians to stake evenly between the validators to maximize their apr%? This would allow the system to be more decentralized as kai would be more evenly distributed between the validators and also incentivize new validators to get on board.

Lightnode/lightclient implementation

Check if an address is KRC20 token

As explorer need to detect if a new created contract is krc20/krc721 token.

Current explorer using a dummy way to detect as call KRC20 interface such as name, symbol, decimals, totalSuppy and ownerBalance, and assume the contract is KRC20.

Could you please provide a better and accurate way to check if an address is KRC20 token?

Completely using type uint instead of big.Int for better performance

Hardcoded Private Key

I was wondering why the private key is hardcoded in the main config
https://github.com/kardiachain/go-kardia/blob/master/cmd/cfg/kai_config.yaml#L19

 P2P:
    PrivateKey: 8843ebcb1021b00ae9a644db6617f9c6d870e5fd53624cefe374c1d2d710fd06
    ListenAddress: tcp://0.0.0.0:3000
    MaxPeers: 25

Mainchain API tx_getTransactionReceipt result is not consistent

Prerequisites

Network is running on branch thangn/p2p_channel on commit f5bb0822be045aae8ab48772bbd8fec5e6d4b353. The result of API tx_getTransactionReceipt of some transaction hashes is inconsistent after calling multiple times. Transactions sent from some addresses (of which balances are 0) have tx_getTransaction results and included in kai_getBlockByNumber results, but null tx_getTransactionReceipt results.

Expected Behavior

The tx_getTransactionReceipt result of a transaction hash must be consistent after each calls, with correct statuses.

Current Behavior

Sometimes the result of API tx_getTransactionReceipt is null although there is a transaction with the same hash mined in block.

ERROR[10-19|10:52:33.644]KARDIA ApplyTransaction failed                  tx=0x3ca4a5099dd77fbdce17c85a447c059b7541f92c5b3fd63bdab382d1f143b985 nonce=2534 err="out of gas"
ERROR[10-19|10:52:33.644]KARDIA ApplyTransaction failed                  tx=0x50c2bfdf162f4a8d7222f1c764332e164f60713fded505edf8b90d1c10a47f5f nonce=2535 err="out of gas"
ERROR[10-19|10:52:33.644]KARDIA ApplyTransaction failed                  tx=0x38fe4f4809415fdfa7232c46465cfaecfc02f9f49d6317ce7f31e4b6009cb107 nonce=2536 err="out of gas"
ERROR[10-19|10:52:33.644]KARDIA ApplyTransaction failed                  tx=0xc7f1f40ff165660cd4005adf77f4beccbbccd4f454cc7c3b9e9435e030117200 nonce=2537 err="out of gas"
ERROR[10-19|10:52:33.644]KARDIA ApplyTransaction failed                  tx=0xdaa655f5a4f94b8e0203151135a8e6b150a9123c1d5b740021b871c30896ab21 nonce=2538 err="out of gas"
ERROR[10-19|10:52:33.645]KARDIA ApplyTransaction failed                  tx=0xf87f2bc02c6f4359f844a0564d490da3fa7d0ea33a073e05a05aef40dbb6c3ed nonce=2539 err="out of gas"
ERROR[10-19|10:52:33.645]KARDIA ApplyTransaction failed                  tx=0x31e5a859fffea0b6f6745f2fae5de0f121afc8a2fe0a89e6da6be077a6072c25 nonce=2540 err="out of gas"
ERROR[10-19|10:52:33.645]KARDIA ApplyTransaction failed                  tx=0x8a7d394645c27a48a24fb2bae214d77acba0e6c9983dc68525a9660a69182300 nonce=2541 err="out of gas"
ERROR[10-19|10:52:33.645]KARDIA ApplyTransaction failed                  tx=0xbfd82c3eea5f7bbdcf418f68bb9f35e56ac5e9803665c7837b2a79f71bd5f858 nonce=2542 err="out of gas"
ERROR[10-19|10:52:33.646]KARDIA ApplyTransaction failed                  tx=0x479414c9f812edb756bba317af1835a878494ea1e2966ed7bf6b875264a96335 nonce=2543 err="out of gas"
ERROR[10-19|10:52:33.646]KARDIA ApplyTransaction failed                  tx=0xb459e1beff76d937d49e9b9a99ca3d42535c22d1ee3617892dbfb89aed49f7c9 nonce=2544 err="out of gas"
ERROR[10-19|10:52:33.646]KARDIA ApplyTransaction failed                  tx=0xa2dc78811082df5db25ab88bdfd31ca76e744c54e6364c771219fda4b769a380 nonce=2545 err="out of gas"
ERROR[10-19|10:52:33.646]KARDIA ApplyTransaction failed                  tx=0x7a1bb60cca1ae24005d449757854f848b72b300236da79fb56c77295eaf635c5 nonce=2546 err="out of gas"
ERROR[10-19|10:52:33.646]KARDIA ApplyTransaction failed                  tx=0x6d08c86105a3d56cb3e3c49ea1a08dd5ad286a92083f885534ced45ad7056004 nonce=2547 err="out of gas"
ERROR[10-19|10:52:33.647]KARDIA ApplyTransaction failed                  tx=0x610b2777dfa1c02a9e51a49c5b40df45c470271d7b0486608e11c3e2abb89469 nonce=2548 err="out of gas"
ERROR[10-19|10:52:33.647]KARDIA ApplyTransaction failed                  tx=0x0bcae488a6e3d6d7e6babcce050c6d09ef01a8ddc0f4f58ab5721f85f4c40fbe nonce=2549 err="out of gas"
ERROR[10-19|10:52:33.647]KARDIA ApplyTransaction failed                  tx=0x3edd42f5f1bf743b6be4bec1f5e6afa4ddc1bfb219e493acb47aef4d0f7db7ec nonce=2550 err="out of gas"
ERROR[10-19|10:52:33.647]KARDIA ApplyTransaction failed                  tx=0x6d942d1df421a99bfd3e1afbe8c9395b72635dbd8b7a1a24f6fa9ab5f3af8b2a nonce=2551 err="out of gas"
ERROR[10-19|10:52:33.647]KARDIA ApplyTransaction failed                  tx=0x4c6c4a700df13d9aabe32f9714df0561833d78c4c77e17eedef7848ad6024992 nonce=2552 err="out of gas"
ERROR[10-19|10:52:33.648]KARDIA ApplyTransaction failed                  tx=0xe2ea0c8cb3e38ac367d51b96771a632a999894ca336598cb95e2147cf2652e0b nonce=2553 err="out of gas"
ERROR[10-19|10:52:33.648]KARDIA ApplyTransaction failed                  tx=0x4abe6744c414fc9d3ac38baee3a5315c8cfa39e295caf196de087eb2e0965ed3 nonce=2554 err="out of gas"

The receipt of above transaction is sometimes is exactly returned with status = 1, sometimes is null

Context

Network nodes are running in our workstation.