Since the last couple of upgrades, I started receiving these two errors in the two different flavors. I am not sure why I will see a consensus failure sometimes and others it is just a panic.
I have found a resolution to me getting these errors. I normally run with pruning set to everything
and then I maintain a min block count of unbonding time. When pruning is set to everything, I do not get more than 10 blocks in before the pebbledb issue presents itself in one way or another. With custom pruning set to an interval of 1000
, I also still get the problem. But I have been able to avoid the problem with my pruning interval set to 3000
.
It appears that smaller intervals for pruning are not being chunked properly for pebbledb, resulting in chunks over 4.0Gb for me.
6:44AM ERR Failed to update lp rewards unclaimed error="0ueden: invalid coins" module=server
6:44AM INF executed block height=8652690 module=state num_invalid_txs=0 num_valid_txs=4
6:44AM ERR Stopping peer for error err="error with peer a97a52a34101fcd7d3186fa5cbcff32b12e6332e: peer did not send us anything" module=p2p peer="Peer{MConn{15.235.204.150:28856} a97a52a34101fcd7d3186fa5cbcff32b12e6332e out}"
6:44AM ERR SendTimeout module=blockchain peer=a97a52a34101fcd7d3186fa5cbcff32b12e6332e reason="peer did not send us anything" timeout=15000
6:44AM ERR Stopping peer for error err="read tcp 10.1.1.69:2694->15.235.204.150:28856: use of closed network connection" module=p2p peer="Peer{MConn{15.235.204.150:28856} a97a52a34101fcd7d3186fa5cbcff32b12e6332e out}"
6:44AM ERR error while stopping peer error="already stopped" module=p2p
6:44AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{173.249.24.244:20273} 258f523c96efde50d5fe0a9faeea8a3e83be22ca out}"
panic: pebble: batch too large: >= 4.0GB
goroutine 280 [running]:
github.com/cockroachdb/pebble.(*Batch).grow(0xc26800dfc0?, 0xc26800dff8?)
/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:1414 +0x12d
github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyRecord(0xc003268f00, 0x31, 0x0)
/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:644 +0x85
github.com/cockroachdb/pebble.(*Batch).DeleteDeferred(...)
/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:781
github.com/cockroachdb/pebble.(*Batch).Delete(0xc003268f00, {0xc2658616c0, 0x31, 0x0?}, 0xc2658616c0?)
/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:764 +0x2c
github.com/cometbft/cometbft-db.(*pebbleDBBatch).Delete(0xc26585d310?, {0xc2658616c0?, 0xc26585b400?, 0xc26800e048?})
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/pebble.go:277 +0x2c
github.com/cometbft/cometbft-db.prefixDBBatch.Delete({{0xc0027d8470, 0x10, 0x10}, {0x3f921a0, 0xc2418ec7a0}}, {0xc26585b440, 0x21, 0xf39665?})
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/prefixdb_batch.go:35 +0x11c
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange.func1({0xc2658715f0, 0x31, 0x31}, {0xc26583bfc0, 0x20, 0x20})
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:545 +0x198
github.com/cosmos/iavl.(*nodeDB).traversePrefix(0xc0007cfd40?, {0xc26258f220?, 0x1?, 0x292a100?}, 0xc26800e320)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:847 +0x154
github.com/cosmos/iavl.(*nodeDB).traverseOrphansVersion(0xc0027d7400, 0x83b86a?, 0xc26800e320)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:808 +0x68
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange(0xc0027d7400, 0x83b86a, 0x83ce9f)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:538 +0x425
github.com/cosmos/iavl.(*MutableTree).DeleteVersionsRange(0xc001f04e60, 0x83b86a, 0x83ce9f)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1089 +0x3f
github.com/cosmos/iavl.(*MutableTree).DeleteVersions(0xc001f04e60, {0xc241a66000, 0x4319, 0x4400})
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1077 +0x2aa
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).DeleteVersions(...)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/iavl/store.go:235
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc0012c6e00, 0x98?, {0x0?, 0x0?, 0x0?})
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:641 +0x3d1
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).handlePruning(0xc0012c6e00, 0x840792)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:603 +0x1e7
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0012c6e00)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:462 +0x405
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0003b4b40)
/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:448 +0x1e5
github.com/cometbft/cometbft/abci/client.(*localClient).CommitSync(0xc003f8eae0)
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:267 +0xab
github.com/cometbft/cometbft/proxy.(*appConnConsensus).CommitSync(0xc00187c4b0)
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:116 +0x13c
github.com/cometbft/cometbft/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc002b862f8, 0x6}}, {0xc002b86570, 0xd}, 0x1, 0x840792, {{0xc22b9da7c0, ...}, ...}, ...}, ...)
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:289 +0x252
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc002b862f8, 0x6}}, {0xc002b86570, 0xd}, 0x1, 0x840792, {{0xc22b9da7c0, ...}, ...}, ...}, ...)
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:241 +0x70e
github.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc00118ec40, 0x0)
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:409 +0xb33
created by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 84
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:128 +0x6e
elys.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
elys.service: Failed with result 'exit-code'.
elys.service: Consumed 1min 49.783s CPU time.
Jul 06 06:47:50 artorias elysd[489609]: 6:47AM ERR CONSENSUS FAILURE!!! err="pebble: batch too large: >= 4.0G
B" module=consensus stack="goroutine 1573 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debu
g/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/validator-hom
e-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:737 +0x46\npanic({0x2b73a20?, 0xc
0006a16c0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/cockroachdb/pebble.(*Batch).grow(0x
d4730a7800?, 0xd4730a7838?)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.g
o:1414 +0x12d\ngithub.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyRecord(0xc00210f980, 0x31, 0x0)\n\t/v
alidator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:644 +0x85\ngithub.com/cockroach
db/pebble.(*Batch).DeleteDeferred(...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/pebble@v1.
1.0/batch.go:781\ngithub.com/cockroachdb/pebble.(*Batch).Delete(0xc00210f980, {0xd7c49b7c80, 0x31, 0x0?}, 0xd
7c49b7c80?)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:764 +0x2c\ngit
hub.com/cometbft/cometbft-db.(*pebbleDBBatch).Delete(0xd7c49abda0?, {0xd7c49b7c80?, 0x0?, 0xd4730a7888?})\n\t
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/pebble.go:277 +0x2c\ngithub.com/com
etbft/cometbft-db.prefixDBBatch.Delete({{0xc00661f030, 0x10, 0x10}, {0x3f76260, 0xd4464f4da0}}, {0xd7c49bed80
, 0x21, 0xf39665?})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/prefixdb_bat
ch.go:35 +0x11c\ngithub.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange.func1({0xd7c49cc920, 0x31, 0x31}, {0xd7
c49a94c0, 0x20, 0x20})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:545 +0x1
98\ngithub.com/cosmos/iavl.(*nodeDB).traversePrefix(0xc000a32c00?, {0xd7c22bdbb0?, 0x1?, 0xd4730a7aa0?}, 0xd4
730a7b60)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:847 +0x154\ngithub.co
m/cosmos/iavl.(*nodeDB).traverseOrphansVersion(0xc003aba400, 0x81be18?, 0xd4730a7b60)\n\t/validator-home-4/el
ys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:808 +0x68\ngithub.com/cosmos/iavl.(*nodeDB).DeleteVers
ionsRange(0xc003aba400, 0x81be18, 0x81c6fa)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]
.1/nodedb.go:538 +0x425\ngithub.com/cosmos/iavl.(*MutableTree).DeleteVersionsRange(0xc0008a8000, 0x81be18, 0x
81c6fa)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1089 +0x3f\ngithu
b.com/cosmos/iavl.(*MutableTree).DeleteVersions(0xc0008a8000, {0xd449ec3000, 0x8e2, 0x900})\n\t/validator-hom
e-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1077 +0x2aa\ngithub.com/cosmos/cosmos-sdk/
store/iavl.(*Store).DeleteVersions(...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.
47.11/store/iavl/store.go:235\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc0027e1800
, 0xd8?, {0x0?, 0x0?, 0x0?})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store
/rootmulti/store.go:641 +0x3d1\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).handlePruning(0xc0027e1
800, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/sto
re.go:603 +0x1e7\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0027e1800)\n\t/validator-ho
me-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:462 +0x405\ngithub.com/co
smos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0023990e0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmo
s/[email protected]/baseapp/abci.go:448 +0x1e5\ngithub.com/cometbft/cometbft/abci/client.(*localClient).Com
mitSync(0xc002409ec0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/l
ocal_client.go:267 +0xab\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).CommitSync(0xc003586cd8)\n\t
/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:116 +0x13c\ngithub.c
om/cometbft/cometbft/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc00f48865a, 0x6}}, {0xc00f488660, 0xd}
, 0x1, 0x81d30b, {{0xd3e8772300, ...}, ...}, ...}, ...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/comet
bft/[email protected]/state/execution.go:289 +0x252\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).Apply
Block(_, {{{0xb, 0x0}, {0xc00f48865a, 0x6}}, {0xc00f488660, 0xd}, 0x1, 0x81d30b, {{0xd3e8772300, ...}, ...},
...}, ...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:241 +
0x70e\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc005898e08, 0x81d30b)\n\t/validator-h
ome-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1711 +0xa4c\ngithub.com/cometbf
t/cometbft/consensus.(*State).tryFinalizeCommit(0xc005898e08, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/
github.com/cometbft/[email protected]/consensus/state.go:1620 +0x2e8\ngithub.com/cometbft/cometbft/consensus.(
*State).handleCompleteProposal(0xc005898e08, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometb
ft/[email protected]/consensus/state.go:2016 +0x372\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg
(0xc005898e08, {{0x3f438c0, 0xd3e804dd10}, {0xcea0fce1b0, 0x28}})\n\t/validator-home-4/elys/go/pkg/mod/github
.com/cometbft/[email protected]/consensus/state.go:847 +0x178\ngithub.com/cometbft/cometbft/consensus.(*State)
.receiveRoutine(0xc005898e08, 0x0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/
consensus/state.go:773 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutin
e 374\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:384 +0x10c
\n"
build_tags: netgo,ledger,muslc,osusergo,pebbledb
commit: 07597e3b1386fcc5f89f788713c0e7045f7873a2
cosmos_sdk_version: v0.47.11
go: go version go1.22.5 linux/amd64
name: elys
server_name: elys
version: v0.39.0
Linux artorias 5.15.0-1057-intel-iot-realtime #59-Ubuntu SMP PREEMPT_RT Mon Jun 17 21:50:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux