casper-network / casper-node Goto Github PK
View Code? Open in Web Editor NEWReference client for CASPER protocol
Home Page: https://casper.network
License: Apache License 2.0
Reference client for CASPER protocol
Home Page: https://casper.network
License: Apache License 2.0
Recently peers jump from 50 to 118 for exact 3 minutes. ( speaking dry-run )
Start of event: 2021-03-28 07:03:00
UTC
End of event: 2021-03-28 07:06:00
UTC
This seen in a log:
"message":"dropping connection due to network name mismatch","our_id":"NodeId::Tls(8735..e0b2)","peer_id":"NodeId::Tls(bf5c..c2d2)","our_network":"\"casper-dryrun\"","their_network":"\"delta-11\""
We aware, some validators run dry-run
release on the same IP which been used in Delta's and they are still listed as trusted IP in Delta-11 config.toml. Possible this nodes act as relay. We can remove them from trusted list, but this issue still introcude possibility of exploit.
On the start of the Delta - 3 exact on the moment of genesis generation, my validator lost peers.
Status replay:
{
"api_version": "1.0.0",
"chainspec_name": "casper-delta-3",
"genesis_root_hash": "e81d..b4c0",
"peers": {},
"last_added_block_info": null,
"build_version": "1.6.0"
}
Expected:
{
"api_version": "1.0.0",
"chainspec_name": "casper-delta-3",
"genesis_root_hash": "e81d..b4c0",
"peers": {
"45cf..1364": "13.57.242.68:35000",
"4e5d..4e53": "194.19.235.78:35000",
"716d..9a06": "167.99.74.47:35000",
"db09..0b9b": "54.183.27.75:35000",
"e52c..55ac": "207.180.221.204:35000"
},
"last_added_block_info": null,
"build_version": "1.6.0"
}
24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
std::sys::unix:🧵:Thread:🆕:thread_start
at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
25: start_thread
26: clone
"DEBUG","fields":{"event":"consensus: Shutdown if current era is inactive","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}
{"timestamp":"Nov 20 22:05:00.003","level":"TRACE","fields":{"event":"Consensus(Shutdown)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}
P.S. Peers was present before genesis time.
node version 1_0_0
Suppose that stop state issued by:
"node panicked: Bootstrap failed for node, but it was not in the set of pending connections
Error: "/var/lib/casper/bin/1_0_0/casper-node" "validator" "/etc/casper/1_0_0/config.toml" exited with error"
Using white IP and VDS. ufw disabled. Ubuntu 20.04
The block specification in the JSON-RPC API shows the era report containing lists of rewards and validator weights. This shows up as expected if a block is fetched, e.g.:
"era_report": {
"equivocators": [],
"inactive_validators": [],
"rewards": [
{
"amount": 1202458594726,
"validator": "01026ca707c348ed8012ac6a1f28db031fadd6eb67203501a353b867a08c8b9a80"
},
{
"amount": 19956734910,
"validator": "010427c1d1227c9d2aafe8c06c6e6b276da8dcd8fd170ca848b8e3e8e1038a6dc8"
},
where rewards
is an array of reward objects. However, if a block is received on the events stream with the BlockAdded
event the rewards
and next_era_validator_weights
are returned as maps with the key being the validator public key.
This discrepancy makes life difficult to use a standard process for parsing blocks, and appears to serve no purpose. Could the block presented with the BlockAdded
event use the same format at that returned by the JSON-API?
Been sending some small spam trough my node, was a sequence 1000 transaction.
Observing event on my monitor it was approximatly 870 pass only.
0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
3: std::panicking::begin_panic_handler::{{closure}}
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
6: std::panicking::begin_panic_fmt
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
7: <casper_node::components::gossiper::Gossiper<T,REv> as casper_node::components::Component<REv>>::handle_event
8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
9: casper_node::cli::Cli::run::{{closure}}
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
11: std::thread::local::LocalKey<T>::with
12: tokio::runtime::enter::Enter::block_on
13: tokio::runtime::thread_pool::ThreadPool::block_on
14: tokio::runtime::context::enter
15: casper_node::main
16: std::sys_common::backtrace::__rust_begin_short_backtrace
17: std::rt::lang_start::{{closure}}
18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
std::panicking::try::do_call
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
std::panicking::try
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
std::panic::catch_unwind
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
std::rt::lang_start_internal
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
19: main
20: __libc_start_main
21: _start
panicked at 'internal error: entered unreachable code: can't have gossiped if we don't hold the complete data', node/src/components/gossiper.rs:269:17
{"timestamp":"Jan 04 12:46:15.800","level":"DEBUG","fields":{"event":"deploy gossiper: check gossip timeout for deploy-hash(d008..134f) with NodeId::Tls(bad4..0f7d)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":37307377,"name":"dispatch events"},"spans":[{"ev":37307377,"name":"crank"},{"ev":37307377,"name":"dispatch events"}]}
Join back was very easy, no issue.
Network dead, we not be able to produce any blocks, nodes crashing with:
panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9
While attempting to join the delta testnet, I provided public hex key 01329cc15cb92fca5d5ce6d7fcc881010161547f452e2caad4dff565708b00b8c7
, which was successfully funded. Upon launching the node and attempting to bond, I received some warnings of this form:
Nov 5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34858"},"target":"casper_node::components::small_network"}
Nov 5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov 5 14:53:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:53:49.348","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"725c..21c8","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 14:57:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:57:36.031","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"99c4..8c92","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:01:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:01:36.250","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"4b5b..adef","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"4712..58c3"},"target":"casper_node::components::small_network"}
Nov 5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:43550"},"target":"casper_node::components::small_network"}
Nov 5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov 5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"6e1a..8a85"},"target":"casper_node::components::small_network"}
Nov 5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:40472"},"target":"casper_node::components::small_network"}
Nov 5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.364","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}
Nov 5 15:13:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:13:42.144","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"b56a..6ef1","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:15:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:15:42.292","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"f9ce..270e","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.752","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"3937..faf0"},"target":"casper_node::components::small_network"}
Nov 5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:41402"},"target":"casper_node::components::small_network"}
Nov 5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov 5 15:17:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:17:48.079","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"e8f1..99eb","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:19:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:19:48.185","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"3024..ecc5","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:21:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:21:48.511","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"7ddf..85d6","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:23:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:23:49.251","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"fe6a..d28b","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov 5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"1a63..5169"},"target":"casper_node::components::small_network"}
Nov 5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:37742"},"target":"casper_node::components::small_network"}
Nov 5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov 5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"8cd6..a450"},"target":"casper_node::components::small_network"}
Nov 5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:41120"},"target":"casper_node::components::small_network"}
Nov 5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}
Which soon turned into a ton of errors of this form:
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.753","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"63dc..34cb","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"f1f7..7af1","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"6ab1..2807","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"d353..5ddf","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"0269..7cfd","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"da45..ead8","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"ff54..4652","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.755","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"1be7..3688","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.764","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"cae0..79ee","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov 5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.775","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"fc9b..46a9","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Which ended in a panic of this form:
Nov 5 18:06:21 casper-node casper-node[24645]: 0: casper_node::panic_hook
Nov 5 18:06:21 casper-node casper-node[24645]: 1: core::ops::function::Fn::call
Nov 5 18:06:21 casper-node casper-node[24645]: 2: std::panicking::rust_panic_with_hook
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:530:17
Nov 5 18:06:21 casper-node casper-node[24645]: 3: rust_begin_unwind
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:437:5
Nov 5 18:06:21 casper-node casper-node[24645]: 4: std::panicking::begin_panic_fmt
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:391:5
Nov 5 18:06:21 casper-node casper-node[24645]: 5: casper_node::effect::EffectBuilder<REv>::fatal::{{closure}}
Nov 5 18:06:21 casper-node casper-node[24645]: 6: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Nov 5 18:06:21 casper-node casper-node[24645]: 7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov 5 18:06:21 casper-node casper-node[24645]: 8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov 5 18:06:21 casper-node casper-node[24645]: 9: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov 5 18:06:21 casper-node casper-node[24645]: 10: tokio::runtime::task::raw::poll
Nov 5 18:06:21 casper-node casper-node[24645]: 11: std::thread::local::LocalKey<T>::with
Nov 5 18:06:21 casper-node casper-node[24645]: 12: tokio::runtime::thread_pool::worker::Context::run_task
Nov 5 18:06:21 casper-node casper-node[24645]: 13: tokio::runtime::thread_pool::worker::Context::run
Nov 5 18:06:21 casper-node casper-node[24645]: 14: tokio::macros::scoped_tls::ScopedKey<T>::set
Nov 5 18:06:21 casper-node casper-node[24645]: 15: tokio::runtime::thread_pool::worker::run
Nov 5 18:06:21 casper-node casper-node[24645]: 16: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Nov 5 18:06:21 casper-node casper-node[24645]: 17: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov 5 18:06:21 casper-node casper-node[24645]: 18: tokio::runtime::task::harness::Harness<T,S>::poll
Nov 5 18:06:21 casper-node casper-node[24645]: 19: tokio::runtime::blocking::pool::Inner::run
Nov 5 18:06:21 casper-node casper-node[24645]: 20: tokio::runtime::context::enter
Nov 5 18:06:21 casper-node casper-node[24645]: 21: tokio::runtime::handle::Handle::enter
Nov 5 18:06:21 casper-node casper-node[24645]: 22: std::sys_common::backtrace::__rust_begin_short_backtrace
Nov 5 18:06:21 casper-node casper-node[24645]: 23: core::ops::function::FnOnce::call_once{{vtable.shim}}
Nov 5 18:06:21 casper-node casper-node[24645]: 24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov 5 18:06:21 casper-node casper-node[24645]: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov 5 18:06:21 casper-node casper-node[24645]: std::sys::unix::thread::Thread::new::thread_start
Nov 5 18:06:21 casper-node casper-node[24645]: at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
Nov 5 18:06:21 casper-node casper-node[24645]: 25: start_thread
Nov 5 18:06:21 casper-node casper-node[24645]: 26: clone
Nov 5 18:06:21 casper-node casper-node[24645]: panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:724]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:366:9
When the block proposer initializes, it currently gets all of the deploy hashes going back to genesis. It must only get deploy hashes as far back as the TTL.
Logs:
casper-logs.log
This Finalized block that hasn't been proposed
error started showing up after I raised my bid in the auction.
Logs after restarting the node:
https://pastebin.com/a0GUqhzZ
System specs:
AMD Ryzen 9 3900 12c/24t
128GB DDR4 ECC
2x 2TB NVMe SSDs RAID0
When running:
$ cargo build -p casper-node --release
I'm getting:
Compiling casper-node v0.9.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:23:32
|
23 | const LIBC_RESOURCE: libc::__rlimit_resource_t;
| ^^^^^^^^^^^^^^^^^^^ not found in `libc`
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:32:32
|
32 | const LIBC_RESOURCE: libc::__rlimit_resource_t = 7;
| ^^^^^^^^^^^^^^^^^^^ not found in `libc`
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`
See https://hackmd.io/Y99IYqXVSWaGAwj_u4wqkA for details, section "3 steps plan".
When i try to run :
cargo run --release -- transfer \
--node-address=http://localhost:50101 \
--secret-key=../utils/nctl/assets/net-1/nodes/node-3/keys/secret_key.pem \
--amount=1234567 \
--target-account=$PUBLIC_KEY \
--chain-name=casper-net-1 \
--payment-amount=10000
I got this error:
Failed sending Request(Request { jsonrpc: "2.0", method: "account_put_deploy", params: Some(Map({"deploy": Object({"approvals": Array([Object({"signature": String("01701aa67c643dd16098e830151b5e6555acb83bb58a1fa75324df9d99f2c8ed646b93d410f6ae4a561bcc94d95e6b4b36b9adcc8a7df2b4e0fa9d43ca1137b50d"), "signer": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226")})]), "hash": String("65e0e5a356038e088dca524734fd61aba7dc92229d88fdd8fc6b9753e015e7fc"), "header": Object({"account": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226"), "body_hash": String("889297a958b5385f092f1918bd3b2f04507286076ad9124148505339703d6a21"), "chain_name": String("casper-net-1"), "dependencies": Array([]), "gas_price": Number(1), "timestamp": String("2021-05-06T20:08:21.389Z"), "ttl": String("1h")}), "payment": Object({"ModuleBytes": Object({"args": Array([Array([String("amount"), Object({"bytes": String("021027"), "cl_type": String("U512"), "parsed": String("10000")})])]), "module_bytes": String("")})}), "session": Object({"Transfer": Object({"args": Array([Array([String("amount"), Object({"bytes": String("0387d612"), "cl_type": String("U512"), "parsed": String("1234567")})]), Array([String("target"), Object({"bytes": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4"), "cl_type": Object({"ByteArray": Number(32)}), "parsed": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4")})]), Array([String("id"), Object({"bytes": String("00"), "cl_type": Object({"Option": String("U64")}), "parsed": Null})])])})})})})), id: Num(-799331664064629152) })
My local network is running and the output of nctl-status is the flowing:
validators-1:casper-net-1-node-1 RUNNING pid 6266, uptime 5:24:14
validators-1:casper-net-1-node-2 RUNNING pid 6267, uptime 5:24:14
validators-1:casper-net-1-node-3 RUNNING pid 6268, uptime 5:24:14
validators-2:casper-net-1-node-4 RUNNING pid 6309, uptime 5:24:12
validators-2:casper-net-1-node-5 RUNNING pid 6310, uptime 5:24:12
validators-3:casper-net-1-node-10 STOPPED Not started
validators-3:casper-net-1-node-6 STOPPED Not started
validators-3:casper-net-1-node-7 STOPPED Not started
validators-3:casper-net-1-node-8 STOPPED Not started
validators-3:casper-net-1-node-9 STOPPED Not started
Any idea how to fix this issue please ?
I've tried to deploy a contract from newly fresh compiled sources but the deployment was completed with an error.
I've tried to use the contract that was assembled when starting Delta-1 - everything works fine
The same error was in delta-2
"error_message": "Interpreter error: host module doesn't export function with name casper_call_contract"
Hello,
Is it possible to create a docker-compose yml?
The releases are also in docker Hub, would it be possible to publish them in Github Docker Images?
docs: https://docs.github.com/es/actions/guides/publishing-docker-images
Thanks
This will allow us to prioritize validator nodes over others in the future.
After network made above 100 validators query status end point takes ages to respond.
Reproducing error:
curl -s http://127.0.0.1:8888/status
Expected:
Fast replay
After first very long wait queries back to normal, fast replay.
This not only happen with /status
query, but also present on Prometheus /metrics
/metrics:
/status
People reporting issues with deploys, takes a while to be able to succeed, expected errors:
FailedToGetResponse(reqwest::Error { kind: Request, url: Url { scheme: "http", host: Some(Ipv4(127.0.0.1)), port: Some(7777), path: "/rpc", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) })', client/src/deploy/put.rs:56:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
I have installed casperlabs in / mountpoint.
The node was running as a validator and the mount point became 100% full and the node crashed. I have cleared the mount point and tried restarting the node but it fils with the below error.
CHain : Delta-10
0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:595:17
3: std::panicking::begin_panic_handler::{{closure}}
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:497:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:493:5
6: std::panicking::begin_panic_fmt
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:435:5
7: casper_node::effect::EffectBuilder::load_state::{{closure}}::{{closure}}
8: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
9: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
10: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
11: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
12: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
14: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
15: tokio::runtime::task::raw::poll
16: std::thread::local::LocalKey::with
17: tokio::runtime::thread_pool::worker::Context::run_task
18: tokio::runtime::thread_pool::worker::Context::run
19: tokio::macros::scoped_tls::ScopedKey::set
20: tokio::runtime::thread_pool::worker::run
21: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
22: tokio::runtime::task::core::Core<T,S>::poll
23: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
24: tokio::runtime::task::harness::Harness<T,S>::poll
25: tokio::runtime::blocking::pool::Inner::run
26: tokio::runtime::context::enter
27: std::sys_common::backtrace::__rust_begin_short_backtrace
28: core::ops::function::FnOnce::call_once{{vtable.shim}}
29: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
std::sys::unix::thread::Thread::new::thread_start
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys/unix/thread.rs:71:17
30: start_thread
31: clone
panicked at 'could not deserialize state from storage type name "core::option::Option<casper_node::components::block_proposer::deploy_sets::BlockProposerDeploySets>" err InvalidTagEncoding(61)', node/src/effect.rs:1252:13
Error: "/var/lib/casper/bin/1_0_0/casper-node" "validator" "/etc/casper/1_0_0/config.toml" exited with error
I'm attempting to obtain historical (balance+stake) information from the JSON-RPC interface. I can obtain historical balance (which is currently always 0 as all balance is staked) and current stake (from the state_get_auction_info method) but not historical stake.
Is historical stake available somewhere? Or some other method to calculate an account's (balance+stake), for example obtaining per-era reward data?
Happen Dec 26 02:52:25 UTC
I noticed Scheduler Queue overload, peak happen 1:00 UTC one hour before node crash:
DELTA BALANCER: scheduler_queue_api_count: 0
DELTA BALANCER: scheduler_queue_network_count: 47.3333
DELTA BALANCER: scheduler_queue_network_incoming_count: 4947216.0417
DELTA BALANCER: scheduler_queue_regular_count: 23.125
DELTA BALANCER: scheduler_queue_total_count: 4947286.5
Trying to recover and restart node so far unsuccessfully:
Main process exited, code=killed, status=6/ABRT
Node can't sync above "height": 4215,
See the attached file for log from the node.
Crash can be reproduced by installing everything on a clean box and syncing the node with trusted hashes.
Tested: Around 20 Resyncs
Crashes: 6
Tested with different trusted hashes - Crash is not always happening (So hash is valid)
One of the hashes tested with: 3b0e4ae85f096b0be2e28402c3f32c471d0f5f4a470d2709c731e0febf3d26ec
Environment: Ubuntu 18.04 (Docker - can provide build scripts)
When querying the deploy with hash f81d0a32890b47a5dfbb55c7bffe75d2a9301c442f7857b1908409b8e9921e83
on mainnet returns a JSON structure with the following:
"session": {
"Transfer": {
"args": [
...
[
"target",
{
"cl_type": {
"ByteArray": 32
},
"bytes": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f",
"parsed": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f"
}
],
...
]
}
},
however cl_type
is meant to be an enum rather than an object. Given the byte array size can be inferred from the bytes
field can this be altered so the cl_type
is just a simple ByteArray
value as per other CL types?
Found in the nightly nctl tests: Happens consistently in itst01.
Drone: https://drone-auto-casper-network.casperlabs.io/casper-network/casper-node/202/7/2
cladmin@admin:~$ tail stdout.log
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"small network: request: gossip: AddressGossiper::(gossip(gossiped-address 127.0.0.1:34557))","q":"Network"},"target":"casper_node::reactor","span":{"ev":157,"name":"dispatch events"},"spans":[{"ev":157,"name":"crank"},{"ev":157,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"address gossiper: gossiped gossiped-address 127.0.0.1:34557 to ","q":"Regular"},"target":"casper_node::reactor","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"message":"paused gossiping since no more peers to gossip to","item":"gossiped-address 127.0.0.1:34557"},"target":"casper_node::components::gossiper","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"linear chain: Initialize timeout","q":"Regular"},"target":"casper_node::reactor","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"hasn't downloaded any blocks in expected time window. Shutting down…"},"target":"casper_node::components::linear_chain_sync","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"control: fatal error [node/src/components/linear_chain_sync.rs:824]: no syncing progress, shutting down…","q":"Control"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"ERROR","fields":{"message":"fatal error via control announcement","file":"node/src/components/linear_chain_sync.rs","line":"824","msg":"no syncing progress, shutting down…"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"shutting down socket, no longer accepting incoming connections","our_id":"NodeId::Tls(0254..87d6)"},"target":"casper_node::components::small_network"}
{"timestamp":"May 09 18:05:18.589","level":"INFO","fields":{"message":"exiting casper-node","exit_code":"101"},"target":"casper_node"}
{"timestamp":"May 09 18:05:18.591","level":"WARN","fields":{"message":"failed running \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/bin/1_0_0/casper-node\" \"validator\" \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/config/1_0_0/config.toml\"","exit_status":"exit code: 101"},"target":"casper_node_launcher::utils"}
Many guys experience sync issues. I observe network and found Scheduler Queue Total Count
constantly increasing:
See spike on gossiper here:
Which correlated with peers been joining my node and then disconnected in same frequence here:
Where is also a glitch in metrics which can indecate restart of my validator unit or can be related to this issue we have presiously, where metrics with all endpoints can't be served properly and fall off periodically.
This error recently popups in my log:
{
"timestamp": "Jan 10 05:15:47.654",
"level": "DEBUG",
"fields": {
"message": "connection error: connection closed before message completed"
},
"target": "hyper::server::conn::spawn_all"
}
Current seed node have 7 peers connected, where one of this peer i check he is from DELTA 5 and he have about 30 peers connected from previous network. Possibility of cross chain talk.
{
"api_version": "1.0.0",
"chainspec_name": "casper-delta-5",
"genesis_root_hash": "a665..29c6",
"peers": [
{
"node_id": "NodeId::Tls(08ba..cc65)",
"address": "157.90.21.155:35000"
},
{
"node_id": "NodeId::Tls(08d4..0273)",
"address": "168.119.250.90:35000"
},
{
"node_id": "NodeId::Tls(1237..8332)",
"address": "159.69.218.20:35000"
},
{
"node_id": "NodeId::Tls(17d5..2f0f)",
"address": "168.119.123.213:35000"
},
{
"node_id": "NodeId::Tls(193d..23ab)",
"address": "157.90.26.198:35000"
},
{
"node_id": "NodeId::Tls(2fc9..7c20)",
"address": "116.202.24.91:35000"
},
{
"node_id": "NodeId::Tls(35b2..a27e)",
"address": "78.47.110.240:35000"
},
{
"node_id": "NodeId::Tls(3e9f..87e4)",
"address": "116.203.182.37:35000"
},
{
"node_id": "NodeId::Tls(4037..0b1e)",
"address": "88.99.171.225:35000"
},
{
"node_id": "NodeId::Tls(4214..1f70)",
"address": "135.181.40.95:35000"
},
{
"node_id": "NodeId::Tls(4a9b..2834)",
"address": "88.99.120.29:35000"
},
{
"node_id": "NodeId::Tls(52f0..b54c)",
"address": "94.130.224.158:35000"
},
{
"node_id": "NodeId::Tls(5a06..7fe9)",
"address": "168.119.173.204:35000"
},
{
"node_id": "NodeId::Tls(5e61..d73f)",
"address": "135.181.42.149:35000"
},
{
"node_id": "NodeId::Tls(64b6..aec8)",
"address": "116.203.206.169:35000"
},
{
"node_id": "NodeId::Tls(64c9..abb4)",
"address": "88.99.36.131:35000"
},
{
"node_id": "NodeId::Tls(6758..caac)",
"address": "49.12.96.112:35000"
},
{
"node_id": "NodeId::Tls(6d85..298b)",
"address": "78.46.121.69:35000"
},
{
"node_id": "NodeId::Tls(75e5..f7d6)",
"address": "78.47.9.189:35000"
},
{
"node_id": "NodeId::Tls(874f..9bcb)",
"address": "95.217.154.113:35000"
},
{
"node_id": "NodeId::Tls(90eb..7a13)",
"address": "95.216.147.88:35000"
},
{
"node_id": "NodeId::Tls(9435..f7d9)",
"address": "78.47.18.35:35000"
},
{
"node_id": "NodeId::Tls(98b6..dd10)",
"address": "78.46.186.163:35000"
},
{
"node_id": "NodeId::Tls(a9e7..4b5e)",
"address": "168.119.59.12:35000"
},
{
"node_id": "NodeId::Tls(ac63..e930)",
"address": "168.119.169.105:35000"
},
{
"node_id": "NodeId::Tls(b27e..72d6)",
"address": "168.119.177.37:35000"
},
{
"node_id": "NodeId::Tls(b328..035e)",
"address": "168.119.126.225:35000"
},
{
"node_id": "NodeId::Tls(b44b..ad4e)",
"address": "78.47.131.222:35000"
},
{
"node_id": "NodeId::Tls(dc8d..7985)",
"address": "116.202.106.191:35000"
},
{
"node_id": "NodeId::Tls(e39e..bab8)",
"address": "78.46.198.83:35000"
},
{
"node_id": "NodeId::Tls(ea95..f095)",
"address": "95.217.213.84:35000"
},
{
"node_id": "NodeId::Tls(f887..28cd)",
"address": "157.90.26.194:35000"
},
{
"node_id": "NodeId::Tls(fbd5..e151)",
"address": "168.119.59.64:35000"
},
{
"node_id": "NodeId::Tls(fef3..6370)",
"address": "195.201.148.78:35000"
}
],
"last_added_block_info": {
"hash": "6225f5a1130929f4e5f59f96d96c8436c007c67a949b3d1b63672d81287c5bab",
"timestamp": "2020-12-27T09:03:14.944Z",
"era_id": 221,
"height": 4396,
"state_root_hash": "d735ceeb4e9ec2f31a9f19e63cdab4445e5df85a7f805d69f30f6c2fd9dfd3f2",
"creator": "01b952f866b554dbb80b420348bffcdc1033b5f97444e47dc8630db2d05da255d6"
},
"build_version": "0.3.1"
}
{
"api_version": "1.0.0",
"chainspec_name": "casper-delta-6",
"genesis_root_hash": "9eb4..aef6",
"peers": [
{
"node_id": "NodeId::Tls(1905..f268)",
"address": "129.146.102.188:35000"
},
{
"node_id": "NodeId::Tls(2455..759c)",
"address": "129.146.58.40:35000"
},
{
"node_id": "NodeId::Tls(3601..bb91)",
"address": "161.97.136.153:56588"
},
{
"node_id": "NodeId::Tls(3d16..a5d8)",
"address": "158.101.3.185:35000"
},
{
"node_id": "NodeId::Tls(59da..01e6)",
"address": "18.191.239.36:35000"
},
{
"node_id": "NodeId::Tls(7a60..f706)",
"address": "101.36.118.242:35000"
},
{
"node_id": "NodeId::Tls(f03f..1d84)",
"address": "18.144.176.168:35000"
}
],
"last_added_block_info": {
"hash": "2a48f3e41a2ee7eaf1b2d04eb8b1b3afa90091bc597872c858445fe328191b74",
"timestamp": "2020-12-30T12:12:07.552Z",
"era_id": 18,
"height": 308,
"state_root_hash": "ffd1515f875a1e27a19db90176ce01cfb6b7368ce5c2ac0be7838755abd4e596",
"creator": "013f774a58f4d40bd9b6cce7e306e53646913860ef2a111d00f0fe7794010c4012"
},
"build_version": "0.4.0"
}
Current peers situation
STATUS IP ADDRESS TRUSTED HASH CHAIN NAME ERA HEIGH VERSION
--------------------------------------------------------------------------------------------
Trusted 18.144.176.168 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Trusted 13.57.200.251 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
--------------------------------------------------------------------------------------------
HOST 127.0.0.1 null casper-delta-6 null null 0.4.0
Useless 49.12.96.112 6225f .... 7c5bab casper-delta-5 221 4396 0.3.1
Available 13.57.200.251 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Available 129.146.102.188 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Useless 95.216.147.88 6225f .... 7c5bab casper-delta-5 221 4396 0.3.1
Available 129.146.58.40 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Available 158.101.3.185 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Available 18.191.239.36 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Useless 135.181.147.7 6225f .... 7c5bab casper-delta-5 221 4396 0.3.1
Available 101.36.118.242 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Useless 168.119.177.37 6225f .... 7c5bab casper-delta-5 221 4396 0.3.1
Available 18.144.176.168 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532
PeersCount: 12
Useless peers: 4
Blocked access: 0
Good condition: 8
Ok, so this above happen on the time of the sync, after sync complete situation change and ancient peers been disconnected from the node accordingly:
STATUS IP ADDRESS TRUSTED HASH CHAIN NAME ERA HEIGH VERSION
--------------------------------------------------------------------------------------------
Trusted 18.144.176.168 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Trusted 13.57.200.251 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
--------------------------------------------------------------------------------------------
HOST 127.0.0.1 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Available 13.57.200.251 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Blocked 88.99.120.29
Available 18.144.176.168 c13a2 .... baa532 casper-delta-6 19 318 0.4.0
Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532
PeersCount: 3
Useless peers: 0
Blocked access: 1
Good condition: 2
Implement CEP-57.
There is already a draft PR #1557
Needs the following touch-ups before it is merged:
for
loops to use iteration insteadcargo bench --bench transfer_bench -- --save-baseline 'dev'
on dev
branch, then on this branch run cargo bench --bench transfer_bench -- --baseline 'dev'
Split up the pending deploys in the block proposer, to separate transfers and other deploys from the start.
Possibly remove DeployType, and split up AppendableBlock::add into two methods, that verify that it's a transfer or not, as expected.
First, thanks for working on this product!
When following the build instructions I get
Compiling warp-json-rpc v0.2.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:23:32
|
23 | const LIBC_RESOURCE: libc::__rlimit_resource_t;
| ^^^^^^^^^^^^^^^^^^^ not found in `libc`
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:31:32
|
31 | const LIBC_RESOURCE: libc::__rlimit_resource_t = libc::RLIMIT_NOFILE;
| ^^^^^^^^^^^^^^^^^^^ not found in `libc`
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`
I've been following CasperLabs' work since I heard a talk about it in DevCon V. I really like the CBC protocol and its designs. However, on license matters, I think there's something I must bring up, as I couldn't find discussions on this elsewhere. There are things that have to be fixed (GPL violations), and things that many developers and me would really appreciate clarifications, so that we can properly utilize the codebase, and avoid any "misuse" defined in your license terms.
If you knew MongoDB, you know that "open source" is a somewhat sensitive word -- as people expect codebases under an "open source" license would comply with the open source definition. OSI reviews provided some reasons why they think the current license is not appropriately "open source". While that's only according to their OSI's definitions, I couldn't find any responses from CasperLabs on how those critics are addressed, and what's CasperLabs' definition of open source. Without this, the word "open source" in the license title "CapserLabs Open Source License" is going to be a source of trouble when the software is made aware by the wider open source community.
The license also contains some reference to "FLOSS", which is supposed to mean free/libre and open source software. The current license also heavily deviates from what the community commonly understand it is.
A recommendation from mine would be if the term "open source" can be removed from the license title and preamble. Why not simply call it "free and open"? No one will question that whether the license is "free and open" at all, but many will question whether the license is "open source" or "FLOSS".
I also hope if there can be a more concrete summary page on what is allowed and what is not allowed under the current license terms, as after reading the whole license, I still find it quiet confusing. Not only does this benefit developers like me, but also ecosystems and enterprises heavily.
The license this codebase use (COSLv1) is an open-codebase license. As it's not compatible with GPL, no code in this codebase shall depend on it. However, currently there's one violation of it:
That dependency has to be removed from the current codebase to avoid the violation. Given it's only a simple dependency, I think this would be relatively easy to do.
hello, when running-one-node), I get this error
Mar 17 02:45:30.638 INFO [casper_node::cli cli.rs:149] node starting up; version=0.9.0-cb1d20ad
Mar 17 02:45:30.638 WARN [casper_node::reactor reactor.rs:437] large event size, consider reducing it or boxing; event_size=272
Error: chainspec error: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1
Caused by:
0: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1
1: timestamp format is invalid for key `network.timestamp` at line 208 column 1
➜ casper-node pwd
/etc/casper-node
➜ casper-node ls
chainspec.toml config.toml
chainspec.toml and config.toml copy from chainspec.toml and config.toml
[network]
# Human readable name for convenience; the genesis_hash is the true identifier. The name influences the genesis hash by
# contributing to the seeding of the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
name = 'casper-example'
# Timestamp for the genesis block. This is the beginning of era 0. By this time, a sufficient majority (> 50% + F/2 —
# see finality_threshold_percent below) of validator nodes must be up and running to start the blockchain. This
# timestamp is also used in seeding the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
timestamp = '${TIMESTAMP}'
How should I do?
Trying to join delta-6 recently, after couple of minutes node crash with provided error:
0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
3: std::panicking::begin_panic_handler::{{closure}}
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
6: std::panicking::begin_panic_fmt
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
7: casper_node::components::consensus::<impl casper_node::components::Component<REv> for casper_node::components::consensus::era_supervisor::EraSupervisor<I>>::handle_event
8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
9: casper_node::cli::Cli::run::{{closure}}
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
11: std::thread::local::LocalKey<T>::with
12: tokio::runtime::enter::Enter::block_on
13: tokio::runtime::thread_pool::ThreadPool::block_on
14: tokio::runtime::context::enter
15: casper_node::main
16: std::sys_common::backtrace::__rust_begin_short_backtrace
17: std::rt::lang_start::{{closure}}
18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
std::panicking::try::do_call
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
std::panicking::try
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
std::panic::catch_unwind
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
std::rt::lang_start_internal
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
19: main
20: __libc_start_main
21: _start
panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9
Probably getting this from nodes which are on previous delta still.
{
"timestamp": "Dec 30 09:32:32.937",
"level": "INFO",
"fields": {
"message": "NodeId::Tls(3da7..bf59): dropping connection to NodeId::Tls(4037..0b1e) due to genesis config hash mismatch",
"our_hash": "42edd6b09c101a65d7aa3626bf0811cb39fa4d2d3957d8a28868edd4aa8d1a9e",
"their_hash": "0e9f4c541abeb97a34708f732b6d0acc7345dbda394ae115ffef213461d1a892"
},
Attached logs are in pure JSON format, can be easily to operate, this is full cycle from start to crash.
Inside zip is tar.xz archive, GitHub only support zip, sorry for it
Currently, the event stream indexes every event it produces. The numbering starts at 0 and increases by 1 for every subsequent event. However, if the node stops and restarts (which it would do e.g. during an upgrade) then the numbering starts from 0 again.
We want to persist the indexing and on the restart, continue from where we left off.
Observing last 6 hours on my monitoring solution, I notice table items are significantly increased.
First thoughts it was someone spamming allot of deploys, but observing deploys count I figure out this is not the issue.
Here is some screenshots from my monitor ( 6 hour frame ):
Unpleasant consequences:
On load average we can observe strange behavior where in same 6 hours time frame 5 and 15 minutes decrease, but small momentum 1 minute spikes is growing.
After recent crash and no success with join it again I try to run validator from scratch.
Service exit error:
Main process exited, code=killed, status=6/ABRT
Valdator log is overloaded and produced about 4G since last 30 minutes, mostly:
message received","msg":"payload: AddressGossiper::(gossip-response
message received","msg":"payload: AddressGossiper::(gossip(gossiped-address
Warnings present in a log:
"WARN","fields":{"message":"Finality signatures not handled in joiner reactor"},"target":"casper_node::reactor::joiner","span":{"ev":7206551,"name":"dispatch events"},"spans":[{"ev":7206551,"name":"crank"},{"ev":7206551,"name":"dispatch events"}]}
"WARN","fields":{"message":"network announcement ignored.","other":"Consensus(Protocol { era_id.0: 214, .. })"},"target":"casper_node::reactor::joiner","span":{"ev":7239826,"name":"dispatch events"},"spans":[{"ev":7239826,"name":"crank"},{"ev":7239826,"name":"dispatch events"}]}
Most of the warning spam is:
"WARN","fields":{"message":"NodeId::Tls(7e9d..fa72): outgoing connection failed","peer_address":
Just two warnings about:
{"timestamp":"Dec 26 14:23:37.920","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"160"},"target":"casper_node::reactor"}
{"timestamp":"Dec 26 14:23:38.245","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"544"},"target":"casper_node::reactor"}
0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
3: std::panicking::begin_panic::{{closure}}
4: std::sys_common::backtrace::__rust_end_short_backtrace
5: std::panicking::begin_panic
6: <casper_node::components::linear_chain_sync::LinearChainSync<I> as casper_node::components::Component<REv>>::handle_event
7: <casper_node::reactor::joiner::Reactor as casper_node::reactor::Reactor>::dispatch_event
8: casper_node::reactor::Runner<R>::crank::{{closure}}
9: casper_node::cli::Cli::run::{{closure}}
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
11: std::thread::local::LocalKey<T>::with
12: tokio::runtime::enter::Enter::block_on
13: tokio::runtime::thread_pool::ThreadPool::block_on
14: tokio::runtime::context::enter
15: tokio::runtime::handle::Handle::enter
16: casper_node::main
17: std::sys_common::backtrace::__rust_begin_short_backtrace
18: std::rt::lang_start::{{closure}}
19: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
std::panicking::try::do_call
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
std::panicking::try
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
std::panic::catch_unwind
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
std::rt::lang_start_internal
at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
20: main
21: __libc_start_main
22: _start
node panicked: Failed to download linear chain.
The /status endpoint seems to currently report the wrong port for bidirectional nodes, using incoming instead of outgoing connections. This behavior should be reverted to be in line with the previous one, where an outgoing connection always wins out over an incoming one.
Accidentally first filed as https://github.com/casper-network/Governance/issues/7
#1413 fixed a regression that caused a node not to reconnect to a crashed node. The conditions are somewhat tough to replicate outside of our integration, but it could be possible to add these to the networking tests using testing::Network
. We should definitely try to replicate this bug (by temporarily turning off the fix) and add a test for it.
Based on:
I've found the "scheduler_queue_total_count" to be a more useful one tbh. It shows the current total backlog of these kind of events waiting to be processed. We aim for it to always be very low - ideally single or double digits. When we saw the recent OOM issues, before the node crashed, that value would invariably skyrocket to hundreds of thousands - meaning we had a huge backlog of events
Here is the moment of actual crash: ( 2021-01-10 14:11:52 UTC )
Since we sure this is OOM most luckily:
Here is no doubts, the reason for node to loose ability providing us with HTTP and RPC end point is coming from the same source. We can observe it on mostly every graph, as example I post graphs below.
From here we can see, issue start accumulating before the actual catastrophe ( 2021-01-10 06:27:00 UTC )
I been tracking network activity, and here is some in case can be usefull.
From here we can see, since my crawler still crawling after node colapse, new IP addresses still joining chain.
At moment of crash my node have 69 peers connected, auction order book have 65 bids and network coun 132 unique IP addresses. My validator crash happen when 133 addresses been count, it was exactly on the moment when new IP join network, not sure if this can be related to some specific missconfigured node joining.
Last IP's joined network:
130 95.216.224.235
131 95.217.0.233
132 95.217.154.113
133 95.217.154.200
134 95.217.213.84
135 95.217.84.84
136 95.217.94.83
List of current "still alive peers":
104.197.195.68 Era: 101 Height: 2647
104.236.61.140 Era: 101 Height: 2647
107.22.247.79 Era: 101 Height: 2647
116.202.106.191 Era: 83 Height: 2277
116.202.24.91 Era: 80 Height: 2205
116.203.157.37 Era: 101 Height: 2647
116.203.206.169 Era: 101 Height: 2647
116.203.214.160 Era: 101 Height: 2647
116.203.254.245 Era: 101 Height: 2647
116.203.69.88 Era: 101 Height: 2647
129.146.102.188 Era: 101 Height: 2647
129.146.102.28 Era: 101 Height: 2647
129.146.137.190 Era: 101 Height: 2647
129.146.166.34 Era: 101 Height: 2647
129.146.17.202 Era: 101 Height: 2647
129.146.243.139 Era: 101 Height: 2647
129.146.58.40 Era: 101 Height: 2647
129.146.83.137 Era: 101 Height: 2647
134.209.16.172 Era: 101 Height: 2647
135.181.129.210 Era: 101 Height: 2647
135.181.143.233 Era: 97 Height: 2605
135.181.163.79 Era: 101 Height: 2647
135.181.192.149 Era: 101 Height: 2647
135.181.25.45 Era: 101 Height: 2647
135.181.30.191 Era: 101 Height: 2647
135.181.35.187 Era: 101 Height: 2647
135.181.40.95 Era: 101 Height: 2647
135.181.89.76 Era: 101 Height: 2647
135.181.97.186 Era: 101 Height: 2647
13.68.180.213 Era: 101 Height: 2647
138.197.65.212 Era: 101 Height: 2647
143.110.225.98 Era: 101 Height: 2647
144.76.199.44 Era: 101 Height: 2647
144.91.109.12 Era: 101 Height: 2647
157.90.19.142 Era: 101 Height: 2647
157.90.26.194 Era: 101 Height: 2647
158.101.3.185 Era: 101 Height: 2647
158.247.204.136 Era: 101 Height: 2647
161.97.124.55 Era: 101 Height: 2647
161.97.152.70 Era: 101 Height: 2647
161.97.82.77 Era: 101 Height: 2647
161.97.96.132 Era: 101 Height: 2647
167.86.75.50 Era: 101 Height: 2647
168.119.123.213 Era: 101 Height: 2647
168.119.173.204 Era: 101 Height: 2647
168.119.179.150 Era: 101 Height: 2647
168.119.59.64 Era: 101 Height: 2647
172.105.13.176 Era: 101 Height: 2647
173.212.205.2 Era: 101 Height: 2647
194.19.235.78 Era: 101 Height: 2647
195.201.148.78 Era: 96 Height: 2565
195.201.220.118 Era: 101 Height: 2647
207.180.198.56 Era: 101 Height: 2647
207.180.245.251 Era: 101 Height: 2647
3.19.76.184 Era: 101 Height: 2647
34.121.43.83 Era: 101 Height: 2647
34.65.104.178 Era: 101 Height: 2647
35.198.176.85 Era: 101 Height: 2647
35.233.165.80 Era: 101 Height: 2647
40.75.89.48 Era: 101 Height: 2647
46.4.15.204 Era: 101 Height: 2647
49.12.109.102 Era: 101 Height: 2647
49.12.66.192 Era: 97 Height: 2605
49.12.96.112 Era: 101 Height: 2647
52.247.126.146 Era: 101 Height: 2647
52.247.219.167 Era: 101 Height: 2647
78.46.121.69 Era: 101 Height: 2647
78.46.186.163 Era: 101 Height: 2647
78.47.10.119 Era: 101 Height: 2647
78.47.110.240 Era: 101 Height: 2647
78.47.9.189 Era: 101 Height: 2647
88.198.227.113 Era: 101 Height: 2647
88.99.120.29 Era: 101 Height: 2647
88.99.36.131 Era: 101 Height: 2647
94.130.224.158 Era: 101 Height: 2647
95.216.147.88 Era: 101 Height: 2647
95.216.164.122 Era: 101 Height: 2647
95.217.154.113 Era: 101 Height: 2647
95.217.154.200 Era: 101 Height: 2647
95.217.213.84 Era: 101 Height: 2647
95.217.84.84 Era: 101 Height: 2647
Dead peers:
103.212.121.38
116.202.17.49
116.203.157.213
116.203.182.37
135.181.118.39
135.181.147.7
135.181.152.29
135.181.42.149
13.57.200.251
144.76.111.74
144.91.123.104
144.91.83.59
157.90.21.155
157.90.26.198
158.101.12.238
159.146.54.127
159.69.218.20
161.97.136.153
161.97.143.67
167.99.74.47
168.119.126.225
168.119.169.105
168.119.177.31
168.119.177.37
168.119.250.90
168.119.49.220
168.119.59.12
173.212.200.158
173.249.31.34
18.144.176.168
18.217.179.179
20.57.161.70
206.189.4.64
207.180.199.157
207.180.213.119
207.180.243.57
34.122.124.107
34.123.112.124
34.75.153.42
35.236.56.221
35.239.228.18
45.61.52.142
46.4.91.24
62.171.165.37
78.46.198.83
78.46.200.17
78.47.131.222
78.47.18.35
78.47.218.195
88.198.200.113
88.99.171.225
88.99.227.165
95.216.224.235
95.217.0.233
95.217.94.83
Last delta-8 geomap:
Geodata CSV attached
delta-8_geo_data.txt
Currently a node uses a single directory to hold its LMDB database files and various "unit" files created by consensus. To avoid problems if a node is run against e.g. the testnet and later the mainnet, we need to create a subdirectory in the storage folder which is named as per the given network, i.e. it will use the chainspec.network.name
value for the subdir name.
There are a few aspects to this task:
Storage::new()
. This will be done similarly to the protocol_version
: it's also specified in the chainspec, and the initializer reactor (which constructs the Storage
component) accesses the chainspec by calling the ChainspecLoader
component.ContractRuntime
component's constructor should be changed. Currently it takes a copy of the Storage component's config (storage_config: WithDir<StorageConfig>
), but only needs that to deduce the path in which to create its own LMDB files. We should replace that variable with a simple PathBuf
or &Path
which will provided via a new getter on the Storage
component. The initializer reactor has access to the newly-constructed Storage
just before it constructs the ContractRuntime
.Storage
component move the 4 relevant files in its constructor. The files are data.lmdb
, data.lmdb-lock
, storage.lmdb
and storage.lmdb-lock
. They just need to be moved from what is currently root
to the new subdir of root
. That code should have a TODO added to help remind us to remove it after the next version is published, as we don't want this behaviour to persist after the upgrade in which this change is introduced.data.lmdb
and data.lmdb-lock
files need not be actual LMDB files for the purposes of this test. They're the DB files backing global state (owned by ContractRuntime
), but the test only needs to ensure they're moved appropriately - they won't be actually read as DB files. We tend to use tempfile::tempdir()
for testing such filesystem-related things.This task is about implementing support for improved local keys, which intends to resolve past problems that lead to their removal. Contract developers will use this to store data in a separate keyspace prefixed by the URef address provisioned to a user by the system.
CEP for reference: casper-network/ceps#39
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.