Performance: v3.1.0 vs v3.0.1 about openethereum HOT 11 OPEN

Here's a utility (https://github.com/mdben1247/web3_test) for measuring performance on head; processing new blocks and incoming transaction as they come in. This particular requirement makes the benchmarks quite difficult to compare as real-time workload changes constantly. So, v3.0.1 and v3.1.0 were run simultaneously on the same machine with the same settings (fatdb, 30GB total cache), changing just the WS and network port numbers. Both started at the same time with same priorities and the system had enough spare resources, so there was little indication of competition for CPU, memory or bandwidth, or at least none of the two processes would have an edge over another. An alternative would be to run one after another switching between them many times, to reduce the impact of varying workload, but that could take a lot of time to get consistent results.

Benchmark consists of the following:

• reading incoming pending transactions and performing tracing on them - PendTrace,
• reading new blocks and performing batched (Batch_x) and sequential (Seq_x) state queries of erc20 balances,
• reading new blocks and getting logs for erc20 transfers (TransTest) and uniswap v2 swaps (UswTest).

These are the result of running it for 30 mintues:

v3.0.1 (old version):
Batch_0   : avg:   111.6 ms, rel:  0.0082%, iter: 137
Batch_1   : avg:   110.8 ms, rel:  0.0081%, iter: 137
Seq_0     : avg:    52.5 ms, rel:  0.0038%, iter: 137
Seq_1     : avg:    50.3 ms, rel:  0.0037%, iter: 137
TransTest : avg:    48.9 ms, rel:  0.0036%, iter: 137
UswTest   : avg:    28.5 ms, rel:  0.0021%, iter: 137
PendTrace : avg:     3.0 ms, rel:  0.1592%, iter: 97882
TOTAL rel:  0.1887%

v3.1.0 (new version):
Batch_0   : avg:   191.4 ms, rel:  0.0140%, iter: 137
Batch_1   : avg:   191.2 ms, rel:  0.0140%, iter: 137
Seq_0     : avg:    94.1 ms, rel:  0.0069%, iter: 137
Seq_1     : avg:    89.3 ms, rel:  0.0065%, iter: 137
TransTest : avg:    61.8 ms, rel:  0.0045%, iter: 137
UswTest   : avg:    47.6 ms, rel:  0.0035%, iter: 137
PendTrace : avg:     5.9 ms, rel:  0.2284%, iter: 72515
TOTAL rel:  0.2779%

For example, v3.0.1 needed an average 111.6ms to execute Batch_0 vs 191.4 ms for the new version. Total number of blocks processed were 137. Its interesting that the old version processed larger number of incoming transaction (97882), but it still took less time to process them all. rel % indicates the amount to total runtime consumed by processing, which is rather low since most of the time nothing is going on.

Peak RAM usage was about 11GB for v3.0.1, and 15GB for v3.1.0.

Complementary to these synthetic results are block import times from openethereum log, which are consistently significantly higher in the new version. Also, there are observations from production environment, where by switching to new version we see a large increase in processing times.

from openethereum.

By also upgrading kvdb-*, memory-db, hash-db and trie-db libraries, we finally get back to performance level comparable to v3.0.1. That is, block verification and various RPC workloads are about factor of two faster than vanilla v3.1.1. I'll be doing some more testing and then make a PR if that's fine?

from openethereum.

@rakita We've seen similar performance degradation trying out 3.1 on kovan. Median (50th percentile) performance seems unchanged, but the 90th percentile and higher is far worse, and seems to be the same across RPCs (e.g. eth_blockNumber and eth_call show very similar graphs), so it's not isolated to for example just accessing the state tries.

from openethereum.

@mdben1247, we backported from 2.5.13 because 3.0.1 was really unstable in some environments, and we priorized stability first. Maybe one of the performance issues is due that 2.5.13 uses some old libraries, and probablly updating them (the same that used 3.0.1) could improve the performance. Currently we plan to update libraries by 1Q2021, after BerlinHF.

If you could provide the performance measuraments you did (including which calls are you doing) it will provide to us an unvaluable information to analize it.

from openethereum.

@adria0 I agree, it could the libraries. For quick comparison, I am attaching logs from both versions running in parallel, cold started at the same time with same configurations. Its pretty clear block import times are about 2-3x larger in the new version. That should be easy to replicate. As for measuring other calls, while I cannot copy our production environment, I am currently writing a test benchmark that captures its main features. I'll share it when it's done.

On a side note, I realize 3.0.1 is not supported anymore, but if you'd make an educated guess, was it pretty much Berlin-ready or were some major features missing?

openethereum_301.txt
openethereum_310.txt

from openethereum.

On a side note, I realize 3.0.1 is not supported anymore, but if you'd make an educated guess, was it pretty much Berlin-ready or were some major features missing?

3.0.1 does not include EIPS like 2929, 2930 or 2565 that have a lot of chances to be in Berlin

from openethereum.

I'm not sure if this is related, the this specific issue, but I'm running 3.1.0 as a virtual machine on 4x 8TB SSDs (RAID10), 32GB of RAM, and a 16 core CPU (Xeon) and at block 9240212 I'm syncing at a rate of something like 0.3 blocks per second...it's terribly slow.

I started syncing a Geth node on the same hardware, like 2 months ago, and the Geth node has been in sync for weeks. I'm looking at another 2 months to reach the CURRENT tip at that sync rate not to mention all the new blocks that will get processed in that time...

from openethereum.

I've been testing a development version with upgraded ethereum-types #253. But, it doesn't make much difference: block verification times are still at least twice as large as in v3.0.1. I'd like to research this further, I've been going through commits from that time, any suggestions on what to focus on?

from openethereum.

Glad to hear that. kvdb-rocksdb upgrades both rocksdb and some configuration, that is maybe the main reason for the boost in performance.

from openethereum.

Hello, I know it s not related but where can I download v3.0.1?

from openethereum.

here is v3.0.1: https://github.com/openethereum/parity-ethereum/releases/tag/v3.0.1

Idea is to merge rocksdb before berlin goes live on mainnet and create v3.2.2-rc1 but not announce it, v3.2.1 is one tested and prepared for berlin and it is okay for most masses, but rc can be used for people that need this upgrade. And proper release can be done shortly after berlin when risks are lower.

from openethereum.