[urgent] Update LND and electrs due to a critical bug as soon as the fix is available about mynode HOT 29 OPEN

A new release with the fix is out now. It has:

• LND v0.15.4
• Electrs v0.9.9 for arm64 and amd64
• Better sig handling for LND allowing better app customization if this happens again and users need to upgrade early

from mynode.

I think(hope) tehelsper is working hard for the update.

from mynode.

No, it's v0.3.04. You may need to click Check for Updates.

Downloading now

from mynode.

Looks like the electrs upgrade was only for 64bit and not for 32bit os?

from mynode.

This may work as well and would be a lot faster. If it works, I can do an easy fix that re-uses the 64-bit image for RP4 on all aarch64 platforms.

sudo systemctl stop electrs
sudo wget https://github.com/mynodebtc/mynode/raw/master/rootfs/raspi4/usr/bin/electrs_arm64 -O /usr/bin/electrs
sudo systemctl restart electrs

from mynode.

Took your second approach and it works fine. Seems like the RP 64 bin can be reused. Thx for the help all green:

from mynode.

Electrum Server Status shows this error in logs:

Nov 01 07:11:46 mynoder systemd[1]: electrs.service: Failed with result 'exit-code'.
Nov 01 07:11:46 mynoder systemd[1]: electrs.service: Main process exited, code=exited, status=1/FAILURE
Nov 01 07:11:46 mynoder electrs[26807]: 2: receiving on an empty and disconnected channel
Nov 01 07:11:46 mynoder electrs[26807]: 1: failed to get block 000000000000000000070742427fa10ec3c66d006160155a704d9f56d090a3ea
Nov 01 07:11:46 mynoder electrs[26807]: 0: sync failed
Nov 01 07:11:46 mynoder electrs[26807]: Caused by:
Nov 01 07:11:46 mynoder electrs[26807]: Error: electrs failed
Nov 01 07:11:46 mynoder electrs[26807]: [2022-11-01T12:11:46.027Z INFO electrs::db] closing DB at /mnt/hdd/mynode/electrs/bitcoin
Nov 01 07:11:46 mynoder electrs[26807]: note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Nov 01 07:11:46 mynoder electrs[26807]: thread 'p2p_recv' panicked at 'invalid message: OversizedVectorAllocation { requested: 12000072, max: 4000000 }', src/p2p.rs:211:62
Nov 01 07:11:46 mynoder electrs[26807]: [2022-11-01T12:11:46.002Z INFO electrs::index] indexing 13 blocks: [761249..761261]
Nov 01 07:11:45 mynoder electrs[26807]: [2022-11-01T12:11:45.951Z INFO electrs::chain] chain updated: tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f, height=761248
Nov 01 07:11:40 mynoder electrs[26807]: [2022-11-01T12:11:40.901Z INFO electrs::chain] loading 761248 headers, tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f
Nov 01 07:11:35 mynoder electrs[26807]: [2022-11-01T12:11:35.996Z INFO electrs::db] "/mnt/hdd/mynode/electrs/bitcoin": 158 SST files, 37.581945005 GB, 4.696187476 Grows
Nov 01 07:11:35 mynoder electrs[26807]: [2022-11-01T12:11:35.869Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
Nov 01 07:11:35 mynoder electrs[26807]: [2022-11-01T12:11:35.869Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
Nov 01 07:11:35 mynoder electrs[26807]: Starting electrs 0.9.3 on aarch64 linux with Config { network: Bitcoin, db_path: "/mnt/hdd/mynode/electrs/bitcoin", daemon_dir: "/home/bitcoin/.bitcoin", daemon_auth: CookieFile("/home/bitcoin/.bitcoin/.cookie"), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 127.0.0.1:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 15s, jsonrpc_timeout: 30s, index_batch_size: 10, index_lookup_limit: Some(200), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.9.3 (Electrum Rust Server)!", args: [] }
Nov 01 07:11:35 mynoder systemd[1]: Started myNode Electrum Server.

from mynode.

CC: @tehelsper

Just FYI: electrs v0.9.9 seems unaffected.

from mynode.

The lnd with an emergency hotfix was just released:

• v0.15.4: https://github.com/lightningnetwork/lnd/releases/tag/v0.15.4-beta

from mynode.

Any status update on this?

from mynode.

I'm having the same issue, was wondering why my LND was stuck in "waiting" and not synced with the blockchain. Will disabling the electrum wallet app resolve this?

from mynode.

No, just wait for update or vendor work-a-round instruction here. IMHO

from mynode.

A new release with the fix is out now. It has:

• LND v0.15.4
• Electrs v0.9.9 for arm64 and amd64
• Better sig handling for LND allowing better app customization if this happens again and users need to upgrade early

Awesome thank you!

Would this be beta_10-27-22_1666923526?

from mynode.

No, it's v0.3.04. You may need to click Check for Updates.

from mynode.

I've upgraded to 0.3.04 from 0.3.03, but it seems my electrs didn't get upgraded. I can see I'm on version v0.9.9 on the "Manage applications" page, but in the electrs logs I see: "Starting electrs 0.9.3 on arm linux with Config".
And the issue remains, it doesn't start successfully and then just keeps restarting.

from mynode.

Yes, the electrs binary was only made for amd64 and aarch64. If you are on the old 32-bit Raspberry Pi 4 OS, you can resolve the issue by re-flashing to the 64-bit image or by building electrs image yourself.

sudo mynode-install-extra electrs

The 32-bit binary will be updated in a future release.

from mynode.

The best solution is to upgrade to 64-bit to avoid future issues.

from mynode.

On pine64 (RockPro64) it updated electrs to v0.9.9 but when the server tries to start it crashes with following error:

myNode systemd[1]: electrs.service: Failed with result 'exit-code'.
myNode systemd[1]: electrs.service: Main process exited, code=exited, status=1/FAILURE
myNode electrs[21400]: /usr/bin/electrs: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/bin/electrs)
myNode systemd[1]: Started myNode Electrum Server.

...

from mynode.

Oh, I think I may have unintentionally upgraded the RockPro64 image from Buster to Bullseye during the last image creation which has a different GLIBC version. Can you run these commands and compare?

admin@myNode:~$ ldd --version
ldd (Debian GLIBC 2.31-13+deb11u5) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

admin@myNode:~$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
admin@myNode:~$ 

from mynode.

Nope seems to be buster:

admin@myNode:~$ ldd --version
ldd (Debian GLIBC 2.28-10+deb10u2) 2.28
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

admin@myNode:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Screenshots to get a picture:

from mynode.

Updated 3.04 and still LND waiting.... Electrum stuck syncing at 99.96%

from mynode.

I just uploaded a new beta that should have fixes for Electrs on arm32 and aarch64.

from mynode.

Really appreciating your effort but I just updated to 3.05:

The electrs issue for rockpro64 is still there. Just for clarity everything else is working just fine:

from mynode.

Same issue here. I'm also on 0.3.05 on rockpro64

from mynode.

Upgraded to 3.05, LND error: [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available

from mynode.

OK, @tompro and @ForteUnited, can you try running:

sudo mynode-install-extra rust
sudo mynode-install-extra electrs

That should get it working, at least until the next update which will overwrite electrs again. I built the latest electrs for aarch in docker for debian-buster but it must have still used a new glibc since I was running bullseye on that machine.

from mynode.