Lightning Terminal Showing red. Still not working after reinstalling package about mynode HOT 22 OPEN

FYI I've now tried deleting the admin.macaroon, readonly.macaroon and invoice.macaroon from /mnt/hdd/mynode/lnd/data/chain/bitcoin (@HamishMacEwan not from the ~/.lnd/ folder, but this folder) and restarting lnd (actually manually stopped lnd.service, deleted those three macaroons and then rebooted entire Rpi) according to the links I posted in #763 (comment) and this fixed the problem for me.

Checking the differences, new in admin.macaroon is macaroon generate read write (was only macaroon generate) and new in readonly.macaroon is an added line maracoon read below the line invoices read.
invoice.macaroon has no readable changes (besides the obvious binary bits that changed).

edit: maybe it was something similar like the following, from https://github.com/lightningnetwork/lnd/blob/master/macaroons/README.md#upgrading-from-v080-beta-or-earlier

Users upgrading from a version prior to v0.9.0-beta might get a permission denied error when trying to use the lncli bakemacaroon command.
This is because the bakery requires a new permission (macaroon/generate) to access.
Users can obtain a new admin.macaroon that contains this permission by removing all three default macaroons (admin.macaroon, invoice.macaroon and readonly.macaroon, NOT the macaroons.db!) from their data/chain/<chain>/<network>/ directory inside the lnd data directory and restarting lnd.

from mynode.

@Eegee Wow, thanks for that link. That may be the root cause for some of these issues. It's been a long time, but pre-0.9.0 lnd was in use on mynode for a while. For others to check if it's a macaroon missing permission issue, you can run this command:

lncli printmacaroon --macaroon_file /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/admin.macaroon

from mynode.

LiT v0.9.0-alpha confirmed working (including lightning node connect) after deleting the admin.macaroon, invoice.macaroon, and readonly.macaroon files from /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/
I verified the information was missing from the macaroons using the printmacaroon command @tehelsper
Thank you @Eegee for linking everything! Extremely quick fix. 👍

Hopefully this helps out @HamishMacEwan and @godSaysHODL to close this issue.
-DJ

from mynode.

The 0.3.16 upgrade and careful revision of my customised lnd.config has resolved the problem. Thanks to all of the encouragement from other sufferers. I did perform the recommended actions and checked the macaroon values, to no avail. It seemed to involve rpcmiddleware settings.

PS.

bitcoin@myNode:~ $ ls -ld .lnd /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/
lrwxrwxrwx 1 bitcoin bitcoin   19 Jun 19  2022 .lnd -> /mnt/hdd/mynode/lnd
drwx--x--x 2 bitcoin bitcoin 4096 May  5 22:16 /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/

from mynode.

I would also reset your custom lnd config and add you changes to the new "additional config" section.

To run commands, try this guide.

https://mynodebtc.github.io/advanced/linux-terminal.html

from mynode.

Update: resetting my config worked

from mynode.

For me as well. Are you on v0.3.14 @godSaysHODL ?
In my situation it might have something to do with permissions as I'm seeing [INF] INVC: Cancelling invoice subscription for client=8 [ERR] RPCS: [/lnrpc.Lightning/RegisterRPCMiddleware]: permission denied [INF] INVC: New invoice subscription client: id=8 in the LND log
and LNDC: Could not receive from interceptor stream: rpc error: code = Unknown desc = permission denied in the Lightning Terminal log.

I have the rpc middleware enabled in [lnd.conf], from commit 12057af (v0.3.14)

How to grant permission for the RPC Middleware and Lightning Terminal?

edit: do I perhaps need to delete admin.macaroon, readonly.macaroon and invoice.macaroon and restart lnd ?
sources: lightninglabs/lightning-terminal#509 and lightninglabs/lightning-terminal#521

from mynode.

@godSaysHODL It looks like you probably have a custom LND that isn't compatible with the latest software versions.

@Eegee I've not seen that error before. Did it just start with the latest version?

from mynode.

I've not seen that error before. Did it just start with the latest version?

Yes, it did. My node is on v0.3.14 and came from v0.3.12.

from mynode.

The error I have since upgrading to v0.3.14 (from v0.3.12) is (in reverse time order):

Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Lnd services finished
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Wait for router to finish
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Wait for invoices to finish
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Wait for chain notifier to finish
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Wait for client to finish
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.809 [DBG] LNDC: Closing lnd connection
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.807 [ERR] LNDC: Could not receive from interceptor stream: rpc error: code = Unknown desc = permission denied
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.808 [ERR] LITD: Could not start subservers: context canceled
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.802 [INF] LITD: Starting LiT middleware manager
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.763 [INF] LITD: Starting LiT account service
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.762 [DBG] LITD: Not resuming session 020ea079c59bbb55b57b1a9b5bca7cb19b687fabd7491729cafcd32e61bba7eac5 with type 3
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.762 [DBG] LITD: Not resuming session 020080985c79f183e027b922a7141b7683bf4799eb13df6b95a53b625ba7bb776f. Could not bake the necessary macaroon: %!w(*status.Error=&{0x40008987e0})
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.748 [DBG] GRPC: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Apr 19 14:43:00 myNode lit[20297]: 2023-04-19 14:43:00.748 [INF] LITD: Starting LiT session server

Everything else is working fine.

If a custom LND configuration is missing a required element for lit.service what might it be?

Currently lnd.config contains:

# This line needs to be uncommented if you have a lnd wallet and commented if you do not
wallet-unlock-password-file=/mnt/hdd/mynode/settings/.lndpw
wallet-unlock-allow-create=true

Which has been the most recommended configuration change for other errors.

Given the new customisation option and the length of time and number of tweaks I've made, perhaps it is time to reset the config… But I would like to understand what the error is and how it arose.

1. Tried the remove macaroons and restart lnd linked above, but despite new macaroons:

bitcoin@myNode:~/.lnd/data/chain/bitcoin/mainnet $ ls -l admin.macaroon readon
ly.macaroon invoice.macaroon
-rw-r----- 1 bitcoin bitcoin 293 Apr 19 15:13 admin.macaroon
-rw-r--r-- 1 bitcoin bitcoin 132 Apr 19 15:13 invoice.macaroon
-rw-r--r-- 1 bitcoin bitcoin 217 Apr 19 15:13 readonly.macaroon
bitcoin@myNode:~/.lnd/data/chain/bitcoin/mainnet $

The problem persists.

from mynode.

I've have the same problem as well.
This started with my attempts to custom upgrade to 0.8.6-alpha. I always reverted back to 0.8.5-alpha and restored capability. After upgrading to myNode 0.3.15 from 0.3.13 I've had the same repeated crashes of LiT. I haven't been able to connect with a pairing phrase for months now. (So I'm certain there's a macaroon issue somewhere)

Manual inspection of the lit.conf all appeared OK.

I also corrected the custom LND conf per the LiT Remote requirements:
tlsextraip=
rpclisten=0.0.0.0:10009
rpcmiddleware.enable=true

LiT still shuts down, here are the log entries with 0.9.0-alpha: (reverse order)

Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.107 [DBG] LNDC: Lnd services finished
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.107 [DBG] LNDC: Wait for router to finish
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.107 [DBG] LNDC: Wait for invoices to finish
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.106 [DBG] LNDC: Wait for chain notifier to finish
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.106 [DBG] LNDC: Wait for client to finish
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.106 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.106 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.106 [DBG] LNDC: Closing lnd connection
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.105 [ERR] LITD: Could not start subservers: context canceled
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.104 [ERR] LNDC: Could not receive from interceptor stream: rpc error: code = Unknown desc = permission denied
Apr 22 16:02:19 myNode lit[15728]: 2023-04-22 16:02:19.102 [INF] LITD: Starting LiT middleware manager
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.980 [DBG] GRPC: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.980 [INF] LITD: Starting LiT account service
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.980 [INF] LITD: Starting LiT session server
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.980 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.980 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
Apr 22 16:02:18 myNode lit[15728]: 2023-04-22 16:02:18.917 [DBG] GRPC: [core] Channel Connectivity change to READY

Also, before clearing the custom bitcoin version, upgrade script could not finish, and the LiT version was listed as "unknown" - attempts to downgrade back to 0.8.5-alpha failed. After clearing the custom bitcoin version, "unknown" went away and now reports 0.9.0-alpha.

I will keep experimenting; hopefully will get something working. Considering just completely reinstalling myNode from a fresh SD card image.

-DJ

from mynode.

I have restored functionality by reverting to 0.8.5-alpha, but I'm sure this is temporary.
Upgrading to 0.8.6-alpha caused it to fail again. And I'm still unable to "connect" to lightning terminal sessions due to rpc failures.

I can force litd to work (0.8.6) and (0.9.0) using the following command line:
sudo -u bitcoin litd --configfile=/mnt/hdd/mynode/lit/lit.conf --rpcmiddleware.disabled

However, any changes made to the lit.conf file are being reverted by mynode periodically; so this doesn't resolve the service issue.

from mynode.

Does LiT actually work when rpcmiddleware has been disabled? I was under the impression it was necessary to connect to the remote lnd instance.

Also, when LiT was running into the above error, was LND up and running properly? I don't fully understand the log messages, but it appears to be some sort of communication issue between lit and lnd.

The "unknown" version means it did not install and if you had a custom bitcoin version failing first, that would explain the unknown version.

from mynode.

I entirely disabled LiT last week to get 'rid' of the red dot, but I just tried your suggestion.

Does LiT actually work when rpcmiddleware has been disabled?

To try this I enabled it again, but first changed rpcmiddleware.enable to false in lnd.conf (thereby making it a custom LND config), rebooted and waited for LND to properly run.
Unfortunately I'm still getting RPCS: [/lnrpc.Lightning/RegisterRPCMiddleware]: permission denied in the LND log and [ERR] LNDC: Could not receive from interceptor stream: rpc error: code = Unknown desc = permission denied in the Terminal log.
So as far as I can tell disabling rpcmiddleware didn't make a difference for me.
I haven't done anything with the macaroons yet btw.

from mynode.

Does LiT actually work when rpcmiddleware has been disabled? I was under the impression it was necessary to connect to the remote lnd instance.

Yes LiT works using the lit.conf flag "rpcmiddleware.disabled=true" or "--rpcmiddleware.disabled" from the command line... At least enough to operate pool, loop, and channel services through the Lightning Terminal interface.

When disabled, I do not have any grpc access denied messages. Thanks for the follow up.
I went back to LiT v0.8.5-alpha for now.

-DJ

from mynode.

@MWDJ Have you been running your node since 2019 or early 2020? The interesting thing about that bug/fix is that it should have only occurred on very old wallets. Glad it's working!

from mynode.

Early 2020.... so agree a GUI method to manage, clean up, and regenerate macaroons would be a great feature!
-DJ

from mynode.

My macaroon files were from June 1st 2020.

edit: I myself did not delete the macaroons.db file, as it was stressed in the README not to do that.

from mynode.

Full management would be a large undertaking, but perhaps I could add a regen macaroon option on the settings page. Did you only delete the macaroon files themselves or the macaroons.db file as well?

from mynode.

@tehelsper just the 3x macaroon files in the lnd mainnet directory.
"admin.macaroon, invoice.macaroon, and readonly.macaroon files from /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/"
When LND restarts it regenerates the proper macaroon files with updated content.
I copied the old files just in case, but it worked immediately after a restart of all the related services.

from mynode.

PS.

bitcoin@myNode:~ $ ls -ld .lnd /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/
lrwxrwxrwx 1 bitcoin bitcoin   19 Jun 19  2022 .lnd -> /mnt/hdd/mynode/lnd
drwx--x--x 2 bitcoin bitcoin 4096 May  5 22:16 /mnt/hdd/mynode/lnd/data/chain/bitcoin/mainnet/

Ah sorry, my bad, I hadn't noticed earlier you showed the path for the bitcoin user. 👍
for user admin it's a local folder.

from mynode.

is there a GUI button to implement the macaroon fix yet? I've updated to 0.3.16 and am reading through this thread, still not sure what to do. Not used to finding and deleting things in linux file structure / terminal. I'm not using a custom LND version, but I do have a custom LND-config, so it is likely my config, and maybe the macaroon issue

from mynode.