[feature]: Persist imported Mission Control Data across restarts. about lnd HOT 4 OPEN

While we're at it, perhaps it makes sense to also introduce the concept of mission control namespaces? This way you can import some outside weights to an independent namespace, then specify later on if you want to use that or the "default" namespace.

This also has some overlap with this issue to support profiles for the configs as well: #7812. A combo of both would facilitate A/B testing of the various knobs and path finding algos we have.

from lnd.

In the context, one should also check whether the parameters from SetMissionControl can be persisted. Currently, the lnd.conf must also be changed if you want to have the same parameters after the next restart.
In my opinion, parameters that are changed via the API should be persisted. To avoid inconsistencies with the command line flags, these should therefore be removed and the defaults for fresh lnd instances should be hard-coded.

Good point, this is basically the idea of namespaces described by roasbeef in #7812, keeping the default (global) config as in the lnd.conf but being able to create router profiles (tagged via namespaces probably and persited to disc) and then also being able to attach those profiles to the send payment requests.

from lnd.

While we're at it, perhaps it makes sense to also introduce the concept of mission control namespaces? This way you can import some outside weights to an independent namespace, then specify later on if you want to use that or the "default" namespace.

Just to clarify my understanding: So we would add a new field to the MC entry called namespace which binds the entry to the specific router profile for example. So every entry can only have 1 namespace attached to it ? If so this would increase the requirement of the MC data, LND would have to keep in memory, or maybe we would have to redesign how we handle the memory MC data so that we only have the loaded namespace related data loaded into memory otherwise the memory consumption might be harmful for the overall stability of the daemon.

Regarding this topic, @mohamedawnallah was doing some stress testing and loading up to 500_000 MC paris into LND which currently LND would just accept and that would lead to a OOM crash. I think we need to make LND more robust in the course of this PR, meaning that we only load for example most recent data into memory and fetch the other data when needed from the db, so we don't have the whole data set loaded in memory ?
I wonder if we should prioritze this architectural issue, because the current Routing Graph seem to not exceed 110000 MC pairs which LND can still handle (depending on the system obviously).

from lnd.

In the context, one should also check whether the parameters from SetMissionControl can be persisted. Currently, the lnd.conf must also be changed if you want to have the same parameters after the next restart.
In my opinion, parameters that are changed via the API should be persisted. To avoid inconsistencies with the command line flags, these should therefore be removed and the defaults for fresh lnd instances should be hard-coded.

from lnd.