Git Product home page Git Product logo

rebalance-lnd's Introduction

rebalance-lnd

Using this script you can easily rebalance individual channels of your lnd node by sending funds out one channel, through the lightning network, back to yourself.

This script helps you move funds between your channels so that you can increase the outbound liquidity in one channel, while decreasing the outbound liquidity in another channel. This way you can, for example, make sure that all of your channels have enough outbound liquidity to send/route transactions.

Because you are both the sender and the receiver of the corresponding transactions, you only have to pay for routing fees. Aside from paid fees, your total liquidity does not change.

Installation

There are several options for installing rebalance-lnd.

  1. Using Python
  2. Using Docker
  3. Using Umbrel's app store
  4. Using the RaspiBolt manual installation guide on any Debian-based OS

Using Python

lnd

This script needs an active lnd (tested with v0.17.5, https://github.com/lightningnetwork/lnd) instance running. If you compile lnd yourself, you need to include the routerrpc build tag:

Example: make tags="autopilotrpc signrpc walletrpc chainrpc invoicesrpc routerrpc"

You need to have admin rights to control this node. By default, this script connects to localhost:10009, using the macaroon file in ~/.lnd/data/chain/bitcoin/mainnet/admin.macaroon. If this does not help, it also tries to find the file in ~/umbrel/lnd/data/chain/bitcoin/mainnet/admin.macaroon (or ~/umbrel/app-data/lightning/data/lnd/data/chain/bitcoin/mainnet/admin.macaroon). If you need to change this, please have a look at the optional arguments --grpc and --lnddir.

rebalance-lnd itself

You need to download the files that are part of this project, for example using git:

cd /some/where/
git clone https://github.com/C-Otto/rebalance-lnd.git
cd rebalance-lnd/

Alternatively, you may also download the files in a ZIP file offered by GitHub: https://github.com/C-Otto/rebalance-lnd/archive/refs/heads/main.zip

Python Dependencies

You need to install Python 3. You also need to install the gRPC dependencies which can be done by running:

pip install -r requirements.txt

If this fails, make sure you are running Python 3. You might want to try pip3 instead of pip.

To test if your installation works, you can run rebalance.py without any arguments. Depending on your system, you can do this in one of the following ways:

  • python3 rebalance.py
  • ./rebalance.py
  • python rebalance.py

Using Docker

Using the containerized version of rebalance-lnd spares you from the installation of python and its dependencies. You start by fetching the latest version of the dedicated docker container.

docker pull rebalancelnd/rebalance-lnd:latest

You can now have the docker image interact with your lnd installation:

docker run --rm --network=host --add-host=host.docker.internal:host-gateway -it -v /home/lnd:/root/.lnd rebalancelnd/rebalance-lnd --grpc host.docker.internal:10009

The above command assumes /home/lnd is your lnd configuration directory. Please adjust as required.

Note for Umbrel/Umbrel-OS users

To inject rebalance-lnd into your umbrel network you can run it using the following command line:

docker run --rm --network=umbrel_main_network -it -v /home/umbrel/umbrel/app-data/lightning/data/lnd:/root/.lnd rebalancelnd/rebalance-lnd --grpc 10.21.21.9:10009

Optionally you can create an alias in your shell's environment file like so:

alias rebalance-lnd="docker run --rm --network=umbrel_main_network -it -v /home/umbrel/umbrel/app-data/lightning/data/lnd:/root/.lnd rebalancelnd/rebalance-lnd --grpc 10.21.21.9:10009"

For older versions of Umbrel please use /home/umbrel/umbrel/lnd instead of /home/umbrel/umbrel/app-data/lightning/data/lnd.

Using Umbrel's app store

The lightning-shell app available in the Umbrel app store comes with rebalance-lnd installed and configured. It should just work out of the box!

Note for BTCPayServer Users

To inject rebalance-lnd into your BTCPayServer network you can run it using the following command line:

docker run --rm --network=generated_default -it -v /var/lib/docker/volumes/generated_lnd_bitcoin_datadir/_data:/root/.lnd rebalancelnd/rebalance-lnd --grpc lnd_bitcoin:10009

Optionally you can create an alias in your shell's environment file like so:

alias rebalance-lnd="docker run --rm --network=generated_default -it -v /var/lib/docker/volumes/generated_lnd_bitcoin_datadir/_data:/root/.lnd rebalancelnd/rebalance-lnd --grpc lnd_bitcoin:10009"

Updating

If you use docker, update the image running docker pull rebalancelnd/rebalance-lnd:latest again, otherwise follow these steps to update the python version:

If you already have a version of rebalance-lnd checked out via git, you can just use git pull to update to the latest version. You may also delete everything and start over with a fresh installation, as the script does not store any data that needs to be kept.

Do not forget to update the Python dependencies as described above.

Using the RaspiBolt guide

If you run a node on a Debian-based OS, you can follow the RaspiBolt guide that explains how to manually install, use, update and uninstall rebalance-lnd on your node.

Usage

List of channels

Run rebalance.py -l (or rebalance.py -l -i) to see a list of channels which can be rebalanced. This list only contains channels where you should increase the outbound liquidity (you can specify --show-all to see all channels).

You can also see the list of channels where the inbound liquidity should be increased by running rebalance.py -l -o.

As an example the following indicates a channel with around 17.7% of the funds on the local side:

Channel ID:       11111111
Alias:            The Best Node Ever
Pubkey:           012345[...]abcdef
Channel Point:    abc0123[...]abc:0
Local ratio:      0.176
Fee rates:        123ppm (own), 456ppm (peer)
Capacity:         5,000,000
Remote available: 4,110,320
Local available:    883,364
Rebalance amount:   116,636
[█████░░░░░░░░░░░░░░░░░░░░░░░]

By sending 116,636 satoshis to yourself using this channel, an outbound liquidity of 1,000,000 satoshis can be achieved. This number is shown as "Rebalance amount" (where negative amounts indicate that you need to increase your inbound liquidity).

The last line shows a graphical representation of the channel. The total width is determined by the channel's capacity, where your largest channel (maximum capacity) occupies the full width of your terminal. The bar () indicates the funds on the local side of the channel, i.e. your outbound liquidity.

Rebalancing a channel

Basic Scenario

In this scenario you already know what you want to do, possibly because you identified a channel with high outbound liquidity.

Let us assume you want to move some funds from this channel to another channel with low outbound liquidity:

  • move 100,000 satoshis around
  • take those funds (plus fees) out of channel 11111111
  • move those funds back into channel 22222222

You can achieve this by running:

rebalance.py --amount 100000 --from 11111111 --to 22222222

The script now tries to find a route through the lightning network that starts with channel 11111111 and ends with channel 22222222. If successful, you take 100,000 satoshis (plus fees) out of channel 11111111 (which means that you decrease your outbound liquidity and increase your inbound liquidity). In return, you get 100,000 satoshis back through channel 22222222 (which means that you increase your outbound liquidity and decrease your inbound liquidity).

Automatically determined amount

If you do not specify the amount, i.e. you invoke rebalance.py --from 11111111 --to 22222222, the script automatically determines the amount. For this two main constraints are taken into consideration:

After the rebalance transaction is finished,

  • the destination channel (22222222) should have (up to) 1,000,000 satoshis outbound liquidity
  • the source channel (11111111) should have (at least) 1,000,000 satoshis outbound liquidity

If the source channel has enough surplus outbound liquidity, the script constructs a transaction that ensures 1,000,000 satoshis of outbound liquidity in the destination channel. However, if the source channel does not have enough outbound liquidity, the amount is determined so that (after the rebalance transaction is performed) the source channel has 1,000,000 satoshis of outbound liquidity and the destination channel has more outbound liquidity than before (but not necessarily 1,000,000 satoshis).

Note that for smaller channels these criteria cannot be met. If that is the case, a ratio of 50% is targeted instead: the destination channel receives up to 50% outbound liquidity, and for the sending channel at least 50% outbound liquidity are maintained.

Limiting the automatically determined amount

If you use both -a to specify an amount and -A (shorthand for --adjust-amount-to-limits), the computed amount is adjusted to the given amount. If, for example, you send funds to a channel that lacks 500,000sat to reach the --min-local value, the computed amount is 500,000sat. If you invoke the script with -A -a 90000 this amount is reduced to 90,000sat. If the adjusted amount is below the --min-amount setting, the script stops.

This way, using both options (-a xxx -A) you can start a rebalance attempt with the given amount, which will only be attempted if it is necessary. Thus, you may want to run ./rebalance.py -t xxx -A -a 50000 in a loop or cron job to automatically send up to 50,000sat to channel xxx until it has reached the --min-local limit. If the channel already satisfies the --min-local limit, the script exits and does not attempt to send any funds.

Only specifying one channel

Instead of specifying both --from and --to, you can also just pick one of those options. The script then considers all of your other channels as possible "partners", still taking into account the constraints described above.

As an example, if you run rebalance.py --amount 100000 --to 22222222, the script tries to find source channels that, after sending 100,000 satoshis (plus fees), still have at least 1,000,000 satoshis of outbound liquidity. In other words, the script does not send funds from already "empty" channels.

Likewise, when only specifying --from, the script only considers target channels which still have at least 1,000,000 satoshis of outbound liquidity after receiving the payment.

If you also let the script determine the amount, e.g. you run rebalance.py --to 22222222, the script first computes the amount that is necessary to reach 1,000,000 satoshis of outbound liquidity in channel 22222222, and then tries to find source channels for the computed amount.

Safety Checks and Limitations

Note that, by default, nothing is done if the amount (either given or computed) is smaller than 10,000 satoshis. You can change this number using --min-amount.

Furthermore, a rebalance transaction is only sent if it is economically viable as described below. This way, by default, you only send rebalance transactions that improve your node's situation, for example by providing outbound liquidity to channels where you charge a lot, and taking those funds out of channels where you charge less.

To protect you from making mistakes, in the fee computation (described below) the fee rate of the destination channel is capped at 2,000ppm (which corresponds to a 0.2% fee), even if the channel is configured with a higher fee rate.

Fees

In order for the network to route your rebalance transaction, you have to pay fees. In addition to the fees charged by the nodes routing your transaction, two other values are taken into account:

  1. The fee you would earn if, instead of sending the funds out of the source channel as part of the rebalance transaction, your node is paid to forward the amount
  2. The fee you would earn if, after the rebalance transaction is done, your node forwards the amount through the destination channel

The first bullet point describes opportunity/implicit costs. If you take lots of funds out of the source channel, you cannot use those funds to earn routing fees via that channel. As such, taking funds out of a channel where you configured a high fee rate can be considered costly.

The second bullet points describes future earnings. If you send funds to the destination channel and increase your outbound liquidity in that channel, you might earn fees for routing those funds. As such, increasing the outbound liquidity of the destination is a good idea, assuming the fee rate configured for the destination channel is higher than the fee rate you configured for the source channel. However, keep in mind that you will only earn those fees if your node actually forwards the funds via the channel!

The rebalance transaction is not performed if the transaction fees plus the implicit costs (1) are higher than the possible future earnings (2).

If you really want to, you may disable these safety checks with --reckless.

Example

You have lots of funds in channel 11111111 and nothing in channel 22222222. You would like to send funds through channel 11111111 (source channel) through the lightning network, and finally back into channel 22222222. This incurs a transaction fee you would have to pay for the transaction.

Furthermore, if in the future there is demand for your node to route funds through channel 11111111, you cannot do that as much (because you decreased your outbound liquidity in the channel). The associated fees are the implicit cost (1).

Finally, you send funds into channel 22222222 in the hope that later on someone requests your node to forward funds from your own node through channel 22222222 towards the peer at the other end, so that you can earn fees for this. These fees are the possible future income (2).

Fee Factor

The value set with --fee-factor is used to scale the future income used in the computation outlined above. As such, you can fool the script into believing that the fee rate configured for the destination channel is higher (fee factor > 1) or lower (fee factor < 1) than it actually is.

As such, if you set --fee-factor to a value higher than 1, more routes are considered. As an example, with --fee-factor 1.5 you can include routes that cost up to 150% of the future income.

With values smaller than 1 only cheaper routes are considered.

Fee Limit

Unrelated to --fee-factor (which is the default, with a value of 1), you can also specify an absolute fee limit using --fee-limit. If you decide to do so, only routes that cost up to the given number (in satoshis) are considered.

Note that the script rejects routes/channels that are deemed uneconomical based on the configured fee rates (i.e. with --fee-factor set to 1) (as explained above).

Fee Rate (ppm) Limit

You can use --fee-ppm-limit as another alternative to specify a fee limit. In this case the amount sent as part of the rebalance is considered, so that the fee is at most

amount * fee-ppm-limit / 1_000_000

Note that the script rejects routes/channels that are deemed uneconomical based on the configured fee rates (i.e. with --fee-factor set to 1) (as explained above).

Warning

To determine the future income, the fee rate you configured for the destination channel is used in the computation. As such, if you set an unrealistic fee rate that will not lead to forward transactions, you would allow more expensive rebalance transactions without earning anything in return. Please make sure to set realistic fee rates, which at best are already known to attract forwardings.

Command line arguments

usage: rebalance.py [-h] [--lnddir LNDDIR] [--network NETWORK] [--grpc GRPC]
                    [-l] [--show-all | --show-only CHANNEL | -c] [-o | -i]
                    [-f CHANNEL] [-t CHANNEL] [-A] [-a AMOUNT | -p PERCENTAGE]
                    [--min-amount MIN_AMOUNT] [--min-local MIN_LOCAL]
                    [--min-remote MIN_REMOTE] [-e EXCLUDE] [--reckless]
                    [--ignore-missed-fee] [--fee-factor FEE_FACTOR]
                    [--fee-limit FEE_LIMIT | --fee-ppm-limit FEE_PPM_LIMIT]

optional arguments:
  -h, --help            show this help message and exit
  --lnddir LNDDIR       (default ~/.lnd) lnd directory
  --network NETWORK     (default mainnet) lnd network (mainnet, testnet,
                        simnet, ...)
  --grpc GRPC           (default localhost:10009) lnd gRPC endpoint

list candidates:
  Show the unbalanced channels.

  -l, --list-candidates
                        list candidate channels for rebalance
  --show-all            also show channels with zero rebalance amount
  --show-only CHANNEL   only show information about the given channel
  -c, --compact         Shows a compact list of all channels, one per line
                        including ID, inbound/outbound liquidity, and alias
  -o, --outgoing        lists channels with less than 1,000,000 (--min-remote)
                        satoshis inbound liquidity
  -i, --incoming        (default) lists channels with less than 1,000,000
                        (--min-local) satoshis outbound liquidity

rebalance:
  Rebalance a channel. You need to specify at least the 'from' channel (-f)
  or the 'to' channel (-t).

  -f CHANNEL, --from CHANNEL
                        Channel ID of the outgoing channel (funds will be
                        taken from this channel). You may also specify the ID
                        using the colon notation (12345:12:1), or the x
                        notation (12345x12x1). You may also use -1 to choose a
                        random candidate.
  -t CHANNEL, --to CHANNEL
                        Channel ID of the incoming channel (funds will be sent
                        to this channel). You may also specify the ID using
                        the colon notation (12345:12:1), or the x notation
                        (12345x12x1). You may also use -1 to choose a random
                        candidate.
  -A, --adjust-amount-to-limits
                        If set, adjust the amount to the limits (--min-local
                        and --min-remote). The script will exit if the
                        adjusted amount is below the --min-amount threshold.
                        As such, this switch can be used if you do NOT want to
                        rebalance if the channel is within the limits.
  -a AMOUNT, --amount AMOUNT
                        Amount of the rebalance, in satoshis. If not
                        specified, the amount computed for a perfect rebalance
                        will be used
  -p PERCENTAGE, --percentage PERCENTAGE
                        Set the amount to a percentage of the computed amount.
                        As an example, if this is set to 50, half of the
                        computed amount will be used. See --amount.
  --min-amount MIN_AMOUNT
                        (Default: 10,000) If the given or computed rebalance
                        amount is below this limit, nothing is done.
  --min-local MIN_LOCAL
                        (Default: 1,000,000) Ensure that the channels have at
                        least this amount as outbound liquidity.
  --min-remote MIN_REMOTE
                        (Default: 1,000,000) Ensure that the channels have at
                        least this amount as inbound liquidity.
  -e EXCLUDE, --exclude EXCLUDE
                        Exclude the given channel. Can be used multiple times.
  --exclude-private     Exclude private channels. This will not affect channel
                        ID used at --to and/or --from but will take effect if
                        you used -1 to get a random channel.
  --reckless            Allow rebalance transactions that are not economically
                        viable. You might also want to set --min-local 0 and
                        --min-remote 0. If set, you also need to set --amount
                        and either --fee-limit or --fee-ppm-limit, and you
                        must not enable --adjust-amount-to-limits (-A).
  --ignore-missed-fee   Ignore missed fee from source channel.
  --fee-factor FEE_FACTOR
                        (default: 1.0) Compare the costs against the expected
                        income, scaled by this factor. As an example, with
                        --fee-factor 1.5, routes that cost at most 150% of the
                        expected earnings are tried. Use values smaller than
                        1.0 to restrict routes to only consider those earning
                        more/costing less. This factor is ignored with
                        --reckless.
  --fee-limit FEE_LIMIT
                        If set, only consider rebalance transactions that cost
                        up to the given number of satoshis.
  --fee-ppm-limit FEE_PPM_LIMIT
                        If set, only consider rebalance transactions that cost
                        up to the given number of satoshis per 1M satoshis
                        sent.

Contributing

Contributions are highly welcome! Feel free to submit issues and pull requests on https://github.com/C-Otto/rebalance-lnd/

You can also send donations via keysend. For example, to send 500 satoshis to C-Otto with a message "Thank you for rebalance-lnd":

lncli sendpayment --amt=500 --data 7629168=5468616e6b20796f7520666f7220726562616c616e63652d6c6e64 --keysend --dest=027ce055380348d7812d2ae7745701c9f93e70c1adeb2657f053f91df4f2843c71

You can also specify an arbitrary message:

lncli sendpayment --amt=500 --data 7629168=$(echo -n "your message here" | xxd -pu -c 10000) --keysend --dest=027ce055380348d7812d2ae7745701c9f93e70c1adeb2657f053f91df4f2843c71

rebalance-lnd's People

Contributors

achristianson avatar adriansmares avatar aphex3k avatar blckbx avatar c-otto avatar caheredia avatar conscott avatar dependabot-preview[bot] avatar dependabot[bot] avatar gcaracuel avatar glencooper avatar ibz avatar lukedevj avatar manreo avatar realspencerdupre avatar stanislavkozlovski avatar styco avatar vajraofindra avatar vindard avatar wamde avatar zapomatic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rebalance-lnd's Issues

Feature Request: Show interesting channels (statistics)

I'd like to show usage information about channels, possibly based on lncli fwdinghistory. One output could be "Channel X had 123 incoming and 456 outgoing transactions in the last 24 hours, with a net change of -789 Satoshi.". Together with the information of rebalancy.py -l (-i/-o) this could help identify the channels worthy of a rebalance.

Error when trying to rebalance

Hi,

I'm trying to use the latest rebalance-lnd on master, with lnd lnd version 0.7.0-beta commit=v0.7.0-beta-rc1-9-gd6d87e12feeb197f599a329c079ecef17a4f2b4a.

I keep getting the following error about

~/scripts/rebalance-lnd (master ✘)✭ ᐅ ./rebalance.py -t 2 -a 10000
Sending 10,000 satoshis to rebalance to channel with ID 628199371496423424

Trying route #1
<id1> -> <id2> -> <id3> -> <id4>
Traceback (most recent call last):
  File "./rebalance.py", line 227, in <module>
    main()
  File "./rebalance.py", line 63, in main
    Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, excluded, max_fee_factor).rebalance()
  File "/home/admin/scripts/rebalance-lnd/logic.py", line 55, in rebalance
    response = self.lnd.send_payment(payment_request, route)
  File "/home/admin/scripts/rebalance-lnd/lnd.py", line 111, in send_payment
    return self.router_stub.SendToRoute(request)
  File "/home/admin/.local/lib/python2.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/admin/.local/lib/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNIMPLEMENTED
	details = "unknown service routerrpc.Router"
	debug_error_string = "{"created":"@1560903041.474343437","description":"Error received from peer ipv4:127.0.0.1:10009","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"unknown service routerrpc.Router","grpc_status":12}"

Could you let me know what I'm doing wrong? Thanks!

Specify a different lnd directory

The script should have an option on the command line to specify a different location to the .lnd directory. I don't have it in my home folder, so currently the only way I can use the script is to edit lnd.py and change the hardcoded directory.

Improve routing efficiency by utilizing source_pub_key argument in QueryRoutes

If the FROM channel is specified on the command line, the script should probably call GetRoutes while specifying the source_pub_key argument as the first node in the route (the node on the other end of the FROM channel). Currently, GetRoutes tries to find N routes indiscriminately and the check to match the FROM channel happens after routes are returned. This is inefficient and thus it takes longer to find the suitable route and could result in higher chances of routing failure.

I observed that often the 15/30/.. routes returned by the RPC call do not match the FROM channel and the rebalancing fails for that reason

Does not work with python 3: AttributeError: 'str' object has no attribute 'decode'

Using Python 3.6.8 on Ubuntu 18.04, latest code from master, lnd version 0.7.0-beta commit=v0.7.0-beta-rc3 installed with routerrpc, latest requirements.txt installed with pip. I'm trying to rebalance a channel with the following command and getting an error.

$ python rebalance.py -f 627916797039411200 -t 630216975379267584 -a 500000 --max-fee-factor 3
Sending 500,000 satoshis to rebalance to channel with ID 630216975379267584
Forced first channel has ID 627916797039411200

Trying route #1
627916797039411200 -> 623648492953534464 -> 631089987556999169 -> 630216975379267584
Traceback (most recent call last):
  File "rebalance.py", line 227, in <module>
    main()
  File "rebalance.py", line 63, in main
    Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, excluded, max_fee_factor).rebalance()
  File "/home/bitcoin/rebalance-lnd/logic.py", line 54, in rebalance
    response = self.lnd.send_payment(payment_request, route)
  File "/home/bitcoin/rebalance-lnd/lnd.py", line 110, in send_payment
    request.payment_hash = self.hex_string_to_bytes(payment_request.payment_hash)
  File "/home/bitcoin/rebalance-lnd/lnd.py", line 115, in hex_string_to_bytes
    return hex_string.decode("hex")
AttributeError: 'str' object has no attribute 'decode'

Daemon mode

Add a daemon mode where the tool auto rebalances every x hours or whenever an imbalance worth correcting presents itself (what this exactly means TBD)

unable to send, no routes provided (but route is there)

Hello,

I have:

  • rebalance-lnd
  • LND: 0.6.1-beta

list of channels works... finding routes works... but when i try to pay the route i get the following error:

$ python rebalance.py -t 17 -a 1000
Sending 1,000 satoshis to rebalance to channel with ID 63xxxxxxxxxxxxxxxx
requesting 15 routes from lnd, please wait.
lnd returned 1 routes that will now be tested
Trying route #1
63xxxxxxxxxxxxxxxx -> 62xxxxxxxxxxxxxxxx -> 63xxxxxxxxxxxxxxxx
Traceback (most recent call last):
File "rebalance.py", line 223, in
main()
File "rebalance.py", line 62, in main
response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
File "/usr/local/src/rebalance-lnd/logic.py", line 48, in rebalance
response = self.lnd.send_payment(payment_request, [route])
File "/usr/local/src/rebalance-lnd/lnd.py", line 98, in send_payment
return self.stub.SendToRouteSync(request)
File "/usr/lib64/python2.7/site-packages/grpc/_channel.py", line 565, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/lib64/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "unable to send, no routes provided"
debug_error_string = "{"created":"@1559085040.643817147","description":"Error received from peer ipv4:127.0.0.1:10009","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"unable to send, no routes provided","grpc_status":2}"

is this issue with LND or rebalance-lnd? any ideas?

Possible failed routing because last leg is private

Running ./rebalance.py -t 6xxxxxxx -a 100000 fails with the message:

Checking validity route:
6xxxxx -> 6xxxxx -> 6xxxxx
Total fees (msat): 100
Local/Remote/ratio: 782306 100345 0.886 (0.500)
Trying route #1
6xxxxx -> 6xxxxx -> 6xxxxx
Error: unable to route payment to destination: FeeInsufficient(htlc_amt==100000000 mSAT, update=(lnwire.ChannelUpdate) {
 Signature: (lnwire.Sig) (len=64 cap=64) {
....
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 0000000000xxxx,
 ShortChannelID: (lnwire.ShortChannelID) 5xxxxx:xxxx:1,
 Timestamp: (uint32) 1554661864,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 30,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 600000 mSAT,
 BaseFee: (uint32) 0,
 FeeRate: (uint32) 800,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 1999800000 mSAT,
 ExtraOpaqueData: ([]uint8) <nil>
}

It looks like the total routing cost returned by QueryRoutesRequest underestimates the actual final routing fees. It is possible that the reason for that is that the last leg of that particular route is private. The Total fees (msat) in this case show 100 msat, which is exactly the fees to route over the second leg in my case. The fee to route to the third (and final) leg should be (an additional) 80000 msat (based on 800 feerate as shown in the error message and 100K rebalancing amount).

The way to reproduce this is to request a private channel with lnbig.com with some balance on their side and try to rebalance to that channel.

Possible solution could be (1) checking if the last leg is private (2) if yes, query the other party's fees (3) add that extra fee to the routing attempt.

Not sure how you can query their fees using RPC, but on the command line it's lncli getchaninfo.

Provide verbose logging, cleanup current default log messages

The script should provide only necessary information in the default setting, with more helpful (debug) information using -v (and possibly -vv etc.). The use of print to STDOUT/STDERR should be cleaned up as part of this, possibly involving a proper logging framework.

Make `low_local_ratio_after_sending` regard fees

Currently, the first channel might be emptied to less than 50% local capacity due to fees. This is not intentional, and should be fixed. The logic in low_local_ratio_after_sending might be a good start to fix this.

Optimization: Add Chan ID to return

FYI: LND can have more than one channel between nodes, so returning only "Node Key" isn't sufficient. The chan ID should be returned, and used as well.

Crash: channel not found

Sometimes this happens, I have no idea why:

Traceback (most recent call last):
File “./rebalance.py”, line 223, in 
main()
File “./rebalance.py”, line 62, in main
response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
File “/home/cotto/rebalance/logic.py”, line 41, in rebalance
if self.route_is_invalid(route):
File “/home/cotto/rebalance/logic.py”, line 71, in route_is_invalid
if self.low_local_ratio_after_sending(first_hop, route.total_amt):
File “/home/cotto/rebalance/logic.py”, line 88, in low_local_ratio_after_sending
remote = channel.remote_balance + total_amount
AttributeError: ‘NoneType’ object has no attribute ‘remote_balance’

Make script faster

Currently, nothing is cached. For some requests that makes the script run slow.

Some debug output I created for a typical rebalance job which shows the total time for the given functions (in seconds):

{'generate_invoice': 0.011147022247314453, 'get_routes': 16.460828065872192, 'send_payment': 13.990042924880981, 'get_policy': 19.16523838043213, 'get_payment_hash': 0.0009760856628417969, 'get_graph': 18.756810426712036, 'init': 9.5367431640625e-07, 'get_info': 1.6338262557983398, 'get_expiry': 0.017612218856811523, 'get_own_pubkey': 0.8776724338531494, 'get_edges': 18.761753797531128, 'get_current_height': 0.7577559947967529, 'get_channels': 1.1018741130828857}

TypeError: not all arguments converted during string formatting

I am following the instructions but I can't get it to work and the error message is helping in solving the issue. What is the problem?

$ python rebalance.py -t 614105831440187392 -p 20
Sending 29,302 satoshis to rebalance to channel with ID 614105831440187392
Traceback (most recent call last):
  File "rebalance.py", line 241, in <module>
    success = main()
  File "rebalance.py", line 70, in main
    max_fee_factor).rebalance()
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 46, in rebalance
    success = self.try_route(payment_request, route, routes, tried_routes)
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 53, in try_route
    if self.route_is_invalid(route, routes):
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 114, in route_is_invalid
    debugnobreak("Target channel is first hop, " % first_hop.chan_id)
TypeError: not all arguments converted during string formatting
(base) 

Gather intel on failing paths in order to drive routing heuristics

There are still many failures on payment attempts. While we don't have much visibility on why these are failing, we can probably infer two pieces of information from these failures, and use that to prioritise certain routes in the future for faster and more successful rebalances.
The main reasons for failure today are:

  • liquidity on channels
  • offline/out of service nodes

A small concept PR on my repo displays paths tried during the rebalancing process (https://github.com/wamde/rebalance-lnd/pull/4/files).

We could start by storing the paths tried for a given amount and whether it worked or failed. We can add the node pubkeys to store directionality information too.
Crossing information between such tries could show that there is some probability that a certain channel is either full in a direction or just offline. If a channel is full in one direction we can use this as a signal that it may have liquidity in the other direction.

There are many caveats with such an approach but it could probably help with rebalancing these days when the network is not so dynamic yet.

Deal with message limits

  File "/home/cotto/rebalance/lnd/lnd.py", line 41, in get_graph
    graph = self.stub.DescribeGraph(ln.ChannelGraphRequest())
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 547, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Received message larger than max (4230348 vs. 4194304)"
	debug_error_string = "{"created":"@1545772110.601449580","description":"Received message larger than max (4230348 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}"

Make it easier to run this tool remotely

The "server address" and "macaroon/cert locations" are currently hardcoded into the script at lnd.py in a way that assumes that the script will be run in the same environment as the lnd node. Specifically for running the scripts remotely, it would be useful if there was some way to allow the user to define:

  1. a common folder location where credentials (admin.macaroon & tls.cert) are stored, and
  2. the ip:port that the lnd node is listening on for rpc connections

Zap Desktop folks have a decent UX defined for how this process works. Their project could be a good place to start for guidance for anyone looking to take a crack at this issue. One idea could be to implement --serverip and --credentialspath arguments to accomplish this.

Support configurable channel ratios

Instead of the command with no arguments returning "channels with 50%+ on one side"
Return channels where X amount is on either side. Default could be 75%.

No one is going to try to rebalance a channel with 50.0001% on local side. Running a large hub, I wouldn't do any rebalancing until the amount left is smaller than a typical transaction... which about 5-20%, depending on the channel.

And, anyone doing manual rebalancing is going to want to know all channels that need rebalancing, not just the ones with high local balance.
You might even make them separate commands... one to show Local balance > X% and a 2nd argument to return only channels with remote balance > X%

Try and force the first hop

The current logic looks at channels which could use some incoming funds, but does not pair that with channels which could be emptied a bit to reach a 50/50 state. The goal is to be able to diagnose candidate routes with this in mind, and to support the rebalance mode with that constraint too.

some route work, some doesn't...

I have and issue where the script would work on some channels but not all, and the one that doesn't, here's the error message:

Traceback (most recent call last):
File ".\rebalance.py", line 227, in
main()
File ".\rebalance.py", line 55, in main
amount = get_amount(arguments, first_hop_channel_id, last_hop_channel)
File ".\rebalance.py", line 70, in get_amount
amount = get_rebalance_amount(last_hop_channel)
File ".\rebalance.py", line 179, in get_rebalance_amount
return abs(int(math.ceil(float(get_remote_surplus(channel)) / 2)))
File ".\rebalance.py", line 201, in get_remote_surplus
return channel.remote_balance - channel.local_balance
AttributeError: 'NoneType' object has no attribute 'remote_balance'

image

UNKNOWN_PAYMENT_HASH

Using Python 2.7.15+ on Ubuntu 18.04, latest code from master, lnd version 0.7.0-beta commit=v0.7.0-beta-rc3 installed with routerrpc, latest requirements.txt installed with pip. I'm trying to rebalance a channel with the following command and getting an error. I've tried this with different "to" channels and different amounts and I get the same error.

$ python rebalance.py -f 627916797039411200 -t 630216975379267584 -a 500000 --max-fee-factor 3
Sending 500,000 satoshis to rebalance to channel with ID 630216975379267584
Forced first channel has ID 627916797039411200

Trying route #1
627916797039411200 -> 623648492953534464 -> 631089987556999169 -> 630216975379267584
failure {
  code: UNKNOWN_PAYMENT_HASH
  failure_source_pubkey: "\003\201\030\340I\017\215=\360{k\334^p4\336\351\rv\220\022vM1\277\353\026ri\206h$\360"
}

Unknown error code 1
Could not find any suitable route

Specify absolute fee max instead of average fee max

Right now it seems that it's only possible to specify the maximum average fee per hop for a rebalancing route, when really I'd be more interested in specifying a maximum absolute fee (still in terms of the lnd default for a single hop). Basically, I'd rather pay 100 satoshis for 2 hops than 120 satoshis for 3 hops, even though the latter has a lower per-hop average.

This could also allow for some shortcuts to prevent searching for routes when the fees for the first or last hop (specified by the -f and -t arguments) exceed the maximum fee. If I don't want to pay more than 10 times the lnd default fee, and I'm trying to rebalance a channel which itself charges 20 times the lnd default fee, then the rebalancing should not even be attempted potentially saving time and headache. I guess this might change if negative fees ever become a thing, however.

Last minor request, it'd be nice to have a single character alias for this argument, like -m or -f instead of --max-fee-factor.

Feature Request: Exclude Channels

It would be helpful to be able to exclude one or more channels when rebalancing.

In my case, I have multiple channels to a merchant. After sending them funds, I like to top-up the local balance on that channel so I have max available to spend in the future. But because I have multiple channels to them, the script will often select the other channel to them to balance from unless I specify a single -f channel. This takes a bit of time since I will often have to try a few channels individually before finding a viable route. If instead I could just indicate which channels I did not want the script to use, it could try all the rest without having to re-run the script manually for each -f channel.

python3: type mismatch in routes.py

trying to run on windows with git bash, when i run ./rebalance.py 26 100000 the result was:

  • File "./rebalance.py", line 113, in
    main()
    File "./rebalance.py", line 52, in main
    response = Logic(lnd, remote_pubkey, amount).rebalance()
    File "D:\tests\rebalance-lnd-master\logic.py", line 22, in rebalance
    if not routes.has_next():
    File "D:\tests\rebalance-lnd-master\routes.py", line 24, in has_next
    self.update_routes()
    File "D:\tests\rebalance-lnd-master\routes.py", line 43, in update_routes
    self.request_routes(num_routes_to_request)
    File "D:\tests\rebalance-lnd-master\routes.py", line 50, in request_routes
    modified_route = self.add_rebalance_channel(route)
    File "D:\tests\rebalance-lnd-master\routes.py", line 74, in add_rebalance_channel
    self.update_amounts(hops)
    File "D:\tests\rebalance-lnd-master\routes.py", line 122, in update_amounts
    hop.amt_to_forward = amount_to_forward_msat / 1000
    File "C:\Users\fr4nz\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 662, in field_setter
    new_value = type_checker.CheckValue(new_value)
    File "C:\Users\fr4nz\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\type_checkers.py", line 133, in CheckValue
    raise TypeError(message)
    TypeError: 100000.0 has type <class 'float'>, but expected one of: (<class 'int'>,)

I don't know if is a os related bug but i solved with
Fr4nZ82@ed9e90e

Debug mode for route candidates

Routes tend to fail for various reasons, often unclear.
We should add a proper debug mode to show routes constructed to be able to diagnose what's up or what they have in common which might fail which may lead to heuristics to try and add diversity to them.

What about the other way around?

Say I've just opened two new channels. Both of these channels will have 100% of their funds on the local side. Now, if I want to be able to route other people's transactions, I need to get some funds on the remote sides of these channels. Could the script be updated to work in this direction as well?

Simple auto mode

Add a simple auto mode, where the tool automatically finds out which is the best rebalance and carries it out (with some constraints, like max fees)

Multiple instances of bitcoind and lnd?

My raspiblitz was running low on memory, so I accessed it via ssh and ran sudo htop. I was surprised to see so many bitcoind and lnd processes. Is this normal?

Screen Shot 2019-03-17 at 11 57 12 AM

bitcoind says it has 20 connections and lnd is connnected to 19 peers, so could it be the case that every connection/peer spawns its own child process? or is this a problem?

Odd behaviour when balancing one of the two channels with same node

I've been using it for a month now. It is useful and works very well! thanks for this tool!

Nevertheless, I've noticed that whenever I try to balance one of the two channels with a the same node (ie, I have two channels open with the same node), it behaves oddly.

First time I experienced that was a around a month ago. I reduced the amount each time it wasn't able to route up until a payment of 1000 sats went through but to the wrong channel. I closed one of the channels and everything worked fine for the only channel available.

Right now I have the same situation with a different node. Two channels with 1Msats and 2Msats. Here it is a list of what I thing was some weird behaviour:

  • Forcing with option '-f' of two channels, no success in any of them.
    -Without using '-f' and specifying an amount with '-a' at 40ksats, it went through, but with what I expected lower fees. I paid 18sats in fees is higher than the default fee isn't it?. If not mistaken, max fee is given by the max fee factor at 10, which is 10 times the fee at 10*(1sat + 1e-6*Amount), and I would have expected a fee at less than 10.4sat.

Any hints on what can I do to give more information or debug what's happening?


admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -f 624######
Sending 40,272 satoshis to rebalance to channel with ID 636######
Forced first channel has ID 624#####
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
requesting 30 routes from lnd, please wait.
lnd returned 30 routes that will now be tested
requesting 45 routes from lnd, please wait.
lnd returned 45 routes that will now be tested
requesting 60 routes from lnd, please wait.
lnd returned 60 routes that will now be tested
admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -f 627######
Sending 20,270 satoshis to rebalance to channel with ID 636######
Forced first channel has ID 627######
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
requesting 30 routes from lnd, please wait.
lnd returned 30 routes that will now be tested
requesting 45 routes from lnd, please wait.
lnd returned 45 routes that will now be tested
requesting 60 routes from lnd, please wait.
lnd returned 60 routes that will now be tested
admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -a 40000
Sending 40,000 satoshis to rebalance to channel with ID 636######
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
Trying route #1
635###### -> 636###### -> 621###### -> 636######
Success! Paid 18 Satoshi in fees
Tried routes:
635###### -> 636###### -> 621###### -> 636######
Successful route:
635###### -> 636###### -> 621###### -> 636######
payment_preimage: "######"
payment_route {
  total_time_lock: 579893
  total_fees: 18
  total_amt: 40018
  hops {
    chan_id: 635######
    chan_capacity: 400000
    amt_to_forward: 40018
    expiry: 579849
    amt_to_forward_msat: 40018040
    fee_msat: 40
    pub_key: "03######"
  }
  hops {
    chan_id: 636######
    chan_capacity: 1000000
    amt_to_forward: 40018
    expiry: 579845
    amt_to_forward_msat: 40018000
    fee_msat: 40
    pub_key: "02######"
  }
  hops {
    chan_id: 621######
    chan_capacity: 5000000
    amt_to_forward: 40000
    fee: 18
    expiry: 579815
    amt_to_forward_msat: 40000000
    fee_msat: 18000
    pub_key: "03######"
  }
  hops {
    chan_id: 636######
    chan_capacity: 4000000
    amt_to_forward: 40000
    expiry: 579815
    amt_to_forward_msat: 40000000
    pub_key: "03######"
  }
  total_fees_msat: 18080
  total_amt_msat: 40018080
}
payment_hash: "######"

Feature Request: Pick entry and exit channels

Balancing needs to be able to happen from both sides.
One needs to know the channels where too much is on the remote side, and during the actual rebalance, be able to pick both the incoming and outgoing channels.

Python3: import problem on the grpc_generated folder

Got the following error on files inside grpc_generated folder:

    import rpc_pb2 as rpc__pb2
ModuleNotFoundError: No module named 'rpc_pb2'

Resolution is adding from grpc_generated to the import line import rpc_pb2 as rpc__pb2
Other files inside grpc_generated folder have similar problem.

P.S. I am using python 3.7.4 on my raspberry pi

Crash: Unable to find a path to destination

I tried rebalancing a channel I opened to my Eclair wallet on my phone.

Traceback (most recent call last):
  File "./rebalance.py", line 223, in <module>
    main()
  File "./rebalance.py", line 62, in main
    response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
  File "/home/cotto/rebalance/logic.py", line 34, in rebalance
    if not routes.has_next():
  File "/home/cotto/rebalance/routes.py", line 24, in has_next
    self.update_routes()
  File "/home/cotto/rebalance/routes.py", line 43, in update_routes
    self.request_routes(num_routes_to_request)
  File "/home/cotto/rebalance/routes.py", line 47, in request_routes
    routes = self.lnd.get_routes(self.last_hop_channel.remote_pubkey, self.get_amount(), num_routes_to_request)
  File "/home/cotto/rebalance/lnd.py", line 78, in get_routes
    response = self.stub.QueryRoutes(request)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 547, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "unable to find a path to destination"
	debug_error_string = "{"created":"@1555085155.113312254","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"unable to find a path to destination","grpc_status":2}"

can't execute

Background
I try to excecute and have this error :

$ sudo python3 rebalance.py -l
> Traceback (most recent call last):
>   File "rebalance.py", line 8, in <module>
>     from lnd import Lnd
>   File "/home/admin/rebalance-lnd/lnd.py", line 4, in <module>
>     import grpc
> ImportError: No module named 'grpc'

My environment
Raspberry Pi3b linux -> nodo as Stadicus Raspibolt
lncli version 0.5.2-beta
bitcoind

Steps to reproduce
I git clone the repository to /home/admin/rebalance-lnd
I cd ~/rebalance-lnd and then python3 rebalance.py -h

Expected behaviour
Script to work

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.