The earned fees calculation seems to be double the actual amount. The problem seems to be around here. I'm not sure why fees are counted on both the incoming and outgoing channels? The relay fee is only earned on the outgoing channel.

Commenting out this line results in a calculation that seems closer.

I checked the suez script calculation with this homemade script (could have its own flaw but I did check a daily period by hand and it was correct):

lncli fwdinghistory --start_time -30d --max_events 5000 | grep '"fee"' | sed -nr 's/.*"([0-9]+).*/\1/p' | awk 'BEGIN {total=0;}{total+=$1;}END {print "Total: ",total}'

On my node, even after applying the fix mentioned above, my script and suez still result in an earned fees discrepancy of 16 sats.

$ uname -a Linux btcaz 5.4.0-1046 #48~18.04.1-Ubuntu SMP Tue Apr 13 19:41:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.5 LTS Release: 18.04 Codename: bionic

$ python3 --version Python 3.6.9

$ poetry --version Poetry version 1.1.6

$ git log commit 1894ea0bfffa87b142501557c2f82def2028a4b5 (HEAD -> master, origin/master, origin/HEAD)

$ poetry run ./suez

Traceback (most recent call last):
  File "./suez", line 2, in <module>
    from suez import suez
  File "/home/azureuser/src/suez/suez.py", line 5, in <module>
    from rich.console import Console
  File "/home/azureuser/.cache/pypoetry/virtualenvs/suez-qnYOxQkH-py3.6/lib/python3.6/site-packages/rich/console.py", line 9, in <module>
    from dataclasses import dataclass, field
ModuleNotFoundError: No module named 'dataclasses'

Hello,

What's the point of setting low commissions for channels where your balance is depleted? Thus, you stimulate payments through such channels, but there is no liquidity in them for sending. It is bad for routing (payments through them are more likely will be FAIL but third-party wallets/senders will try to choose these channels as there are low commissions).

It would be quite logical to increase the commission for its part where your local balance is low.

P.S. Honestly, I do not understand - why use here the distribution of Gauss?

Attempting to install suez on Ubuntu Ubuntu 20.04.3 LTS, but when running poetry install, I am receiving: No module named 'virtualenv.seed.via_app_data'.

I am not sure what version of virtualenv is required, but mine is virtualenv 20.13.0

I think remote fees are not calculated correcty:

https://github.com/prusnak/suez/blob/master/lndclient.py#L75

self.channels[cin].remote_fees += fee

The fee on the incoming channel (the remote party fee) is their base fee + amount * fee_rate but here you're using our fee.

This should be:

`self.channels[cin].remote_fees += (self.channels[cin].remote_fee_base + forward_amount * self.channels[cin].remote_fee_base / 10000) / 1000

or something along these lines.

What do you think @prusnak ?

Looking to run on a remote node (Voltage) and an Umbrel. I think this breaks because the lncli call has to be changed. I tried changing ["lncli"] in lndclient.py def _run with the call commands I would use but it fails still.

Voltage lncli --rpcserver=xxxxx.votageapp.io:10009 --macaroonpath=/path/to/admin.macaroon --tlscertpath=""

Umbrel docker exec -i lnd lncli

This is what I get on my testnet LND instance with both python 3.6 and 3.7.

[bitcoin@lnd-tst suez]$ ./suez
Traceback (most recent call last):
  File "./suez", line 5, in <module>
    suez()
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/bitcoin/Download/src/suez/suez.py", line 53, in suez
    ln = clients[client](client_args)
  File "/home/bitcoin/Download/src/suez/lndclient.py", line 10, in __init__
    self.refresh()
  File "/home/bitcoin/Download/src/suez/lndclient.py", line 38, in refresh
    int(info["node1_policy"]["fee_base_msat"]),
TypeError: 'NoneType' object is not subscriptable
[bitcoin@lnd-tst suez]$

List of all the channels: http://transfer.sh/1MKZDnc/listchannels.txt

Fee report: http://transfer.sh/1WS7ZZL/feereport.txt

Found this today when testing CLN v0.11.0.1

₿ tlightning-cli listpeers
{
   "peers": [
      {
         "id": "026a86832609bf420c56a658ef2504c8eb4f6f538d0b56824ef240d7454872894a",
         "connected": true,
         "netaddr": [
            "127.0.0.1:37530"
         ],
         "features": "800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000808252a1",
         "channels": [
            {
               "state": "CHANNELD_NORMAL",
               "scratch_txid": "4e60cbaadbc7c3236b1bc19b124e8de065c46efcab76cca2c8dd5ecb53b0fb4d",
               "last_tx_fee": "183000msat",
               "last_tx_fee_msat": "183000msat",
               "feerate": {
                  "perkw": 253,
                  "perkb": 1012
               },
               "owner": "channeld",
               "short_channel_id": "2197375x70x0",
               "direction": 1,
               "channel_id": "b59b31d18f11204c65c233387a67d1de5b6ca5beb8220a7e14f6fecd28cda006",
               "funding_txid": "06a0cd28cdfef6147e0a22b8bea56c5bded1677a3833c2654c20118fd1319bb5",
               "funding_outnum": 0,
               "close_to_addr": "tb1qfh2fssw0u9mejljrr9g53h3fruynlfjkfdtq8v",
               "close_to": "00144dd49841cfe177997e43195148de291f093fa656",
               "private": false,
               "opener": "remote",
               "closer": null,
               "features": [
                  "option_static_remotekey"
               ],
               "funding_allocation_msat": {
                  "026a86832609bf420c56a658ef2504c8eb4f6f538d0b56824ef240d7454872894a": 500000000,
                  "03b2b374a259d2ec8ca46f7db079fbb12b6e307dbb319b1d2f911500e332e41d41": 0
               },
               "funding_msat": {
                  "026a86832609bf420c56a658ef2504c8eb4f6f538d0b56824ef240d7454872894a": "500000000msat",
                  "03b2b374a259d2ec8ca46f7db079fbb12b6e307dbb319b1d2f911500e332e41d41": "0msat"
               },
               "funding": {
                  "local_msat": "0msat",
                  "remote_msat": "500000000msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 0,
               "to_us_msat": "0msat",
               "msatoshi_to_us_min": 0,
               "min_to_us_msat": "0msat",
               "msatoshi_to_us_max": 0,
               "max_to_us_msat": "0msat",
               "msatoshi_total": 500000000,
               "total_msat": "500000000msat",
               "fee_base_msat": "2168msat",
               "fee_proportional_millionths": 2,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 5000,
               "their_reserve_msat": "5000000msat",
               "our_channel_reserve_satoshis": 5000,
               "our_reserve_msat": "5000000msat",
               "spendable_msatoshi": 0,
               "spendable_msat": "0msat",
               "receivable_msatoshi": 494460000,
               "receivable_msat": "494460000msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "1msat",
               "maximum_htlc_out_msat": "495000000msat",
               "their_to_self_delay": 6,
               "our_to_self_delay": 144,
               "max_accepted_htlcs": 483,
               "state_changes": [
                  {
                     "timestamp": "2022-04-27T06:15:15.081Z",
                     "old_state": "CHANNELD_AWAITING_LOCKIN",
                     "new_state": "CHANNELD_NORMAL",
                     "cause": "remote",
                     "message": "Lockin complete"
                  }
               ],
               "status": [
                  "CHANNELD_NORMAL:Reconnected, and reestablished.",
                  "CHANNELD_NORMAL:Funding transaction locked. Channel announced."
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            },
            {
               "state": "CHANNELD_NORMAL",
               "scratch_txid": "aaba8cad2e010f6993138d1dbb2cf90008d9d7dec5b3044645db0efd25c0f33e",
               "last_tx_fee": "183000msat",
               "last_tx_fee_msat": "183000msat",
               "feerate": {
                  "perkw": 253,
                  "perkb": 1012
               },
               "owner": "channeld",
               "short_channel_id": "2197376x20x0",
               "direction": 1,
               "channel_id": "9bbe6c61431fcaab9ae02b8cd975606e9f25186e55cbca9b0c84f0a9808ea00f",
               "funding_txid": "0fa08e80a9f0840c9bcacb556e18259f6e6075d98c2be09aabca1f43616cbe9b",
               "funding_outnum": 0,
               "close_to_addr": "tb1qmf30ruw45h37hdydu4nw43la6p4rfusm5p635r",
               "close_to": "0014da62f1f1d5a5e3ebb48de566eac7fdd06a34f21b",
               "private": false,
               "opener": "remote",
               "closer": null,
               "features": [
                  "option_static_remotekey"
               ],
               "funding_allocation_msat": {
                  "026a86832609bf420c56a658ef2504c8eb4f6f538d0b56824ef240d7454872894a": 250000000,
                  "03b2b374a259d2ec8ca46f7db079fbb12b6e307dbb319b1d2f911500e332e41d41": 0
               },
               "funding_msat": {
                  "026a86832609bf420c56a658ef2504c8eb4f6f538d0b56824ef240d7454872894a": "250000000msat",
                  "03b2b374a259d2ec8ca46f7db079fbb12b6e307dbb319b1d2f911500e332e41d41": "0msat"
               },
               "funding": {
                  "local_msat": "0msat",
                  "remote_msat": "250000000msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 0,
               "to_us_msat": "0msat",
               "msatoshi_to_us_min": 0,
               "min_to_us_msat": "0msat",
               "msatoshi_to_us_max": 0,
               "max_to_us_msat": "0msat",
               "msatoshi_total": 250000000,
               "total_msat": "250000000msat",
               "fee_base_msat": "2168msat",
               "fee_proportional_millionths": 2,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 2500,
               "their_reserve_msat": "2500000msat",
               "our_channel_reserve_satoshis": 2500,
               "our_reserve_msat": "2500000msat",
               "spendable_msatoshi": 0,
               "spendable_msat": "0msat",
               "receivable_msatoshi": 246960000,
               "receivable_msat": "246960000msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "1msat",
               "maximum_htlc_out_msat": "247500000msat",
               "their_to_self_delay": 6,
               "our_to_self_delay": 144,
               "max_accepted_htlcs": 483,
               "state_changes": [
                  {
                     "timestamp": "2022-04-27T06:18:15.335Z",
                     "old_state": "CHANNELD_AWAITING_LOCKIN",
                     "new_state": "CHANNELD_NORMAL",
                     "cause": "remote",
                     "message": "Lockin complete"
                  }
               ],
               "status": [
                  "CHANNELD_NORMAL:Reconnected, and reestablished.",
                  "CHANNELD_NORMAL:Funding transaction locked. Channel announced."
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            }
         ]
      }
   ]
}

Any idea how to best debug this?

Versions:

lnd version 0.13.0-beta commit=v0.13.0-beta
Poetry version 1.1.7
Python 3.7.3

lnd.conf (relevant parts):

listen=localhost
rpclisten=localhost:10009
bitcoin.active=true
bitcoin.mainnet=true
bitcoin.node=bitcoind
tor.active=true
tor.socks=9050
tor.v3=true
tor.v2=false

Log:

$ lncli getinfo
{
    "version": "0.13.0-beta commit=v0.13.0-beta",
    [...]
}

$ poetry run ./suez
Traceback (most recent call last):
  File "./suez", line 5, in <module>
    suez()
  File "/home/asdf/.cache/pypoetry/virtualenvs/suez-aZJboPJE-py3.7/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/asdf/.cache/pypoetry/virtualenvs/suez-aZJboPJE-py3.7/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/asdf/.cache/pypoetry/virtualenvs/suez-aZJboPJE-py3.7/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/asdf/.cache/pypoetry/virtualenvs/suez-aZJboPJE-py3.7/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/asdf/suez/suez.py", line 190, in suez
    info = info_box(ln, score)
  File "/home/asdf/suez/suez.py", line 29, in info_box
    grid.add_row("score  : ", "{:,}".format(score.get(ln.local_pubkey)))
TypeError: unsupported format string passed to NoneType.__format__

I'm not sure if this project intends to cover all weird edge cases, but just in case, here's another one:

...
XXX 2171602535068008448
XXX 2130060786746851328
XXX 1598976879324692480
XXX 2163737728395575296
[lncli] rpc error: code = Unknown desc = edge not found
[lncli] rpc error: code = Unknown desc = unable to find node
Traceback (most recent call last):
  File "./suez", line 5, in <module>
    suez()
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bitcoin/.local/lib/python3.6/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/bitcoin/Download/src/suez/suez.py", line 53, in suez
    ln = clients[client](client_args)
  File "/home/bitcoin/Download/src/suez/lndclient.py", line 10, in __init__
    self.refresh()
  File "/home/bitcoin/Download/src/suez/lndclient.py", line 61, in refresh
    chan.remote_alias = self._run("getnodeinfo", chan.remote_node_id)["node"][
  File "/home/bitcoin/Download/src/suez/lndclient.py", line 115, in _run
    return json.loads(j.stdout)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[bitcoin@lnd-tst suez]$ lncli getchaninfo 2163737728395575296
[lncli] rpc error: code = Unknown desc = edge not found
[bitcoin@lnd-tst suez]$

        {
            "active": false,
            "remote_pubkey": "02bda6b2aeb315edd874d72ac2873e343448e7e5f126f62cee225f284de392debe",
            "channel_point": "b605fc85e5b95b744d065877de19762697df1721d95ca02b44a3b3a0d6ca0b8e:0",
            "chan_id": "2163737728395575296",
            "capacity": "1999810",
            "local_balance": "0",
            "remote_balance": "1999627",
            "commit_fee": "183",
            "commit_weight": "552",
            "fee_per_kw": "253",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 1008,
            "private": true,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "19998",
            "remote_chan_reserve_sat": "19998",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "84858",
            "uptime": "0",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 1008,
                "chan_reserve_sat": "19998",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "1999810000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 5
            },
            "remote_constraints": {
                "csv_delay": 240,
                "chan_reserve_sat": "19998",
                "dust_limit_sat": "546",
                "max_pending_amt_msat": "1979812000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },

I've added support for C-Lightning and:

1. Introduced new --client and --client-args options
2. --client can be LND or C-Lightning
3. --client-args - I needed to pass additional options to lightning-cli
4. Fixed remote fee calculation in clnclient.py (not in lndclient.py)
5. Added a new opener column (for both LND and C-LN)
6. Updated the README

It would be very nice to use the RPC API provided by lnd rather than relying on lncli being installed. I am trying to run suez in a Docker container and I'd rather not install lncli when it could directly communicate over RPC with lnd from another container.

I saw that that the choice of backend is abstracted away in separate classes, one for lnd and one for c-lightning. So in theory I could just add another one for RPC. Did somebody already look into that? Or is there some good reason to rely on lncli?

Hey, it would be great if there is the option to include Min and Max HTLCs of both sides of each channels. Because these are 4 additional figures it is better when they are only displayed if a command line parameter is set.

Regards Steff

https://docs.lightning.engineering/lightning-network-tools/lnd/safety#prevent-data-corruption

The following (non-exhaustive) list of things can lead to data corruption: [...] Aborting channel operation commands (see next chapter). [...]

Next chapter (https://docs.lightning.engineering/lightning-network-tools/lnd/safety#dont-interrupt-lncli-commands):

Things can start to take a while to execute if a node has more than 50 to 100 channels. It is extremely important to never interrupt an lncli command if it is manipulating the channel database, which is true for the following commands: [...] updatechanpolicy [...]

Thus, I assume we should "never interrupt" suez.py?

This change adds abstraction to enable multiple LN implementations and implements basic Eclair support. It is not yet capable of updating channel policy but can show all relevant channels.

I seek feedback for this change first. It's also a bit hackish, I mainly wanted to make basic features work with my node for starters.