Somewhat a follow-up to 649ce979ab.
This adds some safety belts so we don't accidentally sign a tx that
contains a dummy address.
Specifically we check that tx does not contain output for dummy addr:
- in wallet.sign_transaction
- in network.broadcast_transaction
The second one is perhaps redundant, but I think it does not hurt.
Due to an indendation error, fw_info was returned only for one
HTLC of the MPP. Thus, maybe_fulfill_htlc was called again with
the remaining HTLCs, which could possibly be failed due to MPP
timeout, resulting in fund loss for the forwarder.
- rename trampoline_forwardings -> final_onion_forwardings,
because this dict is used for both trampoline and hold invoices
- remove timeout from hold_invoice_callbacks (redundant with invoice)
- add test_failure boolean parameter to TestPeer._test_simple_payment,
in order to test correct propagation of OnionRoutingFailures.
- maybe_fulfill_htlc: raise an OnionRoutingFailure if we do not have
the preimage for a payment that does not have a hold invoice callback.
Without this, the above unit tests stall when we use test_failure=True
- introduce SentHtlcInfo named tuple
- some previously unnamed tuples are now much shorter:
create_routes_for_payment no longer returns an 8-tuple!
- sent_htlcs_q (renamed from sent_htlcs), is now keyed on payment_hash+payment_secret
(needed for proper trampoline forwarding)
- add RecvMPPResolution enum for possible states of a pending incoming MPP,
and use it in check_mpp_status
- new state: "FAILED", to allow nicely failing back the whole MPP set
- key more things with payment_hash+payment_secret, for consistency
(just payment_hash is insufficient for trampoline forwarding)
triggered a payment forwarding.
Final onions may trigger a payment forwarding, through the callback
returned by maybe_fulfill_htlc. In that case, we should not fail the
HTLC later; doing so might result in fund loss.
Remove test_simple_payment_with_hold_invoice_timing_out: once we
have accepted to forward a payment HTLC with a hold invoice, we
do not want to time it out, for the same reason.
- maybe_fulfill_htlc returns a forwarding callback that
covers both cases.
- previously, the callback of hold invoices was called as a
side-effect of lnworker.check_mpp_status.
- the same data structures (lnworker.trampoline_forwardings,
lnworker.trampoline_forwarding_errors) are used for both
trampoline forwardings and hold invoices.
- maybe_fulfill_htlc still recursively calls itself to perform
checks on trampoline onion. This is ugly, but ugliness is now
contained to that method.
- fix parameters passed to maybe_forward_trampoline
- use lnworker.trampoline_forwardings as a semaphore for ongoing
trampoline payments
- if a trampoline payment fails, fail all received HTLCs
- qt chan details dlg: show both local and remote aliases
- lnchannel: more descriptive names, add clarification in doctstrings,
and also save the "local_scid_alias" in the wallet file (to remember if
we sent it)
- lnpeer:
- resend channel_ready msg after reestablish, to upgrade old existing channels
to having local_scid_alias
- forwarding bugfix, to follow BOLT-04:
> - if it returns a `channel_update`:
> - MUST set `short_channel_id` to the `short_channel_id` used by the incoming onion.
A new config API is introduced, and ~all of the codebase is adapted to it.
The old API is kept but mainly only for dynamic usage where its extra flexibility is needed.
Using examples, the old config API looked this:
```
>>> config.get("request_expiry", 86400)
604800
>>> config.set_key("request_expiry", 86400)
>>>
```
The new config API instead:
```
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS
604800
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS = 86400
>>>
```
The old API operated on arbitrary string keys, the new one uses
a static ~enum-like list of variables.
With the new API:
- there is a single centralised list of config variables, as opposed to
these being scattered all over
- no more duplication of default values (in the getters)
- there is now some (minimal for now) type-validation/conversion for
the config values
closes https://github.com/spesmilo/electrum/pull/5640
closes https://github.com/spesmilo/electrum/pull/5649
Note: there is yet a third API added here, for certain niche/abstract use-cases,
where we need a reference to the config variable itself.
It should only be used when needed:
```
>>> var = config.cv.WALLET_PAYREQ_EXPIRY_SECONDS
>>> var
<ConfigVarWithConfig key='request_expiry'>
>>> var.get()
604800
>>> var.set(3600)
>>> var.get_default_value()
86400
>>> var.is_set()
True
>>> var.is_modifiable()
True
```
closes https://github.com/spesmilo/electrum/issues/8403
> In Python 3.10 that worked fine, however in Python 3.11 large integer check https://github.com/python/cpython/issues/95778, so now this throws an error.
Apparently this change was deemed a security fix and was backported to all supported branches of CPython (going back to 3.7). i.e. it affects ~all versions of python (if sufficiently updated with bugfix patches), not just 3.11
> Some offending node aliases:
> ```
> ergvein-fiatchannels
> test-mainnet
> arakis
> ```
The features bits set by some of these nodes:
```
(1, 7, 8, 11, 13, 14, 17, 19, 23, 27, 45, 32973, 52973)
(1, 7, 8, 11, 13, 14, 17, 19, 23, 27, 39, 45, 55, 32973, 52973)
```
> P.S. I see there are a lot of nodes with 253 bytes in their feature vectors. Any idea why that could happen?
Note that the valid [merged-into-spec features](50b2df24a2/09-features.md) currently only go as high as ~51.
However the spec does not specify how to choose feature bits for experimental stuff, so I guess some people are using values in the 50k range. The only limit imposed by the spec on the length of the features bitvector is an implicit one due to the max message size: every msg must be smaller than 65KB, and the features bitvector needs to fit inside the init message, hence it can be up to ~524K bits.
(note that the features are not stored in a sparse representation in the init message and in gossip messages, so if many nodes set such high feature bits, that would noticably impact the size of the gossip).
-----
Anyway, our current implementation of LnFeatures is subclassing IntFlag, and it looks like it does not work well for such large integers. I've managed to make IntFlags reasonably in python 3.11 by overriding __str__ and __repr__ (note that in cpython it is apparently only the base2<->base10 conversions that are slow, power-of-2 conversions are fast, so we can e.g. use `hex()`). However in python 3.10 and older, enum.py itself seems really slow for bigints, e.g. enum._decompose in python 3.10.
Try e.g. this script, which is instant in py3.11 but takes minutes in py3.10:
```py
from enum import IntFlag
class c(IntFlag):
known_flag_1 = 1 << 0
known_flag_2 = 1 << 1
known_flag_3 = 1 << 2
if hasattr(IntFlag, "_numeric_repr_"): # python 3.11+
_numeric_repr_ = hex
def __repr__(self):
return f"<{self._name_}: {hex(self._value_)}>"
def __str__(self):
return hex(self._value_)
a = c(2**70000-1)
q1 = repr(a)
q2 = str(a)
```
AFAICT we have two options: either we rewrite LnFeatures so that it does not use IntFlag (and enum.py), or, for the short term as workaround, we could just reject very large feature bits.
For now, I've opted to the latter, rejecting feature bits over 10k.
(note that another option is bumping the min required python to 3.11, in which case with the overrides added in this commit the performance looks perfectly fine)
I was calling methods from the Qt console (e.g. peer.pay()) and seeing weird behaviour...
htlc_switch() (running on asyncio thread) was racing with pay() (running on GUI thread).
- save remote alias for use in invoices
- derive local alias from wallet xpub
- send channel_type without the option_scid_alias bit
(apparently LND does not like it)
note: Would it be ok to log potentially secret (semi-sensitive) data?
We take care not to log onchain private keys as they are extremely sensitive,
but what about logging a LN transport message that might contain channel secrets?
Decided not to, for now.
ThomasV
SomberNight