- introduce SentHtlcInfo named tuple
- some previously unnamed tuples are now much shorter:
create_routes_for_payment no longer returns an 8-tuple!
- sent_htlcs_q (renamed from sent_htlcs), is now keyed on payment_hash+payment_secret
(needed for proper trampoline forwarding)
- add RecvMPPResolution enum for possible states of a pending incoming MPP,
and use it in check_mpp_status
- new state: "FAILED", to allow nicely failing back the whole MPP set
- key more things with payment_hash+payment_secret, for consistency
(just payment_hash is insufficient for trampoline forwarding)
The "python:rc" tag on dockerhub has not been updated for 2+ years
and is still at 3.10.0rc2. Even 3.11 has been released for many months now.
see https://hub.docker.com/_/python/tags
follow-up https://github.com/spesmilo/electrum/pull/8536
```
1.52 | E | gui.qt.ElectrumGui |
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/gui/qt/__init__.py", line 342, in start_new_window
wallet = self.daemon.load_wallet(path, None)
File "/home/user/wspace/electrum/electrum/daemon.py", line 469, in func_wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/daemon.py", line 479, in load_wallet
wallet = self._load_wallet(path, password, manual_upgrades=manual_upgrades, config=self.config)
File "/home/user/wspace/electrum/electrum/util.py", line 466, in do_profile
o = func(*args, **kw_args)
File "/home/user/wspace/electrum/electrum/daemon.py", line 504, in _load_wallet
db = WalletDB(storage.read(), manual_upgrades=manual_upgrades)
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 117, in __init__
self._after_upgrade_tasks()
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 247, in _after_upgrade_tasks
self._load_transactions()
File "/home/user/wspace/electrum/electrum/util.py", line 466, in do_profile
o = func(*args, **kw_args)
File "/home/user/wspace/electrum/electrum/wallet_db.py", line 1536, in _load_transactions
self.data = StoredDict(self.data, self, [])
File "/home/user/wspace/electrum/electrum/json_db.py", line 117, in __init__
self.__setitem__(k, v)
File "/home/user/wspace/electrum/electrum/json_db.py", line 49, in wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/json_db.py", line 135, in __setitem__
v = self.db._convert_dict(self.path, key, v)
File "/home/user/wspace/electrum/electrum/json_db.py", line 247, in _convert_dict
v = dict((k, constructor(**x)) for k, x in v.items())
File "/home/user/wspace/electrum/electrum/json_db.py", line 247, in <genexpr>
v = dict((k, constructor(**x)) for k, x in v.items())
TypeError: ImportedChannelBackupStorage.__init__() missing 1 required positional argument: 'local_payment_pubkey'
```
Without this,
`$ python3 -m unittest electrum.tests.regtest.TestUnixSockets.test_unixsockets`
was failing on my machine but succeeding on CI, due to timing differences.
triggered a payment forwarding.
Final onions may trigger a payment forwarding, through the callback
returned by maybe_fulfill_htlc. In that case, we should not fail the
HTLC later; doing so might result in fund loss.
Remove test_simple_payment_with_hold_invoice_timing_out: once we
have accepted to forward a payment HTLC with a hold invoice, we
do not want to time it out, for the same reason.
- maybe_fulfill_htlc returns a forwarding callback that
covers both cases.
- previously, the callback of hold invoices was called as a
side-effect of lnworker.check_mpp_status.
- the same data structures (lnworker.trampoline_forwardings,
lnworker.trampoline_forwarding_errors) are used for both
trampoline forwardings and hold invoices.
- maybe_fulfill_htlc still recursively calls itself to perform
checks on trampoline onion. This is ugly, but ugliness is now
contained to that method.
- fix parameters passed to maybe_forward_trampoline
- use lnworker.trampoline_forwardings as a semaphore for ongoing
trampoline payments
- if a trampoline payment fails, fail all received HTLCs
This clears up log spam for regtest tests.
related:
- https://bugs.python.org/issue44665
- https://github.com/python/cpython/issues/88831
- https://textual.textualize.io/blog/2023/02/11/the-heisenbug-lurking-in-your-async-code/
- https://github.com/python/cpython/issues/91887#issuecomment-1434816045
- "Task was destroyed but it is pending!"
Perhaps we should inspect all our usages of
- asyncio.create_task
- loop.create_task
- asyncio.ensure_future
- asyncio.run_coroutine_threadsafe
?
Example log for running a regtest test:
```
$ python3 -m unittest electrum.tests.regtest.TestLightningAB.test_collaborative_close
***** test_collaborative_close ******
initializing alice
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
return fmt.format(record)
File "/home/user/wspace/electrum/electrum/logging.py", line 44, in format
record = copy.copy(record) # avoid mutating arg
File "/usr/lib/python3.10/copy.py", line 92, in copy
rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
File "/usr/lib/python3.10/asyncio/base_events.py", line 1781, in call_exception_handler
self._exception_handler(self, context)
File "/home/user/wspace/electrum/electrum/util.py", line 1535, in on_exception
loop.default_exception_handler(context)
File "/usr/lib/python3.10/asyncio/base_events.py", line 1744, in default_exception_handler
logger.error('\n'.join(log_lines), exc_info=exc_info)
Message: "Task was destroyed but it is pending!\ntask: <Task pending name='Task-2' coro=<Abstract_Wallet.on_event_adb_set_up_to_date() running at /home/user/wspace/electrum/electrum/wallet.py:485> wait_for=<Future finished result=0> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.10/asyncio/futures.py:392]>"
Arguments: ()
[--- SNIP --- more of the same --- SNIP ---]
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
return fmt.format(record)
File "/home/user/wspace/electrum/electrum/logging.py", line 44, in format
record = copy.copy(record) # avoid mutating arg
File "/usr/lib/python3.10/copy.py", line 92, in copy
rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
File "/usr/lib/python3.10/asyncio/base_events.py", line 1781, in call_exception_handler
self._exception_handler(self, context)
File "/home/user/wspace/electrum/electrum/util.py", line 1535, in on_exception
loop.default_exception_handler(context)
File "/usr/lib/python3.10/asyncio/base_events.py", line 1744, in default_exception_handler
logger.error('\n'.join(log_lines), exc_info=exc_info)
Message: "Task was destroyed but it is pending!\ntask: <Task pending name='Task-31' coro=<Abstract_Wallet.on_event_adb_set_up_to_date() running at /home/user/wspace/electrum/electrum/wallet.py:485> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.10/asyncio/futures.py:385, Task.task_wakeup()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.10/asyncio/futures.py:392]>"
Arguments: ()
true
true
true
true
funding alice
```
scenario:
- user opens a lightning channel and exports an "imported channel backup"
- user closes channel via local-force-close
- local ctx is published, to_local output has user's funds and they are CSV-locked for days
- user restores wallet file from seed and imports channel backup
- new wallet file should be able to sweep coins from to_local output (after CSV expires)
This was not working previously, as the local_payment_basepoint was not included in the
imported channel backups, and the code was interpreting the lack of this as the channel not
having option_static_remotekey enabled. This resulted in lnutil.extract_ctn_from_tx
using an incorrect funder_payment_basepoint, and lnsweep not recognising the ctx due to
the garbage ctn value.
The imported channel backup serialisation format is slightly changed to include the
previously missing field, and its version number is bumped (0->1). We allow importing
both version 0 and version 1 backups, however v0 backups cannot handle the above
described scenario (they can only be used to request a remote-force-close).
Note that we were/are setting the missing local_payment_basepoint to the pubkey of
one of the wallet change addresses, which is bruteforceable if necessary, but I
think it is not worth the complexity to add this bruteforce logic. Also note
that the bruteforcing could only be done after the local-force-close was broadcast.
Ideally people with existing channels and already exported v0 backups should re-export
v1 backups... Not sure how to handle this.
closes https://github.com/spesmilo/electrum/issues/8516
SomberNight
ThomasV
Sander van Grieken
3rd Iteration