Plugins.stop was never called, so the Plugins thread only stopped
because of the is_running() check in run(), which triggers too late:
the Plugins thread was stopping after the main thread stopped.
E.g. playing around in the qt wizard with wallet creation for a Trezor,
and closing the wizard (only window):
``` 24.85 | E | p/plugin.Plugins |
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/util.py", line 386, in run_jobs
job.run()
File "/home/user/wspace/electrum/electrum/plugin.py", line 430, in run
client.timeout(cutoff)
File "/home/user/wspace/electrum/electrum/plugin.py", line 363, in wrapper
return run_in_hwd_thread(partial(func, *args, **kwargs))
File "/home/user/wspace/electrum/electrum/plugin.py", line 355, in run_in_hwd_thread
fut = _hwd_comms_executor.submit(func)
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
```
Without this,
`$ python3 -m unittest electrum.tests.regtest.TestUnixSockets.test_unixsockets`
was failing on my machine but succeeding on CI, due to timing differences.
- standardize_path is made more lenient: it no longer calls os.path.realpath,
as that was causing issues on Windows with some mounted drives
- daemon._wallets is still keyed on the old strict standardize_path, but
filesystem operations in WalletStorage use the new lenient standardize_path.
- daemon._wallets is strict to forbid opening the same logical file twice concurrently
- fs operations however work better on the non-resolved paths, so they use those
closes https://github.com/spesmilo/electrum/issues/8495
A new config API is introduced, and ~all of the codebase is adapted to it.
The old API is kept but mainly only for dynamic usage where its extra flexibility is needed.
Using examples, the old config API looked this:
```
>>> config.get("request_expiry", 86400)
604800
>>> config.set_key("request_expiry", 86400)
>>>
```
The new config API instead:
```
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS
604800
>>> config.WALLET_PAYREQ_EXPIRY_SECONDS = 86400
>>>
```
The old API operated on arbitrary string keys, the new one uses
a static ~enum-like list of variables.
With the new API:
- there is a single centralised list of config variables, as opposed to
these being scattered all over
- no more duplication of default values (in the getters)
- there is now some (minimal for now) type-validation/conversion for
the config values
closes https://github.com/spesmilo/electrum/pull/5640
closes https://github.com/spesmilo/electrum/pull/5649
Note: there is yet a third API added here, for certain niche/abstract use-cases,
where we need a reference to the config variable itself.
It should only be used when needed:
```
>>> var = config.cv.WALLET_PAYREQ_EXPIRY_SECONDS
>>> var
<ConfigVarWithConfig key='request_expiry'>
>>> var.get()
604800
>>> var.set(3600)
>>> var.get_default_value()
86400
>>> var.is_set()
True
>>> var.is_modifiable()
True
```
B023 Function definition does not bind loop variable 'already_selected_buckets_value_sum'
in keepkey/qt.py, looks like this was an actual bug
(fixed in trezor plugin already: 52a4810752 )
The qt, qml, and kivy GUIs have a first-start network-setup screen
that allows the user customising the network settings before creating a wallet.
Previously the daemon used to create the network and start it, before this screen,
before the GUI even starts. If the user changed network settings, those would
be set on the already running network, potentially including restarting the network.
Now it becomes the responsibility of the GUI to start the network, allowing this
first-start customisation to take place before starting the network at all.
The qt and the qml GUIs are adapted to make use of this. Kivy, and the other
prototype GUIs are not adapted and just start the network right away, as before.
During wallet-open, we load all invoices/payreqs. This involved decoding the lnaddrs twice.
Now we only decode once.
For a wallet with ~1000 payreqs, this noticeably sped up wallet-open:
(before:)
8.83 | D | util.profiler | Daemon._load_wallet 6.4317 sec
(after:)
5.69 | D | util.profiler | Daemon._load_wallet 3.4450 sec
It is very expensive to parse all the lnaddrs...
old traceback:
```
$ ./run_electrum --testnet -o setconfig rpchost qweasdfcsdf
$ ./run_electrum --testnet -o setconfig rpcport 7777
$ ./run_electrum --testnet daemon
E | daemon.Daemon | taskgroup died.
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/daemon.py", line 419, in _run
async with self.taskgroup as group:
File "/home/user/wspace/electrum/packages/aiorpcx/curio.py", line 297, in __aexit__
await self.join()
File "/home/user/wspace/electrum/electrum/util.py", line 1335, in join
task.result()
File "/home/user/wspace/electrum/electrum/daemon.py", line 281, in run
await site.start() #
File "/home/user/wspace/electrum/packages/aiohttp/web_runner.py", line 121, in start
self._server = await loop.create_server(
File "/usr/lib/python3.10/asyncio/base_events.py", line 1471, in create_server
infos = await tasks.gather(*fs)
File "/usr/lib/python3.10/asyncio/base_events.py", line 1408, in _create_server_getaddrinfo
infos = await self._ensure_resolved((host, port), family=family,
File "/usr/lib/python3.10/asyncio/base_events.py", line 1404, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
File "/usr/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
```
If running in daemon mode, and daemon.taskgroup died, we were not stopping.
Instead, we should gracefully stop and exit the process.
To reproduce:
```
$ ./run_electrum --testnet -o setconfig rpcport 1
$ ./run_electrum --testnet daemon -v
```
debian packager would like to replace vendored libs with system provided ones
(using symlinks). This requires "follow_symlinks=True".
discussion of security implications: https://serverfault.com/q/244592
to minimise attack surface, we only set this option for the "vendor/" directory.
related: https://github.com/spesmilo/electrum/issues/8023
Note in particular that check_password_for_directory was not safe to use while the daemon had wallets loaded,
as the same file would have two corresponding Wallet() instances in memory. This was specifically handled in
the kivy GUI, on the caller side, by stopping-before and reloading-after the wallets; but it was dirty to
have the caller handle this.
asyncio.get_event_loop() became deprecated in python3.10. (see https://github.com/python/cpython/issues/83710)
```
.../electrum/electrum/daemon.py:470: DeprecationWarning: There is no current event loop
self.asyncio_loop = asyncio.get_event_loop()
.../electrum/electrum/network.py:276: DeprecationWarning: There is no current event loop
self.asyncio_loop = asyncio.get_event_loop()
```
Also, according to that thread, "set_event_loop() [... is] not deprecated by oversight".
So, we stop using get_event_loop() and set_event_loop() in our own code.
Note that libraries we use (such as the stdlib for python <3.10), might call get_event_loop,
which then relies on us having called set_event_loop e.g. for the GUI thread. To work around
this, a custom event loop policy providing a get_event_loop implementation is used.
Previously, we have been using a single asyncio event loop, created with
util.create_and_start_event_loop, and code in many places got a reference to this loop
using asyncio.get_event_loop().
Now, we still use a single asyncio event loop, but it is now stored as a global in
util._asyncio_event_loop (access with util.get_asyncio_loop()).
I believe these changes also fix https://github.com/spesmilo/electrum/issues/5376
aiorpcx 0.20 changed the behaviour/API of TaskGroups.
When used as a context manager, TaskGroups no longer propagate
exceptions raised by their tasks. Instead, the calling code has
to explicitly check the results of tasks and decide whether to re-raise
any exceptions.
This is a significant change, and so this commit introduces "OldTaskGroup",
which should behave as the TaskGroup class of old aiorpcx. All existing
usages of TaskGroup are replaced with OldTaskGroup.
closes https://github.com/spesmilo/electrum/issues/7446
The daemon by default listens for RPC over a socket.
Before this change, this socket was a TCP/IP network socket (over localhost),
but now we change it to be a unix domain socket by default (if available).
This socket is used not only when sending commands over JSON-RPC,
but also when using the CLI, and to a tiny extent even used when running the GUI.
The type of socket used (and hence this change) is invisible to CLI and GUI users.
JSON-RPC users might want to use either the --rpcsock CLI flag
or set the "rpcsock" config key (to "tcp") to switch back to using a TCP/IP socket.
In practice it seems unix domain sockets are available on
all mainstream OSes/platforms, except for Windows.
closes https://github.com/spesmilo/electrum/issues/7553
If daemon.taskgroup dies
- in GUI mode, show a crash reporter window to the user,
instead of immediately stopping the whole process.
- in daemon mode, log exception and stop process, as before.
- trampoline is enabled by default in config, to prevent download of `gossip_db`.
(if disabled, `gossip_db` will be downloaded, regardless of the existence of channels)
- if trampoline is enabled:
- the wallet can only open channels with trampoline nodes
- already-existing channels with non-trampoline nodes are frozen for sending.
- there are two types of trampoline payments: legacy and end-to-end (e2e).
- we decide to perform legacy or e2e based on the invoice:
- we use trampoline_routing_opt in features to detect Eclair and Phoenix invoices
- we use trampoline_routing_hints to detect Electrum invoices
- when trying a legacy payment, we add a second trampoline to the path to preserve privacy.
(we fall back to a single trampoline if the payment fails for all trampolines)
- the trampoline list is hardcoded, it will remain so until `trampoline_routing_opt` feature flag is in INIT.
- there are currently only two nodes in the hardcoded list, it would be nice to have more.
- similar to Phoenix, we find the fee/cltv by trial-and-error.
- if there is a second trampoline in the path, we use the same fee for both.
- the final spec should add fee info in error messages, so we will be able to fine-tune fees
SomberNight
ThomasV
Federico
sgmoore
yanmaani
bitromortac
MrNaif2018