When called via jsonrpc (but not via cli) with non-string amounts,
there could be a rounding error resulting in sending 1 sat less.
example:
```
$ ./run_electrum --testnet -w ~/.electrum/testnet/wallets/test_segwit_2 paytomany '[["tb1q6k5h4cz6ra8nzhg90xm9wldvadgh0fpttfthcg", 0.00033389]]' --fee 0
02000000000101b9e6018acb16952e3c9618b069df404dc85544eda8120e5f6e7cd7e94ce5ae8d0100000000fdffffff02fd8100000000000016001410c5b97085ec1637a9f702852f5a81f650fae1566d82000000000000160014d5a97ae05a1f4f315d0579b6577daceb5177a42b024730440220251d2ce83f6e69273de8e9be8602fbcf72b9157e1c0116161fa52f7e04db6e4302202d84045cc6b7056a215d1db3f59884e28dadd5257e1a3960068f90df90b452d1012102b0eff3bf364a2ab5effe952cba33521ebede81dac88c71951a5ed598cb48347b3a022500
$ curl --data-binary '{"id":"curltext","method":"paytomany","params":{"outputs":[["tb1q6k5h4cz6ra8nzhg90xm9wldvadgh0fpttfthcg", 0.00033389]], "fee": 0, "wallet": "/home/user/.electrum/testnet/wallets/test_segwit_2"}}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "result": "02000000000101b9e6018acb16952e3c9618b069df404dc85544eda8120e5f6e7cd7e94ce5ae8d0100000000fdffffff02fe8100000000000016001410c5b97085ec1637a9f702852f5a81f650fae1566c82000000000000160014d5a97ae05a1f4f315d0579b6577daceb5177a42b0247304402206ef66b845ca298c14dc6e8049cba9ed19db1671132194518ce5d521de6f5df8802205ca4b1aee703e3b98331fb9b88210917b385560020c8b2a8a88da38996b101c4012102b0eff3bf364a2ab5effe952cba33521ebede81dac88c71951a5ed598cb48347b39022500"}
```
^ note that first tx has output for 0.00033389, second tx has output for 0.00033388
fixes https://github.com/spesmilo/electrum/issues/8274
fixes https://github.com/spesmilo/electrum/issues/8240#8240 was triggering an AssertionError in wallet.get_invoice_status,
as code there was assuming conf >= 0. To trigger, force-close
a LN channel, and while the sweep is waiting on the CSV, try to
make a payment in the Send tab to the ismine change address used
for the sweep in the future_tx. (order of events can also be reversed)
ShortIDs were originally designed for lightning channels, and are now
understood by some block explorers.
This allows to remove one column in the UTXO tab (height is redundant).
In the transaction dialog, the space saving ensures that all inputs fit
into one line (it was not the case previously with p2wsh addresses).
For clarity and consistency, the ShortID is displayed for both inputs
and outputs in the transaction dialog.
With a PyCharm debugger attached, sometimes the python process is so
CPU-starved for me that create_and_start_event_loop() returned
before the event loop actually started, resulting in weird errors.
I guess this could happen even without a debugger attached on a
sufficiently slow CPU.
```
...\electrum\electrum\wallet.py:3580: in restore_wallet_from_text
wallet = Wallet(db, storage, config=config)
...\electrum\electrum\wallet.py:3501: in __new__
wallet = WalletClass(db, storage, config=config)
...\electrum\electrum\wallet.py:3345: in __init__
Deterministic_Wallet.__init__(self, db, storage, config=config)
...\electrum\electrum\wallet.py:3135: in __init__
self.synchronize()
...\electrum\electrum\wallet.py:3283: in synchronize
count += self.synchronize_sequence(False)
...\electrum\electrum\wallet.py:3267: in synchronize_sequence
self.create_new_address(for_change)
...\electrum\electrum\wallet.py:3254: in create_new_address
self.adb.add_address(address)
...\electrum\electrum\address_synchronizer.py:213: in add_address
self.up_to_date_changed()
...\electrum\electrum\address_synchronizer.py:680: in up_to_date_changed
util.trigger_callback('adb_set_up_to_date', self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <electrum.util.CallbackManager object at 0x000002B1788AD6F0>
event = 'adb_set_up_to_date'
args = (<electrum.address_synchronizer.AddressSynchronizer object at 0x000002B17A687670>,)
def trigger_callback(self, event, *args):
"""Trigger a callback with given arguments.
Can be called from any thread. The callback itself will get scheduled
on the event loop.
"""
if self.asyncio_loop is None:
self.asyncio_loop = get_asyncio_loop()
> assert self.asyncio_loop.is_running(), "event loop not running"
E AssertionError: event loop not running
...\electrum\electrum\util.py:1734: AssertionError
```
Always use "." as decimal point, and " " as thousands separator.
Previously,
- for decimal point, we were using
- "." in some places (e.g. AmountEdit, most fiat amounts), and
- `locale.localeconv()['decimal_point']` in others.
- for thousands separator, we were using
- "," in some places (most fiat amounts), and
- " " in others (format_satoshis)
I think it is better to be consistent even if whatever we pick differs from the locale.
Using whitespace for thousands separator (vs comma) is probably less confusing for people
whose locale would user "." for ts and "," for dp (as in e.g. German).
The alternative option would be to always use the locale. Even if we decide to do that later,
this refactoring should be useful.
closes https://github.com/spesmilo/electrum/issues/2629
We would reject bip21 URIs that contained both an "address=" and a "lightning=" key with a bolt11 invoice,
where the bolt11 invoice did not contain a fallback address.
fixes https://github.com/spesmilo/electrum/issues/7952
see in particular https://github.com/spesmilo/electrum/issues/7952#issuecomment-1227225602
> So the issue is with the aiorpcx monkey patch in util.py, as it
> relies on side-effecting the asyncio.Task, and it patches Task.cancel.
> However, aiohttp also uses Task.cancel for its own timeouting of the
> http request, with the same Task object, and this confuses timeout_after.
> Ultimately FxThread.run exits.
related https://github.com/kyuupichan/aiorpcX/pull/47
---
note: I am not content at all with this monkey-patching approach,
but at the same time I don't see how to properly fix this handling all
edge-cases in aiorpcx.
python 3.11 is finally adding an implementation of TaskGroup [0] and
an async timeout context manager [1] in the asyncio module of the stdlib.
Looking at the implementations, they look unfeasible to backport:
much of the implementation of asyncio.Task had to be changed for them
to work, and TaskGroup in particular relies on the new ExceptionGroups.
Some of these edge cases we are battling with aiorpcx.curio look
difficult to fix without those stdlib changes...
Anyway, when we bump the min python to 3.11, I look forward to switching
to that code instead, and ripping this stuff out.
[0]: https://docs.python.org/3.11/library/asyncio-task.html#task-groups
[1]: https://docs.python.org/3.11/library/asyncio-task.html#asyncio.timeout
If keystore.check_password is called with some pw on a keystore that does not have a password set,
it now raises better exceptions: it should now always raise InvalidPassword, and with a nicer msg.
Previously the exc type would depend on the ks type.
Examples before change:
```
>>> wallet.keystore.check_password("asd")
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/keystore.py", line 580, in check_password
xprv = pw_decode(self.xprv, password, version=self.pw_hash_version)
File "/home/user/wspace/electrum/electrum/crypto.py", line 311, in pw_decode
plaintext_bytes = pw_decode_bytes(data, password, version=version)
File "/home/user/wspace/electrum/electrum/crypto.py", line 270, in pw_decode_bytes
data_bytes = bytes(base64.b64decode(data))
File "/usr/lib/python3.10/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
```
```
>>> wallet.keystore.check_password("asd")
Traceback (most recent call last):
s = aes_decrypt_with_iv(secret, iv, e)
File "/home/user/wspace/electrum/electrum/crypto.py", line 157, in aes_decrypt_with_iv
data = decryptor.update(data) + decryptor.finalize()
File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/ciphers/base.py", line 148, in finalize
data = self._ctx.finalize()
File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 193, in finalize
raise ValueError(
ValueError: The length of the provided data is not a multiple of the block length.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/gui/qt/console.py", line 254, in exec_command
result = eval(command, self.namespace, self.namespace)
File "<string>", line 1, in <module>
File "/home/user/wspace/electrum/electrum/keystore.py", line 248, in check_password
self.get_private_key(pubkey, password)
File "/home/user/wspace/electrum/electrum/keystore.py", line 267, in get_private_key
sec = pw_decode(self.keypairs[pubkey], password, version=self.pw_hash_version)
File "/home/user/wspace/electrum/electrum/crypto.py", line 311, in pw_decode
plaintext_bytes = pw_decode_bytes(data, password, version=version)
File "/home/user/wspace/electrum/electrum/crypto.py", line 271, in pw_decode_bytes
return _pw_decode_raw(data_bytes, password, version=version)
File "/home/user/wspace/electrum/electrum/crypto.py", line 255, in _pw_decode_raw
raise InvalidPassword() from e
electrum.util.InvalidPassword: Incorrect password
```
-----
Examples after change:
```
>>> wallet.keystore.check_password("asd")
Traceback (most recent call last):
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\keystore.py", line 68, in wrapper
return check_password_fn(self, password)
File "...\electrum\keystore.py", line 605, in check_password
xprv = pw_decode(self.xprv, password, version=self.pw_hash_version)
File "...\electrum\crypto.py", line 311, in pw_decode
plaintext_bytes = pw_decode_bytes(data, password, version=version)
File "...\electrum\crypto.py", line 267, in pw_decode_bytes
raise CiphertextFormatError("ciphertext not valid base64") from e
electrum.crypto.CiphertextFormatError: ciphertext not valid base64
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\gui\qt\console.py", line 254, in exec_command
result = eval(command, self.namespace, self.namespace)
File "<string>", line 1, in <module>
File "...\electrum\keystore.py", line 76, in wrapper
raise InvalidPassword("password given but keystore has no password") from e
electrum.util.InvalidPassword: password given but keystore has no password
```
```
>>> wallet.keystore.check_password("asd")
Traceback (most recent call last):
s = aes_decrypt_with_iv(secret, iv, e)
File "...\electrum\crypto.py", line 158, in aes_decrypt_with_iv
data = cipher.decrypt(data)
File "...\Python310\site-packages\Cryptodome\Cipher\_mode_cbc.py", line 246, in decrypt
raise ValueError("Data must be padded to %d byte boundary in CBC mode" % self.block_size)
ValueError: Data must be padded to 16 byte boundary in CBC mode
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\keystore.py", line 68, in wrapper
return check_password_fn(self, password)
File "...\electrum\keystore.py", line 272, in check_password
self.get_private_key(pubkey, password)
File "...\electrum\keystore.py", line 291, in get_private_key
sec = pw_decode(self.keypairs[pubkey], password, version=self.pw_hash_version)
File "...\electrum\crypto.py", line 311, in pw_decode
plaintext_bytes = pw_decode_bytes(data, password, version=version)
File "...\electrum\crypto.py", line 268, in pw_decode_bytes
return _pw_decode_raw(data_bytes, password, version=version)
File "...\electrum\crypto.py", line 249, in _pw_decode_raw
raise InvalidPassword() from e
electrum.util.InvalidPassword: Incorrect password
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\electrum\gui\qt\console.py", line 254, in exec_command
result = eval(command, self.namespace, self.namespace)
File "<string>", line 1, in <module>
File "...\electrum\keystore.py", line 76, in wrapper
raise InvalidPassword("password given but keystore has no password") from e
electrum.util.InvalidPassword: password given but keystore has no password
```
follow-up fe74e4b9c9
Just for sanity -- there could be a collision of class names (defined in separate modules).
```
>>> wallet.is_up_to_date.__qualname__
'Abstract_Wallet.is_up_to_date'
>>> wallet.is_up_to_date.__module__
'electrum.wallet'
```
and use that registration in the iterator.
That way, we do not invoke getattr on arbitrary methods, which
could trigger unwanted code execution (for example, if a method
is decorated with @property, getattr will execute its code).
asyncio.get_event_loop() became deprecated in python3.10. (see https://github.com/python/cpython/issues/83710)
```
.../electrum/electrum/daemon.py:470: DeprecationWarning: There is no current event loop
self.asyncio_loop = asyncio.get_event_loop()
.../electrum/electrum/network.py:276: DeprecationWarning: There is no current event loop
self.asyncio_loop = asyncio.get_event_loop()
```
Also, according to that thread, "set_event_loop() [... is] not deprecated by oversight".
So, we stop using get_event_loop() and set_event_loop() in our own code.
Note that libraries we use (such as the stdlib for python <3.10), might call get_event_loop,
which then relies on us having called set_event_loop e.g. for the GUI thread. To work around
this, a custom event loop policy providing a get_event_loop implementation is used.
Previously, we have been using a single asyncio event loop, created with
util.create_and_start_event_loop, and code in many places got a reference to this loop
using asyncio.get_event_loop().
Now, we still use a single asyncio event loop, but it is now stored as a global in
util._asyncio_event_loop (access with util.get_asyncio_loop()).
I believe these changes also fix https://github.com/spesmilo/electrum/issues/5376
fixes https://github.com/spesmilo/electrum/issues/7447
Consider this trace for 4.2.0:
```
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 332, in start_new_window
File "electrum/gui/qt/__init__.py", line 363, in _start_wizard_to_select_or_create_wallet
File "electrum/gui/qt/installwizard.py", line 302, in select_storage
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "electrum/gui/qt/__init__.py", line 426, in main
File "electrum/gui/qt/__init__.py", line 307, in wrapper
File "electrum/gui/qt/__init__.py", line 349, in start_new_window
File "electrum/util.py", line 504, in get_new_wallet_name
PermissionError: [Errno 1] Operation not permitted: '/Users/admin/Documents/Peach/MS'
```
Note that `get_new_wallet_name` (os.listdir) can raise OSError,
and we were calling that on the main entrypoint codepath without exception-handling.
We were also calling it in the fallback codepath without exception-handling.
i.e. the GUI errored out on every startup for affected users, and without CLI usage
it was not possible to recover.
aiorpcx 0.20 changed the behaviour/API of TaskGroups.
When used as a context manager, TaskGroups no longer propagate
exceptions raised by their tasks. Instead, the calling code has
to explicitly check the results of tasks and decide whether to re-raise
any exceptions.
This is a significant change, and so this commit introduces "OldTaskGroup",
which should behave as the TaskGroup class of old aiorpcx. All existing
usages of TaskGroup are replaced with OldTaskGroup.
closes https://github.com/spesmilo/electrum/issues/7446
I think this was originally needed due to incorrect management of group lifecycles,
which our current code is doing better.
also note that if we needed this, in newer aiorpcx, the name of
the field was ~changed from `_closed` to `joined`:
239002689a
SomberNight
Jan Sarenik
ThomasV
myxmaster
avirgovi
bitromortac
Siddhant Chawla