This is a regression from 7ca89f56ee, which introduced StoredList.
The newly added test was failing without the change.
```
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/gui/qt/main_window.py", line 1786, in new_contact_dialog
self.set_contact(line2.text(), line1.text())
File "/home/user/wspace/electrum/electrum/gui/qt/main_window.py", line 1435, in set_contact
self.contacts[address] = ('address', label)
File "/home/user/wspace/electrum/electrum/contacts.py", line 75, in __setitem__
self.save()
File "/home/user/wspace/electrum/electrum/contacts.py", line 62, in save
self.db.put('contacts', dict(self))
File "/home/user/wspace/electrum/electrum/json_db.py", line 42, in wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/json_db.py", line 318, in put
self.data[key] = copy.deepcopy(value)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object
```
The following exceptions should be expected:
FileNotFoundError: given wallet path does not exist
StorageReadWriteError: given file is not readable/writable or containing folder is not writable
InvalidPassword: wallet requires a password but no password or an invalid password was given
WalletFileException: any internal wallet data issue. specific subclasses can be caught separately:
- WalletRequiresSplit: wallet needs splitting (split_data passed in Exception)
- WalletRequiresUpgrade: wallet needs upgrade, and no upgrade=True was passed to load_wallet
- WalletUnfinished: wallet file contains an action and needs additional information to finalize. (WalletDB passed in exception)
Removed qml/qewalletdb.py
This patch also fixes load_wallet calls in electrum/scripts and adds a qml workaround for dialogs opening and closing so
fast that the dialog opened==true property change is missed (which we need to manage the dialog/page stack)
- @amount_msat.validator prevents the creation of invoices with e.g. too large amounts
- however the qml gui is mutating invoices by directly setting the `amount_msat` field,
and it looks like attrs validators only run during init.
We can use `on_setattr` (introduced in attrs==20.1.0).
- a wallet db upgrade is added to rm existing insane invoices
- btw the qml gui was already doing its own input validation on the textedit
(see qeconfig.btcAmountRegex). however that only limits the input to not have more
chars than what is needed to represent 21M BTC (e.g. you can still enter 99M BTC,
which the invoice logic does not tolerate later on - but is normally caught).
fixes https://github.com/spesmilo/electrum/issues/8582
follow-up https://github.com/spesmilo/electrum/pull/8536
This replaces 69336befee, which was insufficient.
#8536 added a new field into the struct, which older versions do not ignore but raise:
opening a wallet file with new code updated the struct to include it,
after which old code could no longer open the wallet file.
i.e. #8536 was an invisible wallet upgrade, breaking compat.
This commit simply formalises the wallet upgrade: old code will now show
an understandable error when trying to open new files.
decorators (instead of overloading JsonDB._convert_dict and
_convert_value)
- stored_in for elements of a StoreDict
- stored_as for singletons
- extra register methods are defined for key conversions
This commit was adapted from the jsonpatch branch
- case 1: in version 4.4.1, 4.4.2, the qml GUI wizard allowed creating multisig wallets with an old_mpk as cosigner.
- case 2: in version 4.4.0, 4.4.1, 4.4.2, the qml GUI wizard allowed creating multisig wallets with mixed xpub/Ypub/Zpub.
The corresponding missing input validation was a bug in the wizard, it was unintended behaviour. Validation was added in d2cf21fc2b. Note however that there might be users who created such wallet files.
Re case 1 wallet files: there is no version of Electrum that allows spending from such a wallet. Coins received at addresses are not burned, however it is technically challenging to spend them. (unless the multisig can spend without needing the old_mpk cosigner in the quorum).
Re case 2 wallet files: it is possible to create a corresponding spending wallet for such a multisig, however it is a bit tricky. The script type for the addresses in such a heterogeneous xpub wallet is based on the xpub_type of the first keystore. So e.g. given a wallet file [Yprv1, Zpub2] it will have sh(wsh()) scripts, and the cosigner should create a wallet file [Ypub1, Zprv2] (same order).
Technically case 2 wallet files could be "fixed" automatically by converting the xpub types as part of a wallet_db upgrade. However if the wallet files also contain seeds, those cannot be converted ("standard" vs "segwit" electrum seed).
Case 1 wallet files are not possible to "fix" automatically as the cosigner using the old_mpk is not bip32 based.
It is unclear if there are *any* users out there affected by this. I suspect for case 1 it is very likely there are none (not many people have pre-2.0 electrum seeds which were never supported as part of a multisig who would also now try to create a multisig using them); for case 2 however there might be.
This commit breaks both case 1 and case 2 wallets: these wallet files can no longer be opened in new Electrum, an error message is shown and the crash reporter opens. If any potential users opt to send crash reports, at least we will know they exist and can help them recover.
Replace get_key_for_outgoing_invoice, get_key_for_incoming_request
with Invoice.get_id()
When a new request is created, reuse addresses of expired requests (fixes#7927)
The API is changed for the following commands:
get_request, get_invoice,
list_requests, list_invoices,
delete_request, delete_invoice
get_unused_addresses() has been broken since #7730, because
addresses are considered as permanently used if they are in
the list of keys of receive_requests. This is true even if
an address is used as fallback for a lightning payment. This
means that the number of lightning payments we can receive
is constrained by the gap limit.
If a payment succeeds off-chain, we want to be able to reuse
its fallback address in other requests (this does not reduce
privacy, because invoices already share the same public key).
This implies that we should not use the onchain address as key
for lightning-enabled requests in wallet.receive_requests. If
we did, paid invoices would be overwritten when the address is
reused. That is the reason for the wallet_db upgrade.
Related: a3faf85e3c
There is no point in adding new hww types to these lists every time support for a new hww is added.
These upgrades got released in 2.7.0 and any hw types added after are unrelated.
reverted to just-after last relevant change:
c820423b00
Change convert_version_25 to delete invoices instead of converting them.
convert_version_25 was released ~2 years ago. Wallet files not opened since will have old bip70 invoices deleted upon upgrading.
In general it is ~unsafe for convert_version_* to depend on other modules of the code.
(using e.g. sha256 is fine as its API will never change,
but using e.g. PaymentRequest is dangerous as its API might change over time)
I still have a mainnet wallet with some pre-static-remotekey channels
(though those channels are closed) that I do not want to delete yet.
follow-up https://github.com/spesmilo/electrum/pull/7636
```
E | gui.qt.exception_window.Exception_Hook | exception caught by crash reporter
Traceback (most recent call last):
File "...\electrum\electrum\gui\qt\__init__.py", line 307, in wrapper
return func(self, *args, **kwargs)
File "...\electrum\electrum\gui\qt\__init__.py", line 332, in start_new_window
wallet = self._start_wizard_to_select_or_create_wallet(path)
File "...\electrum\electrum\gui\qt\__init__.py", line 377, in _start_wizard_to_select_or_create_wallet
db = WalletDB(storage.read(), manual_upgrades=False)
File "...\electrum\electrum\wallet_db.py", line 73, in __init__
self.load_data(raw)
File "...\electrum\electrum\wallet_db.py", line 106, in load_data
self.upgrade()
File "...\electrum\electrum\util.py", line 439, in <lambda>
return lambda *args, **kw_args: do_profile(args, kw_args)
File "...\electrum\electrum\util.py", line 435, in do_profile
o = func(*args, **kw_args)
File "...\electrum\electrum\wallet_db.py", line 195, in upgrade
self._convert_version_44()
File "...\electrum\electrum\wallet_db.py", line 859, in _convert_version_44
if item['static_remotekey_enabled']:
KeyError: 'static_remotekey_enabled'
```
This is a regression from 64a931f21e,
which introduced "onchain_channel_backups", and renamed
the old "channel_backups" key to "imported_channel_backups".
The `save_backup` method was not changed to use the new "imported_channel_backups" key,
so the channel backups are in the backup file but they are ignored.
- use_recoverable_channel is a user setting, available
only in standard wallets with a 'segwit' seed_type
- if enabled, 'lightning_xprv' is derived from seed
- otherwise, wallets use the existing 'lightning_privkey2'
Recoverable channels:
- channel recovery data is added funding tx using an OP_RETURN
- recovery data = 4 magic bytes + node id[0:16]
- recovery data is chacha20 encrypted using funding_address as nonce.
(this will allow to fund multiple channels in the same tx)
GUI:
- whether channels are recoverable is shown in wallet info dialog.
- if the wallet can have recoverable channels but has an old node_id,
users are told to close their channels and restore from seed
to have that feature.
bump_fee was returning an invalid tx if its input was a
PartialTransaction that had signatures. It was relying on
line 1441 to remove signatures.
Relatedly, the WalletDB used to store such PartialTransactions as
PartialTransaction objects, but only until the program was restarted.
This is because serialising and de-serialising such a tx results in a
Transaction object.
So, combining these two, to reproduce a bug:
- create a tx, sign it, save as local
- bump fee, sign it, save as local
- bump fee --> tx already signed!? --> has old sigs, so it is invalid
The standard json module has an optimized C encoder, but that doesn't
currently support indentation. So if you request indentation, it falls
back on the slower Python encoder.
Readability doesn't matter for encrypted wallets, so this disables
indentation when the wallet is encrypted.
-----
based on b2399b6a3e
For a large encrypted wallet, compare:
before change:
JsonDB.dump 1.3153 sec
zlib.compress 1.281 sec
ECPubkey.encrypt_message 0.1744 sec
after change:
JsonDB.dump 0.5059 sec
zlib.compress 1.3120 sec
ECPubkey.encrypt_message 0.1630 sec
Co-authored-by: SomberNight <somber.night@protonmail.com>
SomberNight
ThomasV
Sander van Grieken
bitromortac
Benoit Verret
Malcolm Smith