Add a validate_cache parameter to the five principal caching methods:
- _get_key_from_path
- _get_keypair_from_path
- _get_pubkey_from_path
- get_script_from_path
- get_address_from_path
and to the five convenience methods that wrap the above:
- get_script
- get_addr
- script_to_addr
- get_new_script
- get_new_addr
The value of this new parameter defaults to False in all but the last
two methods, where we are willing to sacrifice speed for the sake of
extra confidence in the correctness of *new* scripts and addresses to
be used for new deposits and new transactions.
4486b10798 Transaction virtual size must be rounded upwards (Kristaps Kaupe)
Pull request description:
Transaction virtual size must be rounded upwards.
Top commit has no ACKs.
Tree-SHA512: 9541ec8754d48aacdc10dc0cfb4323b60c4514f8f61a2d827ee7354d58cf46db530b4b68850f450a58e2053a9f69504fd80ca9f4d3614b7df24bc8d38f09a7fe
Deriving private keys from BIP32 paths, public keys from private keys,
scripts from public keys, and addresses from scripts are some of the
most CPU-intensive tasks the wallet performs. Once the wallet inevitably
accumulates thousands of used paths, startup times become painful due to
needing to re-derive these data items for every used path in the wallet
upon every startup. Introduce a persistent cache to avoid the need to
re-derive these items every time the wallet is opened.
Introduce _get_keypair_from_path and _get_pubkey_from_path methods to
allow cached public keys to be used rather than always deriving them on
the fly.
Change many code paths that were calling CPU-intensive methods of
BTCEngine so that instead they call _get_key_from_path,
_get_keypair_from_path, _get_pubkey_from_path, get_script_from_path,
and/or get_address_from_path, all of which can take advantage of the new
cache.
Hoist _populate_script_map from BIP32Wallet into BaseWallet, rename it
to _populate_maps, and have it populate the new _addr_map in addition to
the existing _script_map. Have the constructor of each concrete wallet
subclass pass to _populate_maps the paths it contributes. Additionally,
do not implement yield_known_paths by iterating over _script_map, but
rather have each wallet subclass contribute its own paths to the
generator returned by yield_known_paths.
The algorithm in get_imported_privkey_branch was O(m*n): for each
imported path, it was iterating over the entire set of UTXOs. Rewrite
the algorithm to make one pass over the set of UTXOs up front to compute
the balance of each script (O(m)) and then, separately, one pass over
the set of imported paths to pluck out the balance for each path (O(n)).
Rather than evaluating wallet_service.get_utxos_by_mixdepth()[md],
instead evaluate wallet_service.get_utxos_at_mixdepth(md). This way
we're not computing a bunch of data that we'll immediately discard.
Sometimes calling code is only interested in the balance or UTXOs at a
single mixdepth. In these cases, it is wasteful to get the balance or
UTXOs at all mixdepths, only to throw away the returned information
about all but the single mixdepth of interest. Implement new methods in
BaseWallet to get the balance or UTXOs at a single mixdepth.
Also, correct an apparent oversight due to apparently misplaced
indentation: the maxheight parameter of get_balance_by_mixdepth was
ignored unless the include_disabled parameter was passed as False. It
appears that the intention was for include_disabled and maxheight to be
independent filters on the returned information.
utxo_d = []
for k, v in disabled.items():
utxo_d.append(k)
{'frozen': True if u in utxo_d else False}
The above was inefficient. Replace with:
{'frozen': u in disabled}
Checking for existence of a key in a dict takes time proportional to
O(1), whereas checking for existence of an element in a list takes time
proportional to O(n).
1cb20d5ae6 Do not reinstall on test (roshii)
Pull request description:
Removing `pip install .[test]` step from test script. It does not make sense to reinstall everything when test are ran especially with editable install.
ACKs for top commit:
kristapsk:
re-ACK 1cb20d5ae6
Tree-SHA512: 7a35f1f7f2160b84e97819f0f91655c551e8343921f734a0092d3fdaad76d46afcf2f053ad55964120240314dced4557d55c44d299e4622a50b55851b4b5df73
4f4945e5c5 Test min and latest Python version only (roshii)
Pull request description:
Taking the same approach as `bitcoind` version testing, setting only minimum required and latest Python version in CI. Testing so many versions does not bring much added value IMHO.
ACKs for top commit:
kristapsk:
ACK 4f4945e5c5
Tree-SHA512: 7678783323828dad9e0899c557077344ba968cc716b6aeccc1f50a4a5eaa23d6b2bf89eb0a607c2e1c31a395e03b26b65187d8ae5386674fc512ae9790cc9560
638200d346 feat(rpc): add block height to session response (theborakompanioni)
Pull request description:
Adds the current block height to the `/session` response.
This can be useful for api clients to display a more sophisticated message on how long to wait for a given UTXO when it has less than 5 confirmations (for [sourcing commitments](https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/docs/SOURCING-COMMITMENTS.md#wait-for-at-least-5-confirmations)). `/session` is usually polled, so when the block height changes, additional information can be reloaded (instead of polling the `/wallet/{walletname}/utxos` endpoint).
The value is only included for authenticated requests when a wallet is loaded.
Additionally, block height changes could be pushed via websocket. However, this is not included in this PR.
ACKs for top commit:
kristapsk:
utACK 638200d346
Tree-SHA512: f5275a469a69678709cd88683463da155f9319dbc8cd64d159a942bcdb644f6e6f41b3f47a5bcd48eba66f17d27cd9fbf3126378aa3135ed5f9bc6fdeb5e7215
3e71df586b Fix ShellCheck warnings (Kristaps Kaupe)
027682ab2c CI: Add ShellCheck (Kristaps Kaupe)
Pull request description:
Could use `./test/lint/lint-shell.sh` instead, but this way was simpler.
https://github.com/marketplace/actions/shellcheck
ACKs for top commit:
roshii:
utACK 3e71df586b
Tree-SHA512: 4f98464b397088293ea0c7736e835d94b7079c53f0958c36aeca62d68a586fd69fa05988591dbbba51f64824ef58e5341cc7930217fba0f2b2b4aa53e3735080
438cb41c23 Replace readfp() (roshii)
Pull request description:
Replace `ConfigParser.readfp()` with `ConfigParser.read_file()` as per Python [docs](https://docs.python.org/3/library/configparser.html?highlight=configparser#configparser.ConfigParser.read_file)
The latter is available as from Python v3.2 while the former breaks as from v3.12
ACKs for top commit:
kristapsk:
ACK 438cb41c23
Tree-SHA512: 9f0c7b35a8410a49f4c7486d25535e8f4e47cc40f794c1e83d8b8ee070aa55fcf5a8e6424fd6ebf09c9978bd4e6259c9f26e7d17e2cac55745725bf44cd97917
1a8d0ea683 Correct help description for --develop (Kristaps Kaupe)
Pull request description:
#1484 changed the meaning of `--develop`.
Top commit has no ACKs.
Tree-SHA512: 162d6255ca6750a691a1b9fcb6897b6397fb0345c092bdceb88e2142f5429919fb73f2c7dd5f4d7a54576eb9f9d2409d00f0d33a9f4d458b60fdbd08e72336a1
70366ffede Bump cryptography to 41.0.4 for all platforms (Kristaps Kaupe)
Pull request description:
Remove conditions for old `cryptography` for 32-bit platforms. Back in a day it was pinned to v3.3.2, because newer versions introduced Rust as a dependency and 32-bit platforms don't have pre-built wheels. I think we should get rid of this hack - 1) not much people are running 32-bit OSes anymore (years ago default Raspberry Pi OS was 32-bit even for 64-bit boards, that's not true anymore), 2) none of developers actually tests stuff on these platforms and against such old `cryptography` versions, 3) it should be still possible to use JM with 32-bit archs, just local installation of Rust will be needed to build.
Also bump to v41.0.4, as v41.0.2 and v41.0.3 is statically linked with vulnerable versions of OpenSSL (although these vulnerabilities should not affect JM).
ACKs for top commit:
roshii:
utACK 70366ffede
Tree-SHA512: 57a75a21f38d0e793bafc89bacf5487131a4848e2a1fbd72c281e84ca5c64bc673a91d1484bb5fd4ec3c69d07a333a12bceb5cabe0e5eb76b62f9631a83b6732
83d7ebb40b Log in case JM loads RPC wallet at startup (Kristaps Kaupe)
Pull request description:
There are some cases when this operation can be slow, better log. Otherwise user might think JM just hanged up.
In my case I was doing some testing on ARM machine where I don't run JM everyday, so that wallet haven't been used for a long time and Core needed to do rescan for almost 8000 blocks.
ACKs for top commit:
roshii:
utACK 83d7ebb40b
Tree-SHA512: b605e5e9310113caf21f540a62c2604a80dcbc054515f85d747a782a7220153631e3e4593d1c05821f9a8dcb50602355129c2903d85dd15007c12dc8f6dbb4e2
71815129e3 Upgrade setuptools also with --docker-install (Kristaps Kaupe)
Pull request description:
This was missed in #1484.
ACKs for top commit:
roshii:
tACK 71815129e3
Tree-SHA512: 4e7d3fb139558d24992bf78b22b8a191fdbf99de14abc34adfd370b7bd2d3ff8b1ceaf1c6c2ccf1b000107e83c87420e73b4fc1cb1453883502f1e83f137737f
1ebb68f119 Update txtorcon to 23.0.0 (Kristaps Kaupe)
Pull request description:
Replaces #1558.
Changes:
* Drop Python2 support
* Fix a bug with stream updates (and CONTROLLER_WAIT)
https://github.com/meejah/txtorcon/releases/tag/v23.0.0
Top commit has no ACKs.
Tree-SHA512: 1a3cc28d05ab90cf1e33fcd5322b099e0674bffa5879a93330857c14113a32c71cad708f2c01e680a3229ad3aaea27c9baaf076ee9786d48db808d61412a5978
Kristaps Kaupe
Matt Whitlock
roshii
st3b1t
dependabot[bot]
theborakompanioni
Adam Gibson