Old wallet format isn't used for years and script is broken since
removal of pyaes dependency. If somebody still needs it, he can use
older JoinMarket version to do conversion.
Prior to this commit, code for scalar multiplication the module
secp256k1_main in jmbitcoin relied on direct access to the secp256k1
linked library, but the API for accessing that object has now changed in
python-bitcointx with version 1.1.5 (there is now a Secp256k1 data class
with lib and ctx entries, see the Readme of the project for details).
After this commit, we update our code for that API (but do not make
functional changes to Joinmarket itself).
Prior to this commit, while the result of the gettransaction rpc call
was being cached so as to not have to repeat these rpc calls, the
deserialized form of the transaction created by a call to
CMutableTransaction.deserialize, was not, and since this call is rather
expensive, the history method was running more slowly than needed.
After this commit, we cache the deserialized form also, resulting in a
speedup to the wallet_utils.wallet_fetch_history method.
Prior to this commit, the calls to get_new_addr in the functions in
the initial sync algo used for recovery, used the default value of the
argument validate_cache, which is True (because in normal running,
get_new_addr is used to derive addresses as destinations, for which it's
safer to not use the cache, and as one-off calls, are not
performance-sensitive). This caused initial sync to be very slow in
recovery, especially if using large gap limits (which is common).
After this commit, we set the argument validate_cache to False, as is
intended during initial sync. This allows the optimised performance from
caching to be in effect. See earlier PRs 1594 and 1614 for context.
Fixes#1614.
Prior to this commit, if data in the persisted cache in the wallet file
were wrong (should be a very extraordinary case), then the joinmarket
code would have to crash with a cache invalid warning. After this
commit, in such an extraordinary case, the option exists to invalidate
or remove the cache on startup, so that it can be rebuilt from scratch.
This is done with a config var wallet_caching_disabled in the POLICY
section.
Add a validate_cache parameter to the five principal caching methods:
- _get_key_from_path
- _get_keypair_from_path
- _get_pubkey_from_path
- get_script_from_path
- get_address_from_path
and to the five convenience methods that wrap the above:
- get_script
- get_addr
- script_to_addr
- get_new_script
- get_new_addr
The value of this new parameter defaults to False in all but the last
two methods, where we are willing to sacrifice speed for the sake of
extra confidence in the correctness of *new* scripts and addresses to
be used for new deposits and new transactions.
Deriving private keys from BIP32 paths, public keys from private keys,
scripts from public keys, and addresses from scripts are some of the
most CPU-intensive tasks the wallet performs. Once the wallet inevitably
accumulates thousands of used paths, startup times become painful due to
needing to re-derive these data items for every used path in the wallet
upon every startup. Introduce a persistent cache to avoid the need to
re-derive these items every time the wallet is opened.
Introduce _get_keypair_from_path and _get_pubkey_from_path methods to
allow cached public keys to be used rather than always deriving them on
the fly.
Change many code paths that were calling CPU-intensive methods of
BTCEngine so that instead they call _get_key_from_path,
_get_keypair_from_path, _get_pubkey_from_path, get_script_from_path,
and/or get_address_from_path, all of which can take advantage of the new
cache.
Hoist _populate_script_map from BIP32Wallet into BaseWallet, rename it
to _populate_maps, and have it populate the new _addr_map in addition to
the existing _script_map. Have the constructor of each concrete wallet
subclass pass to _populate_maps the paths it contributes. Additionally,
do not implement yield_known_paths by iterating over _script_map, but
rather have each wallet subclass contribute its own paths to the
generator returned by yield_known_paths.
The algorithm in get_imported_privkey_branch was O(m*n): for each
imported path, it was iterating over the entire set of UTXOs. Rewrite
the algorithm to make one pass over the set of UTXOs up front to compute
the balance of each script (O(m)) and then, separately, one pass over
the set of imported paths to pluck out the balance for each path (O(n)).
Rather than evaluating wallet_service.get_utxos_by_mixdepth()[md],
instead evaluate wallet_service.get_utxos_at_mixdepth(md). This way
we're not computing a bunch of data that we'll immediately discard.
Sometimes calling code is only interested in the balance or UTXOs at a
single mixdepth. In these cases, it is wasteful to get the balance or
UTXOs at all mixdepths, only to throw away the returned information
about all but the single mixdepth of interest. Implement new methods in
BaseWallet to get the balance or UTXOs at a single mixdepth.
Also, correct an apparent oversight due to apparently misplaced
indentation: the maxheight parameter of get_balance_by_mixdepth was
ignored unless the include_disabled parameter was passed as False. It
appears that the intention was for include_disabled and maxheight to be
independent filters on the returned information.
utxo_d = []
for k, v in disabled.items():
utxo_d.append(k)
{'frozen': True if u in utxo_d else False}
The above was inefficient. Replace with:
{'frozen': u in disabled}
Checking for existence of a key in a dict takes time proportional to
O(1), whereas checking for existence of an element in a list takes time
proportional to O(n).
Kristaps Kaupe
Matt Whitlock
spacebear
Adam Gibson