Fixes#1614.
Prior to this commit, if data in the persisted cache in the wallet file
were wrong (should be a very extraordinary case), then the joinmarket
code would have to crash with a cache invalid warning. After this
commit, in such an extraordinary case, the option exists to invalidate
or remove the cache on startup, so that it can be rebuilt from scratch.
This is done with a config var wallet_caching_disabled in the POLICY
section.
Add a validate_cache parameter to the five principal caching methods:
- _get_key_from_path
- _get_keypair_from_path
- _get_pubkey_from_path
- get_script_from_path
- get_address_from_path
and to the five convenience methods that wrap the above:
- get_script
- get_addr
- script_to_addr
- get_new_script
- get_new_addr
The value of this new parameter defaults to False in all but the last
two methods, where we are willing to sacrifice speed for the sake of
extra confidence in the correctness of *new* scripts and addresses to
be used for new deposits and new transactions.
Deriving private keys from BIP32 paths, public keys from private keys,
scripts from public keys, and addresses from scripts are some of the
most CPU-intensive tasks the wallet performs. Once the wallet inevitably
accumulates thousands of used paths, startup times become painful due to
needing to re-derive these data items for every used path in the wallet
upon every startup. Introduce a persistent cache to avoid the need to
re-derive these items every time the wallet is opened.
Introduce _get_keypair_from_path and _get_pubkey_from_path methods to
allow cached public keys to be used rather than always deriving them on
the fly.
Change many code paths that were calling CPU-intensive methods of
BTCEngine so that instead they call _get_key_from_path,
_get_keypair_from_path, _get_pubkey_from_path, get_script_from_path,
and/or get_address_from_path, all of which can take advantage of the new
cache.
Hoist _populate_script_map from BIP32Wallet into BaseWallet, rename it
to _populate_maps, and have it populate the new _addr_map in addition to
the existing _script_map. Have the constructor of each concrete wallet
subclass pass to _populate_maps the paths it contributes. Additionally,
do not implement yield_known_paths by iterating over _script_map, but
rather have each wallet subclass contribute its own paths to the
generator returned by yield_known_paths.
Sometimes calling code is only interested in the balance or UTXOs at a
single mixdepth. In these cases, it is wasteful to get the balance or
UTXOs at all mixdepths, only to throw away the returned information
about all but the single mixdepth of interest. Implement new methods in
BaseWallet to get the balance or UTXOs at a single mixdepth.
Also, correct an apparent oversight due to apparently misplaced
indentation: the maxheight parameter of get_balance_by_mixdepth was
ignored unless the include_disabled parameter was passed as False. It
appears that the intention was for include_disabled and maxheight to be
independent filters on the returned information.
Prior to this commit, p2wsh inputs from fidelity bonds resulted in
miscalculation of transaction fees, even in cases where the exact set of
inputs were known (such as a direct send).
In this commit we change the estimation to a model in which the caller
of jmbitcoin.secp256k1_transaction.estimate_tx_size must specify a list
of types, one for each input to the transaction, and the same for
outputs. In some cases, the caller of the function uses the default
script type of the wallet, but in other cases where the caller can know
the exact types of each utxo used as input, and each destination used as
output, they are specified explicitly. In particular, the use of
fidelity bond outputs as input to transactions can be accounted for.
Currently this is only done in direct send payments; coinjoins still
fall back to assuming all inputs the same type (note that it is not
possible to use fidelity bond utxos as inputs to coinjoins).
Note also that the burn destination calculation in direct send is
removed, since it is not used, so the maintenance burden is best
avoided.
Fixes#1294.
Before this commit, calls to query_utxo_set with default arguments
would ignore the mempool and thus return utxos which were spent in
unconfirmed transactions. Thus, takers would continue negotiation of
coinjoins with makers who sent them already-spent utxos, leading to
failure at broadcast time. This was not intended behaviour; we want
takers to reject utxos that are double spent in the mempool.
This commit changes that default argument to True so that utxo set
changes in the mempool are accounted for. It also switches the name of
the includeunconf argument, which was misleading, to include_mempool,
with appropriately updated docstring.
Finally, in this commit we also ensure that callers of this function
check, where necessary, the returned confirmations field to disallow
unconfirmed utxos where that is necessary.
Closes#1247.
This consists of an inclusion of the bond value exponent into the config
that the user can alter, and a change of that from 2 to default 1.3.
Also updates the fidelity bond documentation to account for those
changes, including the units used in ob-watcher, but not the calculation
of fidelity bond attack resistance (which remains a TODO).
Adds code to validate bech32m encoded taproot addresses
and scriptPubKeys. The underlying functions are in
python-bitcointx upcoming version 1.1.3.
Adds tests at both the basic address validation layer,
and also at the layer of sending transactions to
taproot outputs.
Fixes#1043.
Prior to this commit, only keys/scripts/addresses
inside the scope of the current wallet script_map
(the keys cached by sync, according to persisted
index in wallet file, including gap limit) would
allow a successful signing operation, otherwise
an assertion was raised.
After this commit, signing can be done with any
arbitrary height index in the wallet (assuming a
valid path for this wallet).
bugfix: use enumerate instead of len
reviewed estimation of transaction sizes
estimates are still a bit conservative with room for improvement;
signatures could still save up to one byte each if using low-r
values. python-bitcointx==1.1.2-dev already supports grinding for
low-r values so when it's stable and referenced version is updated,
this should be reviewed again so as to utilize that benefit.
added utility method `estimate_extra_bytes`
the purpose of this method is for the computation
of extra bytes when the coinjoin or direct send output
type is different from that of the wallet
updated tests to reflect new transaction size
computation
p2pkh transactions are now 1 byte larger for
the inputs hence the change amount should be
less 4 * 30 sats.
add private keys for utxos that we may not be
tracking
some transactions (e.g. opt-in rbf) may require signing
with private keys for utxos that we may have stopped
tracking. this commit will search through all inputs
and for those we own and retrieve their private keys
so we can sign with them.
added support for p2wsh output scripts in
refactored the estimation of the transaction size
when outputs of a different type is the target
Previously, a different method was employed which
was kind of kludgy considering the fact that the
`extra_bytes` parameter is really for `OP_RETURN`
outputs. This method modifies the `estimate_tx_size`
method to accept an optional extra parameter called
`outtype` which is used to estimate the correct
transaction size if the target output is different
from that of the wallet.
added missing import
added a note about preserving the order of wallet type constants
Fix bug with timelocked addrs in receive payjoin
Previously there would be a crash if the wallet receiving a payjoin
had a timelocked UTXO.
This commit changes how timelocked addresses are created from the seed and bip32 tree.
A good thing to do would be to have each locktime (e.g. 1st jan 2020,
1st feb 2020, 1st march 2020, etc) actually use a different pubkey from
the HD tree (i.e. ("m/84'/1'/0'/2/0" + 1st jan 2020) ("m/84'/1'/0'/2/1" + 1st feb 2020), etc).
This now means that the sync code doesnt need to know what keys have been associated with
a fidelity bond to scan for the next one. Previously when a user funded a single timelocked
address, the wallet will generate _another_ pubkey and import _another_ ~960 addresses, so
funding one address would actually mean watching and generating ~1920 addresses not ~960.
This should help with the problem found by some people that fidelity bond wallets are slower
to sync. Other optimizations are possible but the structure of fidelity bond wallets will
probably be fixed for decades, so this change is worth doing now.
Parse incoming and announce outgoing fidelity bond messages
Fidelity bond proof messages will be checked and added to the internal
database just like offers. Such messages are not announced in public
but only directly to takers who ask for them, this is because the
signature proofs must commmit to the maker's and taker's IRC nicknames
in order to avoid replay attacks.
This removes any reference to the `key-type` parameter on the
`importprivkey` wallet command from the code and from the help output.
This parameter was already effectively being ignored, the key type would
always default to the key type of the wallet. This brings the help info
and the code comments in line with the existing functionality.
Prior to this commit, several test functions were using
"True" to flag internal and "False" to flag external for
the HD branch for the wallet, but we now use BaseWallet vars
ADDRESS_TYPE_[IN/EX]TERNAL (1/0), so this
is changed to explicitly reference those. There is no change
to the live code (which calls get_[internal/external]_addr).
In addition _index_cache updates in wallet are protected
with a wrapper function to ensure that the branch requested
is valid.
Tests pass both before and after this change.
* yg scripts set reloffer/absoffer only:
Prior to this commit, the yield generator user
level scripts required the user to specify offer
types depending on the wallet, but only 'rel/abs'
distinction is user choice; the other element
(native segwit, p2sh or p2pkh) must be defined
by the wallet, so we now call `wallet.get_txtype()`
to translate from reloffer/absoffer to sw0.. etc.
* Taker chooses nversion, nlocktime per wallet type:
Takers who are still using p2sh-p2wpkh wallets will
not want to flag their transactions with different
tx metadata than previous versions that are still
running, so we check the `get_txtype()` output to
decide which nVersion and nLockTime to use.
Also, the SNICKER locktime is reverted to zero as
according to draft spec.
* change offer type in test_coinjoin
* update docs for bech32 wallets
Also, ensure witness_utxo field is populated,
plus minor bugfixes related to presence of
NONWITNESS_UTXO field in provided payment PSBT.
Tested as being functional either with or without
NONWITNESS_UTXO field, for all-segwit inputs.
This commit implements a command line script and a GUI
dialog to receive a payment using the BIP78 protocol,
by setting up an ephemeral hidden service.
It also deprecates the pre-existing inter-Joinmarket
protocol for payjoin payments, since we now have
both sending and receiving support for BIP78. Thus,
much code in Maker, Taker and client-daemon protocol
is removed, as is some documentation in docs/PAYJOIN.md.
Also the script `sendpayment.py` is altered to support
only the BIP78 variant.
The test in jmclient/test/test_payjoin now implements
BIP78 over a TCP connection, while the custom tests in
test/payjoinserver.py can support hidden service based
tests, but the latter is not included in the test suite
and may not always work (it is only for manual
investigations).
The following features of BIP78 are supported:
minfeerate
additionalfeeoutputindex - but *only* for single
change output transactions
maxadditionalfeecontribution
The receiver does not have nor request payment
output substitution.
Utxo selection is no longer sophisticated, instead
we only choose a single utxo to keep the size
increase of the transaction minimal. Thus UIH is
not addressed at the moment.
Errors returned are in line with BIP78.
Sequence numbers are checked by receiver, and
kept identical if uniform, otherwise respected.
Receiver uses transaction monitor to shut down
when the payment is seen.
The workflow is almost entirely implemented in
jmclient/payjoin.py and the command line script
is in scripts/receive-payjoin.py. The setup, including
configuration changes for Tor, are documented in
docs/PAYJOIN.md, including a user guide video linked.
Upgrade python-bitcointx to 1.1.0:
Address requirements of python-bitcointx 1.1.0:
Specifically, the witness `utxo` field can no longer be
assumed to be of type CTxOut, so we should access the
CTxOut with the field witness_utxo and also when updating
the `utxo` field we now use `set_utxo()`.
Use PartiallySignedTransaction.get_fee() method.
Use PartiallySignedTransaction.set_utxo.
Additionally some minor typos/comment corrections and removal
of the now defunct `apply_freeze_signature`.
Add custom load location for libsecp where needed;
falls back to system installation if Joinmarket custom
installation is not found.
Decode error msg from server in payjoin
Cleanup test file test_proposals.txt (delete after test)
Human readable function names (names for human readable
conversions are now themselves human readable).
Remove unused get_*_vbyte functions and cleanup
Removes old unused files (electrum*.py).
Fixes core nohistory sync test to use both standard
wallet types, and fixes address import counter.
Fixes that test to use the right chain params so that
native segwit wallets can work in regtest with
nohistory mode.
Removes some now unneeded imports.
Fixes commontest.create_wallet_for_sync to hash all
parameters, including optional ones.
Replaces usage of binascii.hexlify with bintohex.
Kristaps Kaupe
Adam Gibson
Matt Whitlock
roshii
Tim Akinbo
undeath
openoms
chris-belcher
Daniel McNally
Jules Comte
jules23