Add a validate_cache parameter to the five principal caching methods:
- _get_key_from_path
- _get_keypair_from_path
- _get_pubkey_from_path
- get_script_from_path
- get_address_from_path
and to the five convenience methods that wrap the above:
- get_script
- get_addr
- script_to_addr
- get_new_script
- get_new_addr
The value of this new parameter defaults to False in all but the last
two methods, where we are willing to sacrifice speed for the sake of
extra confidence in the correctness of *new* scripts and addresses to
be used for new deposits and new transactions.
Deriving private keys from BIP32 paths, public keys from private keys,
scripts from public keys, and addresses from scripts are some of the
most CPU-intensive tasks the wallet performs. Once the wallet inevitably
accumulates thousands of used paths, startup times become painful due to
needing to re-derive these data items for every used path in the wallet
upon every startup. Introduce a persistent cache to avoid the need to
re-derive these items every time the wallet is opened.
Introduce _get_keypair_from_path and _get_pubkey_from_path methods to
allow cached public keys to be used rather than always deriving them on
the fly.
Change many code paths that were calling CPU-intensive methods of
BTCEngine so that instead they call _get_key_from_path,
_get_keypair_from_path, _get_pubkey_from_path, get_script_from_path,
and/or get_address_from_path, all of which can take advantage of the new
cache.
This allows running tests that do not require, e.g., bitcoind, without having to run these setup fixtures at all.
Signed-off-by: PulpCattel <PulpCattel@users.noreply.github.com>
Prior to this commit, p2wsh inputs from fidelity bonds resulted in
miscalculation of transaction fees, even in cases where the exact set of
inputs were known (such as a direct send).
In this commit we change the estimation to a model in which the caller
of jmbitcoin.secp256k1_transaction.estimate_tx_size must specify a list
of types, one for each input to the transaction, and the same for
outputs. In some cases, the caller of the function uses the default
script type of the wallet, but in other cases where the caller can know
the exact types of each utxo used as input, and each destination used as
output, they are specified explicitly. In particular, the use of
fidelity bond outputs as input to transactions can be accounted for.
Currently this is only done in direct send payments; coinjoins still
fall back to assuming all inputs the same type (note that it is not
possible to use fidelity bond utxos as inputs to coinjoins).
Note also that the burn destination calculation in direct send is
removed, since it is not used, so the maintenance burden is best
avoided.
bugfix: use enumerate instead of len
reviewed estimation of transaction sizes
estimates are still a bit conservative with room for improvement;
signatures could still save up to one byte each if using low-r
values. python-bitcointx==1.1.2-dev already supports grinding for
low-r values so when it's stable and referenced version is updated,
this should be reviewed again so as to utilize that benefit.
added utility method `estimate_extra_bytes`
the purpose of this method is for the computation
of extra bytes when the coinjoin or direct send output
type is different from that of the wallet
updated tests to reflect new transaction size
computation
p2pkh transactions are now 1 byte larger for
the inputs hence the change amount should be
less 4 * 30 sats.
add private keys for utxos that we may not be
tracking
some transactions (e.g. opt-in rbf) may require signing
with private keys for utxos that we may have stopped
tracking. this commit will search through all inputs
and for those we own and retrieve their private keys
so we can sign with them.
added support for p2wsh output scripts in
refactored the estimation of the transaction size
when outputs of a different type is the target
Previously, a different method was employed which
was kind of kludgy considering the fact that the
`extra_bytes` parameter is really for `OP_RETURN`
outputs. This method modifies the `estimate_tx_size`
method to accept an optional extra parameter called
`outtype` which is used to estimate the correct
transaction size if the target output is different
from that of the wallet.
added missing import
added a note about preserving the order of wallet type constants
Fix bug with timelocked addrs in receive payjoin
Previously there would be a crash if the wallet receiving a payjoin
had a timelocked UTXO.
Fixes#976. Joinmarket will allow, but warn, when
non-zero change values (which are not included in
the transaction) result from sweeps (due to the
impossibility of knowing the exact value in advance).
However, prior to this commit, if that estimation
inaccuracy resulted in a negative change value that
was to be ignored, there was a crash due to the fact
that `jmbitcoin.amount_to_str` does not allow negative
values. Hence we instead print the number in the warning
without going through this formatting function.
Adds tests case for negative sweep change.
Before this commit, attempting to spend in a taker-
side Joinmarket coinjoin, a utxo which is of the
tiemelock type (after its timelock expired), had
a potential to crash, because it would be selected
for the generation of a PoDLE but this cannot be
supported (even with taker-side code changes, the
maker could not reconstruct it).
After this commit, any non-standard wallet-type
scriptPubKey is filtered out of being a candidate
for PoDLE generation.
Also a trivial variable name typo is fixed (too-old).
Also the test framework `DummyWallet` now inherits
from `LegacyWallet` because its utxos are of that
type; a future commit should update to `SegwitWallet`.
Also add tests cases for no custom-script PoDLEs.
Fixes#797.
Adds `custom_change_addr` argument to `direct_send()`
joinmarket-qt: Adds input field for optional external change address
joinmarket-qt: Better handle PayJoin/CoinJoin state changes for changeInput widget
Adds `custom_change_address` argument to Taker constructor and use it in joinmarket-qt
Custom change also allowed in sendpayment CLI with `-u` flag (not
supported in tumbler).
Explicitly disallows using custom change with BIP78 Payjoin, though that
could change later.
Both sendpayment and CLI provide detailed warnings to avoid misuse. In
particular, they have an extra warning for using a nonstandard or
non-wallet scriptpubkey type.
Setting custom change to the recipient address is explicitly forbidden.
Tests: Adds custom_change usage test in test_taker.
Before this commit, the taker would choose offers from
the pit based on the setting of `native` in the `[POLICY]`
section of the config object; however this could lead to
users unwittingly choosing the wrong offer type, i.e. one
that is incompatible with their own wallets, which could
result in coinjoins with mixed address types.
This commit fixes that error by only selecting offers that
are compatible with the return value of `BaseWallet.get_txtype()`.
Also fixes up tests for this wallet type enforcement.
Note in particular that:
bitcoin.mktx in this PR now does support script
entries in outputs to account for nonstandard
destinations (as is needed for burn).
bitcoin.sign now supports p2wsh (as is needed
for timelocks).
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
To facilitate easier management by users and to
follow generally accepted standards, this PR moves
the following all to user home directory, subdir
.joinmarket :
joinmarket.cfg file
wallets/ directory
logs/ directory
cmtdata/ directory
commitmentlist file
User can override location with --datadir option.
An info message is added on startup showing location.
Prior to this commit, the user was prompted to check for
the acceptability of fees in Qt via a dialog, but settings
used in CLI (max_cj_fee_***) were not also being used the
same way in Qt.
After this commit, if the user has not added those settings,
a dialog is presented with new randomised defaults (as for
CLI), and otherwise any settings in the config file are read
and used.
The schedule format gets an extra field added denoting the number of
significant figures to round the coinjoin amounts to, with 16 meaning
no rounding.
This is part of the 2/2019 Plan to improve the privacy of JoinMarket's
tumbler script:
https://gist.github.com/chris-belcher/7e92810f07328fdfdef2ce444aad0968
Introduces WalletService object which is in control of
blockchain and wallet access.
The service manages a single transaction monitoring loop,
instead of multiple, and allows updates to the wallet from
external sources to be handled in real time, so that both Qt
and other apps (yg) can respond to deposits or withdrawals
automatically.
The refactoring also controls access to both wallet and
blockchain so that client apps (Taker, Maker) will not need
to be changed for future new versions e.g. client-side filtering.
Also updates and improves Wallet Tab behaviour in Qt (memory
of expansion state).
Additionally, blockchain sync is now --fast by default, with
the former default of detailed sync being renamed --recoversync.
Prior to this commit, there was duplicated code in maker
and taker modules to import addresses, now all calls to
the wallet for fresh addresses can optionally pass a
blockchaininterface instance and if this is done, the
new address will be imported to the BCI at the same time.
As for maker, to use this feature set segwit = false
in the POLICY section of the config file.
This commit does *not* include support in the GUI.
Additionally, minor fixes to utility scripts for non-sw wallets,
and updated add-utxo tool to support non-sw wallets.
Bugfixes to wallet_utils to support non-sw wallets
and signmessage.
Signatures sent a second time now are correctly ignored, rather than prompting
a removal for the utxo entry which fails since it's already been removed.
Also corrects the case where a junk signature can trigger a crash in verify_tx_input,
if its script deserialization results in non-string entries (integer or None).
Test cases in test_taker updated to include these error cases, now passing.
Some modifications to Wallet to remove user interaction from module.
Removal of ElectrumWrapWallet, moved to the plugin files.
Add tests for aes, mnemonics.
Add harness for tests in jmclient, conftest.py and commontest
Add .coveragerc for these tests locally.
Also, some minor changes: filter_orders_callback taken outside
Taker since it involves user input necessarily. Related, answeryes
removed from module, can be done outside.
Donation code explicitly removed from all code branches for now.
A couple of very minor changes thrown up from testing.
Coverage of Taker now 100%, will work on the rest of jmclient.
Kristaps Kaupe
Matt Whitlock
roshii
PulpCattel
Adam Gibson
Tim Akinbo
undeath
chris-belcher
csH7KmCC9
PulpCattel
jules23
James Hilliard