Prior to this commit, p2wsh inputs from fidelity bonds resulted in
miscalculation of transaction fees, even in cases where the exact set of
inputs were known (such as a direct send).
In this commit we change the estimation to a model in which the caller
of jmbitcoin.secp256k1_transaction.estimate_tx_size must specify a list
of types, one for each input to the transaction, and the same for
outputs. In some cases, the caller of the function uses the default
script type of the wallet, but in other cases where the caller can know
the exact types of each utxo used as input, and each destination used as
output, they are specified explicitly. In particular, the use of
fidelity bond outputs as input to transactions can be accounted for.
Currently this is only done in direct send payments; coinjoins still
fall back to assuming all inputs the same type (note that it is not
possible to use fidelity bond utxos as inputs to coinjoins).
Note also that the burn destination calculation in direct send is
removed, since it is not used, so the maintenance burden is best
avoided.
Adds code to validate bech32m encoded taproot addresses
and scriptPubKeys. The underlying functions are in
python-bitcointx upcoming version 1.1.3.
Adds tests at both the basic address validation layer,
and also at the layer of sending transactions to
taproot outputs.
1. Moves the JMWalletDaemon service class into
the jmclient package (see the wallet_rpc.py module).
2. Adds dependencies "klein" and "autobahn" to the
jmclient package, as well as "pyjwt".
3. Adds another module websocketserver.py, using
autobahn, to allow the JMWalletDaemon service to
serve subscriptions over a websocket, for e.g.
transaction notifications.
4. Adds tests both for the websocket connection
and for the JSON-RPC HTTP connection.
JmwalletdWebSocketServerFactory.sendTxNotification
sends the json-ified transaction details using
jmbitcoin.human_readable_transaction (as is currently
used in our CLI), along with the txid.
Also adds a coinjoin state update event sent via
the websocket (switch from taker/maker/none).
Require authentication to connect to websocket.
5. Add OpenApi definition of API in yaml;
also auto-create human-readable API docs in markdown.
6. Add fidelity bond function to API
7. Add config read/write route to API
8. Remove snicker rpc calls temporarily
9. Updates to docoinjoin: corrects taker_finished
for this custom case, does not shut down at end.
10. Address detailed review comments of @PulpCattel.
The /utxos route is the equivalent of the showutxos
wallet tool method.
The heartbeat route /session allows a client to
make sure the backend is still running and in the
expected state (but see later commits for the
coinjoin state update via the websocket).
Fixes to sendpayment, the maker service,
the create wallet function.
sendpayment fix
Also substantially improved and made
functional the coinjoin route, a schedule
is now created and a complete taker-side
coinjoin is now possible.
bugfix: use enumerate instead of len
reviewed estimation of transaction sizes
estimates are still a bit conservative with room for improvement;
signatures could still save up to one byte each if using low-r
values. python-bitcointx==1.1.2-dev already supports grinding for
low-r values so when it's stable and referenced version is updated,
this should be reviewed again so as to utilize that benefit.
added utility method `estimate_extra_bytes`
the purpose of this method is for the computation
of extra bytes when the coinjoin or direct send output
type is different from that of the wallet
updated tests to reflect new transaction size
computation
p2pkh transactions are now 1 byte larger for
the inputs hence the change amount should be
less 4 * 30 sats.
add private keys for utxos that we may not be
tracking
some transactions (e.g. opt-in rbf) may require signing
with private keys for utxos that we may have stopped
tracking. this commit will search through all inputs
and for those we own and retrieve their private keys
so we can sign with them.
added support for p2wsh output scripts in
refactored the estimation of the transaction size
when outputs of a different type is the target
Previously, a different method was employed which
was kind of kludgy considering the fact that the
`extra_bytes` parameter is really for `OP_RETURN`
outputs. This method modifies the `estimate_tx_size`
method to accept an optional extra parameter called
`outtype` which is used to estimate the correct
transaction size if the target output is different
from that of the wallet.
added missing import
added a note about preserving the order of wallet type constants
Fix bug with timelocked addrs in receive payjoin
Previously there would be a crash if the wallet receiving a payjoin
had a timelocked UTXO.
Before this commit, while sending from mixdepth
zero in a sweep, where 1 or more p2wsh timelock
type outputs (now past their timelock), were present,
was working correctly, non-sweep sends in the same
circumstances were failing due to a key error.
This is now fixed.
Fixes#797.
Adds `custom_change_addr` argument to `direct_send()`
joinmarket-qt: Adds input field for optional external change address
joinmarket-qt: Better handle PayJoin/CoinJoin state changes for changeInput widget
Adds `custom_change_address` argument to Taker constructor and use it in joinmarket-qt
Custom change also allowed in sendpayment CLI with `-u` flag (not
supported in tumbler).
Explicitly disallows using custom change with BIP78 Payjoin, though that
could change later.
Both sendpayment and CLI provide detailed warnings to avoid misuse. In
particular, they have an extra warning for using a nonstandard or
non-wallet scriptpubkey type.
Setting custom change to the recipient address is explicitly forbidden.
Tests: Adds custom_change usage test in test_taker.
Upgrade python-bitcointx to 1.1.0:
Address requirements of python-bitcointx 1.1.0:
Specifically, the witness `utxo` field can no longer be
assumed to be of type CTxOut, so we should access the
CTxOut with the field witness_utxo and also when updating
the `utxo` field we now use `set_utxo()`.
Use PartiallySignedTransaction.get_fee() method.
Use PartiallySignedTransaction.set_utxo.
Additionally some minor typos/comment corrections and removal
of the now defunct `apply_freeze_signature`.
Add custom load location for libsecp where needed;
falls back to system installation if Joinmarket custom
installation is not found.
Decode error msg from server in payjoin
Cleanup test file test_proposals.txt (delete after test)
Human readable function names (names for human readable
conversions are now themselves human readable).
Remove unused get_*_vbyte functions and cleanup
Removes old unused files (electrum*.py).
Fixes core nohistory sync test to use both standard
wallet types, and fixes address import counter.
Fixes that test to use the right chain params so that
native segwit wallets can work in regtest with
nohistory mode.
Removes some now unneeded imports.
Fixes commontest.create_wallet_for_sync to hash all
parameters, including optional ones.
Replaces usage of binascii.hexlify with bintohex.
Note in particular that:
bitcoin.mktx in this PR now does support script
entries in outputs to account for nonstandard
destinations (as is needed for burn).
bitcoin.sign now supports p2wsh (as is needed
for timelocks).
Fix bug in Coins tab, get_utxos_by_mixdepth calls
Fix bug in jmbitcoin.mktx, now respects nVersion choice
Fix bug in tumbler restart wait
Fix bug in makercount for payjoin for fee check, set to 0:
The value of `options.makercount` is set to zero
so that the fee sanity check in the sendpayment script
operates approximately correctly (the receiver will bump
the fee to keep the fee rate the same if necessary).
Also the `bip79` variable is better named `payjoinurl`.
Fix bug in "freeze" context menu function in Qt
See:
https://github.com/bitcoin/bips/blob/master/bip-0078.mediawiki
Adds a new module jmclient.payjoin which implements
the full sender workflow from a BIP21 uri to a
payjoin broadcast, state is managed in JMPayjoinManager,
includes all checks as per documentation of
btcpayserver (and later, BIP78).
Added simple client and server implementations in
test/payjoinclient.py and test/payjoinserver.py
which allow a full end to end test on regtest.
Add TLS support to payjoin tests:
Note: the jmclient.payjoin module already
supports TLS by default (Agent object), but
here we add the ability to test without
certificate verification. Both test/payjoinclient.py
and test/payjoinserver.py now support TLS, but
the server needs a key and certificate in its
directory to run.
Adds BIP78 payjoin option to sendpayment.py
Users can use a bip21 uri with the "pj" field to
send a payment to a remote server.
Removes require_path_templates setting from KeyStore call
in PSBTWalletMixin.sign_psbt
Human readable representation for CTransaction
objects in jmbitcoin.secp256k1_transaction.py and for
PartiallySignedTransaction objects in jmclient.wallet.
PSBTWalletMixin, use of these in maker, taker, direct
send and in tests. Users should note that PSBT human
readable representations can in some cases be really
huge.
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
Subclassed btcointx.CKeyBase to create a privkey type
that uses our signing code.
These are then used to allow signing of inputs in
transactions owned by our wallet.
Created a PSBTWalletMixin to perform the above function,
and added it to our three wallet types.
Wrote a detailed creation, sign and broadcast test
for a psbt using utxos from within and outside the wallet.
Detailed tests across all 3 wallet types for psbt
Tests cover also mixed inputs of different
types and owned/unowned. Direct send now exported to be used
in tests rather than only script usage, also supports returning
a tx object rather than only a txid.
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
Watchonly wallets use pubkeys instead of privkeys, but in a bit of
hack the functions previously called "_get_priv_from_path" would
actually return public keys for watchonly wallets. This could have
pretty terrible consequences one day, so functions like that have
been renamed to use the word "key" instead, which could be either
private or public.
The cryptoengine class BTC_Timelocked_P2WSH now implements
sign_transaction() which can be used to spend timelocked UTXOs.
FidelityBondMixin.is_timelocked_path() is now used outside the class
so its leading underscore has been removed.
Prior to this commit, if a tumbler coinjoin negotiation
failed in Phase 2, then the retry as per the logic in
taker_utils.tumbler_taker_finished_update would always
attempt to retry the transaction with those counterparties
that returned valid !sig responses. However this ignored
the case that all the counterparties responded validly,
but there was a mempool conflict in the created transaction.
After this commit, if it is detected that all counterparties
responded, it is assumed that a mempool conflict or similar
occurred with the transaction, and therefore it is better
to fallback to a schedule tweak and choose randomly again,
not to fix the counterparty set (which is likely to result
in failing again).
Prior to this commit, the user was prompted to check for
the acceptability of fees in Qt via a dialog, but settings
used in CLI (max_cj_fee_***) were not also being used the
same way in Qt.
After this commit, if the user has not added those settings,
a dialog is presented with new randomised defaults (as for
CLI), and otherwise any settings in the config file are read
and used.
The schedule format gets an extra field added denoting the number of
significant figures to round the coinjoin amounts to, with 16 meaning
no rounding.
This is part of the 2/2019 Plan to improve the privacy of JoinMarket's
tumbler script:
https://gist.github.com/chris-belcher/7e92810f07328fdfdef2ce444aad0968
Introduces WalletService object which is in control of
blockchain and wallet access.
The service manages a single transaction monitoring loop,
instead of multiple, and allows updates to the wallet from
external sources to be handled in real time, so that both Qt
and other apps (yg) can respond to deposits or withdrawals
automatically.
The refactoring also controls access to both wallet and
blockchain so that client apps (Taker, Maker) will not need
to be changed for future new versions e.g. client-side filtering.
Also updates and improves Wallet Tab behaviour in Qt (memory
of expansion state).
Additionally, blockchain sync is now --fast by default, with
the former default of detailed sync being renamed --recoversync.
Prior to this commit, there was duplicated code in maker
and taker modules to import addresses, now all calls to
the wallet for fresh addresses can optionally pass a
blockchaininterface instance and if this is done, the
new address will be imported to the BCI at the same time.
Fixes#362. Prior to this commit, if tumbler was invoked with
the --restart flag, it waited for the presence of the 0 index
output of the last seen tumbler transaction in the utxo set,
which was an unreliable indicator as that output could have,
in the meantime, been spent.
After this commit, we utilise the fact that the transaction
must be an in-wallet transaction, and use the rpc gettransaction
to check whether it has 1 or more confirmations, instead, which
should always work whether any outputs are spent or not.
- update mktx() to allow optional locktime setting (and sequence)
- add a mk_shuffled_tx method to the wallet module
- add a P2EPTaker and P2EPMaker class (inherit from Taker, Maker)
- add a -T option to sendpayment script for doing payjoins
- add a receive_payjoin script for receivers.
- add payjoin tests in jmclient/test/test_payjoin.py
- add a custom utxo selection method select_one_utxo to support.py
- support bech32 wallets (SegwitWallet, p2wpkh) with native=true
in config POLICY for PayJoin and direct send (not Joinmarket CJ)
- add a PayJoin.md usage guide in docs/
- include version bytes in pubkey message for forward compat
- taker pays fees but controls size (utxo number and fee/kB)
- add P2WPKH fee estimator
- Enforce INFO level logging in payjoin
- refactors regtest config settings into one place
- bugfix: script_to_address vbyte argument is bytes not integer
* parsing of scripts and addresses of all segwit types.
* ability to verify arbitrary tx inputs for all segwit types.
* simplify mktx syntax (only allow [],[] args)
* add p2wpkh and p2wsh spending test and fixes to sign calls in wallet
* simplify wallet signing calls in cryptoengine
* add p2wpkh engine, add bip84 wallet
Previous to this commit, the jmbitcoin package was accessed via
a file in jmclient `btc.py` which was originally added as an
interface to allow the client to use a non-jmbitcoin package to
provide the implementation; while this idea is useful, the way
it was implemented was not, moreover it is not currently used
and contained duplicated code that was unmanaged. Also, the
original usage of this was only by the electrum plugin, which
has currently been abandoned. This simplifies the code and
avoids spurious error messages. Note that most of the changes
are a result of pulling the logging function directly from the
jmbase package instead of indirectly via this interface (which
was unnecessary and not connected with jmbitcoin).
Adam Gibson
abhishek0405
Tim Akinbo
csH7KmCC9
Kristaps Kaupe
chris-belcher
James Hilliard
cindiv
undeath