This commit updates sendtomany.py and add-utxo.py to reflect
the new default wallet type (p2wpkh) and the new bitcoin
backend code; in doing this it slightly modifies the functions
in commitment_utils.py.
* yg scripts set reloffer/absoffer only:
Prior to this commit, the yield generator user
level scripts required the user to specify offer
types depending on the wallet, but only 'rel/abs'
distinction is user choice; the other element
(native segwit, p2sh or p2pkh) must be defined
by the wallet, so we now call `wallet.get_txtype()`
to translate from reloffer/absoffer to sw0.. etc.
* Taker chooses nversion, nlocktime per wallet type:
Takers who are still using p2sh-p2wpkh wallets will
not want to flag their transactions with different
tx metadata than previous versions that are still
running, so we check the `get_txtype()` output to
decide which nVersion and nLockTime to use.
Also, the SNICKER locktime is reverted to zero as
according to draft spec.
* change offer type in test_coinjoin
* update docs for bech32 wallets
Also, ensure witness_utxo field is populated,
plus minor bugfixes related to presence of
NONWITNESS_UTXO field in provided payment PSBT.
Tested as being functional either with or without
NONWITNESS_UTXO field, for all-segwit inputs.
Also manually fire order creation in coinjoin tests.
This clarification and test change is required due
to the fact that LoopingCalls are designed to fire
immediately by default, before the reactor is
initialized (and therefore in a `running` state),
making it not possible to shutdown the reactor as
a result of events happening in that first call;
so we delay the first call of the maker's orderbook
populating code, so that if a no-coins error
occurs, it will actually shut down the reactor and
hence the whole yield generator program, as intended.
This commit implements a command line script and a GUI
dialog to receive a payment using the BIP78 protocol,
by setting up an ephemeral hidden service.
It also deprecates the pre-existing inter-Joinmarket
protocol for payjoin payments, since we now have
both sending and receiving support for BIP78. Thus,
much code in Maker, Taker and client-daemon protocol
is removed, as is some documentation in docs/PAYJOIN.md.
Also the script `sendpayment.py` is altered to support
only the BIP78 variant.
The test in jmclient/test/test_payjoin now implements
BIP78 over a TCP connection, while the custom tests in
test/payjoinserver.py can support hidden service based
tests, but the latter is not included in the test suite
and may not always work (it is only for manual
investigations).
The following features of BIP78 are supported:
minfeerate
additionalfeeoutputindex - but *only* for single
change output transactions
maxadditionalfeecontribution
The receiver does not have nor request payment
output substitution.
Utxo selection is no longer sophisticated, instead
we only choose a single utxo to keep the size
increase of the transaction minimal. Thus UIH is
not addressed at the moment.
Errors returned are in line with BIP78.
Sequence numbers are checked by receiver, and
kept identical if uniform, otherwise respected.
Receiver uses transaction monitor to shut down
when the payment is seen.
The workflow is almost entirely implemented in
jmclient/payjoin.py and the command line script
is in scripts/receive-payjoin.py. The setup, including
configuration changes for Tor, are documented in
docs/PAYJOIN.md, including a user guide video linked.
Upgrade python-bitcointx to 1.1.0:
Address requirements of python-bitcointx 1.1.0:
Specifically, the witness `utxo` field can no longer be
assumed to be of type CTxOut, so we should access the
CTxOut with the field witness_utxo and also when updating
the `utxo` field we now use `set_utxo()`.
Use PartiallySignedTransaction.get_fee() method.
Use PartiallySignedTransaction.set_utxo.
Additionally some minor typos/comment corrections and removal
of the now defunct `apply_freeze_signature`.
Add custom load location for libsecp where needed;
falls back to system installation if Joinmarket custom
installation is not found.
Decode error msg from server in payjoin
Cleanup test file test_proposals.txt (delete after test)
Human readable function names (names for human readable
conversions are now themselves human readable).
Remove unused get_*_vbyte functions and cleanup
Removes old unused files (electrum*.py).
Fixes core nohistory sync test to use both standard
wallet types, and fixes address import counter.
Fixes that test to use the right chain params so that
native segwit wallets can work in regtest with
nohistory mode.
Removes some now unneeded imports.
Fixes commontest.create_wallet_for_sync to hash all
parameters, including optional ones.
Replaces usage of binascii.hexlify with bintohex.
Note in particular that:
bitcoin.mktx in this PR now does support script
entries in outputs to account for nonstandard
destinations (as is needed for burn).
bitcoin.sign now supports p2wsh (as is needed
for timelocks).
Human readable representation for CTransaction
objects in jmbitcoin.secp256k1_transaction.py and for
PartiallySignedTransaction objects in jmclient.wallet.
PSBTWalletMixin, use of these in maker, taker, direct
send and in tests. Users should note that PSBT human
readable representations can in some cases be really
huge.
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
This commit uses the now created PSBTWalletMixin and additionally
creates a SNICKERWalletMixin, and adds a SNICKERReceiver object
to jmclient. A test of the end to end workflow of create and then
co-sign a SNICKER coinjoin as per the draft BIP is in test_snicker.
Additional changes:
updated python-bitcointx dependency to >=1.0.5
Minor refactoring of callbacks in tests and additional redeem
script checks to PSBTWalletMixin.sign_psbt.
Note that this work replaces #403 .
Subclassed btcointx.CKeyBase to create a privkey type
that uses our signing code.
These are then used to allow signing of inputs in
transactions owned by our wallet.
Created a PSBTWalletMixin to perform the above function,
and added it to our three wallet types.
Wrote a detailed creation, sign and broadcast test
for a psbt using utxos from within and outside the wallet.
Detailed tests across all 3 wallet types for psbt
Tests cover also mixed inputs of different
types and owned/unowned. Direct send now exported to be used
in tests rather than only script usage, also supports returning
a tx object rather than only a txid.
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
Fixes#442.
First, the CONNREFUSED socket error is handled in jsonrpc.
Second, we respond to this (but *not* to resets) with a reactor
shutdown in BitcoinCoreInterface.rpc(). This also necessitates
early-quitting in the calling function
(WalletService.transaction_monitor) since the reactor stop
will only stop future deferred calls, not the currently running
one. The obvious sys.exit approach is only used in startup,
because the reactor is not currently running at that point.
Also minor change to DummyBlockchainInterface for test.
Watchonly wallets use pubkeys instead of privkeys, but in a bit of
hack the functions previously called "_get_priv_from_path" would
actually return public keys for watchonly wallets. This could have
pretty terrible consequences one day, so functions like that have
been renamed to use the word "key" instead, which could be either
private or public.
Previously an example of a BIP32 path
would be: m/wallet-type'/mixdepth'/internal/index
The 'internal' name referred to internal and external addresses (also
called change and receive). The renaming to 'address_type' is in
preparation to add more branches for timelocked addresses and burner
outputs.
The variable formally known as 'internal' is now no longer a boolean
but always an integer. This almost-always seemlessly fits because the
values False and Ture correspond to 0 and 1. The function
_get_internal_type therefore has no purpose anymore. Delete it.
Fixes#477. Removes nonstandard script support
from 0.19+ bitcoind configuration for regtest and
changes scripts in test_wallet and test_tx_creation
to make them standard.
Those names as confusing. They could imply that the function obtains
a path or address given a script. To help the code be more
self-documenting I add the verb from.
To facilitate easier management by users and to
follow generally accepted standards, this PR moves
the following all to user home directory, subdir
.joinmarket :
joinmarket.cfg file
wallets/ directory
logs/ directory
cmtdata/ directory
commitmentlist file
User can override location with --datadir option.
An info message is added on startup showing location.
updated xenial and stretch dockerfiles since they bundle old versions of libsodium
refacatoring to make BlockchainInterface and BitcoinCoreNoHistoryInterface python2 compatible
add the -acceptnonstdtxn flag when invoking bitcoind for versions >= 0.19.0
refactor the use of nonlocal to make it compatible with python2
document the get_bitcoind_version method
The above commit introduces auto freezing for utxos below
a threshold, but erroneously auto freezes new utxos in
almost all cases because transactions are processed
multiple times (add_utxos handles this in the wallet).
The problem here is solved as with the same issue of
duplication in the logging of new transactions; we keep
track of new txids that arrive in the wallet and make
sure not to process the same txid twice.
Additionally, the setting of WalletService.used_addresses
is fixed. Test for this function is also fixed.
Closes#274. Utxos are disabled if they are sent
to a reused address, and are below a threshold
set by the value `max_sats_freeze_reuse` in the
`POLICY` section of the config file. If the value
is -1, such utxos are always frozen irrespective of
the value.
Users are prompted with a warning level logging message
on CLI and a popup on Joinmarket-Qt. Such disabled utxos
can of course be re-enabled by the existing methods.
Also adds test case for address reuse freezing function.
Prior to this commit, the user was prompted to check for
the acceptability of fees in Qt via a dialog, but settings
used in CLI (max_cj_fee_***) were not also being used the
same way in Qt.
After this commit, if the user has not added those settings,
a dialog is presented with new randomised defaults (as for
CLI), and otherwise any settings in the config file are read
and used.
The schedule format gets an extra field added denoting the number of
significant figures to round the coinjoin amounts to, with 16 meaning
no rounding.
This is part of the 2/2019 Plan to improve the privacy of JoinMarket's
tumbler script:
https://gist.github.com/chris-belcher/7e92810f07328fdfdef2ce444aad0968
The tumbler schedule is split into two stages. Stage 2 is the same
as before while stage 1 attempts to fully spend each mixdepth in a
sweep coinjoin with no change address.
The wait time between these stage 1 coinjoins is longer than for
stage 2 coinjoins, the increase is determined by a new parameter
called `stage1_timelambda_increase`.
This is part of the 2/2019 Plan to improve the privacy of JoinMarket's
tumbler script:
https://gist.github.com/chris-belcher/7e92810f07328fdfdef2ce444aad0968
Introduces WalletService object which is in control of
blockchain and wallet access.
The service manages a single transaction monitoring loop,
instead of multiple, and allows updates to the wallet from
external sources to be handled in real time, so that both Qt
and other apps (yg) can respond to deposits or withdrawals
automatically.
The refactoring also controls access to both wallet and
blockchain so that client apps (Taker, Maker) will not need
to be changed for future new versions e.g. client-side filtering.
Also updates and improves Wallet Tab behaviour in Qt (memory
of expansion state).
Additionally, blockchain sync is now --fast by default, with
the former default of detailed sync being renamed --recoversync.
This adds simple unit tests for the core functionality of
YieldGeneratorBasic in the new file jmclient/test/test_yieldgenerator.py.
The tests verify that offer creation, reannouncement of offers when a
change is made to the maxsize and translation of offers to orders works
as it should.
remove unused import
Prior to this commit, there was duplicated code in maker
and taker modules to import addresses, now all calls to
the wallet for fresh addresses can optionally pass a
blockchaininterface instance and if this is done, the
new address will be imported to the BCI at the same time.
Wallet persists utxo metadata; currently only
contains a field 'disabled' indexed by utxo.
User can switch this on or off (enabled) via
wallet-tool 'freeze' method.
Disabled utxos will not be used in coin
selection in any transaction.
Wallet still displays all utxo balances in
display method (and in GUI).
Add tests of disabling to test_utxomanager
Add Coins tab to Qt, with freeze/unfreeze feature.
Coins tab shows updated utxo info txid:n, amt, address and
enabled/disabled, which can be toggled from right click menu.
- update mktx() to allow optional locktime setting (and sequence)
- add a mk_shuffled_tx method to the wallet module
- add a P2EPTaker and P2EPMaker class (inherit from Taker, Maker)
- add a -T option to sendpayment script for doing payjoins
- add a receive_payjoin script for receivers.
- add payjoin tests in jmclient/test/test_payjoin.py
- add a custom utxo selection method select_one_utxo to support.py
- support bech32 wallets (SegwitWallet, p2wpkh) with native=true
in config POLICY for PayJoin and direct send (not Joinmarket CJ)
- add a PayJoin.md usage guide in docs/
- include version bytes in pubkey message for forward compat
- taker pays fees but controls size (utxo number and fee/kB)
- add P2WPKH fee estimator
- Enforce INFO level logging in payjoin
- refactors regtest config settings into one place
- bugfix: script_to_address vbyte argument is bytes not integer
* parsing of scripts and addresses of all segwit types.
* ability to verify arbitrary tx inputs for all segwit types.
* simplify mktx syntax (only allow [],[] args)
* add p2wpkh and p2wsh spending test and fixes to sign calls in wallet
* simplify wallet signing calls in cryptoengine
* add p2wpkh engine, add bip84 wallet
Kristaps Kaupe
Adam Gibson
Jules Comte
jules23
chris-belcher
James Hilliard
Tim Akinbo
Daniel Kraft