This commit changes how timelocked addresses are created from the seed and bip32 tree.
A good thing to do would be to have each locktime (e.g. 1st jan 2020,
1st feb 2020, 1st march 2020, etc) actually use a different pubkey from
the HD tree (i.e. ("m/84'/1'/0'/2/0" + 1st jan 2020) ("m/84'/1'/0'/2/1" + 1st feb 2020), etc).
This now means that the sync code doesnt need to know what keys have been associated with
a fidelity bond to scan for the next one. Previously when a user funded a single timelocked
address, the wallet will generate _another_ pubkey and import _another_ ~960 addresses, so
funding one address would actually mean watching and generating ~1920 addresses not ~960.
This should help with the problem found by some people that fidelity bond wallets are slower
to sync. Other optimizations are possible but the structure of fidelity bond wallets will
probably be fixed for decades, so this change is worth doing now.
Parse incoming and announce outgoing fidelity bond messages
Fidelity bond proof messages will be checked and added to the internal
database just like offers. Such messages are not announced in public
but only directly to takers who ask for them, this is because the
signature proofs must commmit to the maker's and taker's IRC nicknames
in order to avoid replay attacks.
The default minfeerate in payjoin is 1.1 sat/vbyte,
however the randomization of 20% for all transactions
created with direct_send means that the previous value
of 1.3 could fall to below 1.1 probabilistically, it
is therefore bumped to a value that never falls below
1.1 after randomization.
Fixes#797.
Adds `custom_change_addr` argument to `direct_send()`
joinmarket-qt: Adds input field for optional external change address
joinmarket-qt: Better handle PayJoin/CoinJoin state changes for changeInput widget
Adds `custom_change_address` argument to Taker constructor and use it in joinmarket-qt
Custom change also allowed in sendpayment CLI with `-u` flag (not
supported in tumbler).
Explicitly disallows using custom change with BIP78 Payjoin, though that
could change later.
Both sendpayment and CLI provide detailed warnings to avoid misuse. In
particular, they have an extra warning for using a nonstandard or
non-wallet scriptpubkey type.
Setting custom change to the recipient address is explicitly forbidden.
Tests: Adds custom_change usage test in test_taker.
Prior to this commit, the receiver code was assuming only
2 inputs always, when it decided how to change the input
ordering (randomly doing a reversal), but this is not correct
according to the BIP78 spec, which requires that the
receiver's inputs are *inserted* randomly, without changing
the ordering of the existing (sender) inputs. After this
commit, the BIP78 protocol is adhered to for any number of
inputs.
Added test for random_insert and for payjoin with 3 inputs
Verification of message signatures against segwit addresses
is not currently functional/possible in Core, and additionally
the signing function used for messages in jmbitcoin, derived
from coincurve, was not compatible with Electrum either.
This commit uses the functionality in python-bitcointx's
signmessage module, and now the wallet method `signmessage`
creates signatures on messages against segwit addresses that
are verifiable in Electrum.
Test cases of p2sh and native are added.
This completes the task of enabling
network isolation by running the receiver
side using a hidden service in the daemon,
and communicating over AMP, as is already
the case for the sender.
Updates test_payjoin for daemon receiver.
Qt BIP78 receiver update for daemon.
This PR creates a client-daemon protocol for
the BIP78 sender, using the base protocol
`HTTPPassThrough` which provides tor and non-tor
agents with POST and GET request functionality.
As for Joinmarket coinjoins, the use of an in-process
daemon is the default option, but it can be isolated
by changing the `[DAEMON]` section of the config.
The receiver side of BIP78 will be addressed in a
future PR.
Prior to this commit, several test functions were using
"True" to flag internal and "False" to flag external for
the HD branch for the wallet, but we now use BaseWallet vars
ADDRESS_TYPE_[IN/EX]TERNAL (1/0), so this
is changed to explicitly reference those. There is no change
to the live code (which calls get_[internal/external]_addr).
In addition _index_cache updates in wallet are protected
with a wrapper function to ensure that the branch requested
is valid.
Tests pass both before and after this change.
... with wallet-tool method `signpsbt`. Specifically,
supports co-signing of PSBTs prepared elsewhere (so,
basic Updater/Signer and Finalizer roles, but not
Creator).
Provides detailed user feedback, command line only, and
supports broadcast of finalized transactions.
Supports native and p2sh segwit (and mixed, of course).
Also adds tests using externally prepared PSBTs.
Documentation of PSBT function added to USAGE.md.
Allow YG settings to be saved to joinmarket.cfg.
Before this commit, yield generator settings were
set inside the python script, which is not good.
After this commit, the order of precedence for the
settings for a yield generator is:
* Command line arguments, or
* Settings in [YIELDGENERATOR] in joinmarket.cfg, or
* default config settings in jmclient.configure.py
Before this commit, the taker would choose offers from
the pit based on the setting of `native` in the `[POLICY]`
section of the config object; however this could lead to
users unwittingly choosing the wrong offer type, i.e. one
that is incompatible with their own wallets, which could
result in coinjoins with mixed address types.
This commit fixes that error by only selecting offers that
are compatible with the return value of `BaseWallet.get_txtype()`.
Also fixes up tests for this wallet type enforcement.
This commit updates sendtomany.py and add-utxo.py to reflect
the new default wallet type (p2wpkh) and the new bitcoin
backend code; in doing this it slightly modifies the functions
in commitment_utils.py.
* yg scripts set reloffer/absoffer only:
Prior to this commit, the yield generator user
level scripts required the user to specify offer
types depending on the wallet, but only 'rel/abs'
distinction is user choice; the other element
(native segwit, p2sh or p2pkh) must be defined
by the wallet, so we now call `wallet.get_txtype()`
to translate from reloffer/absoffer to sw0.. etc.
* Taker chooses nversion, nlocktime per wallet type:
Takers who are still using p2sh-p2wpkh wallets will
not want to flag their transactions with different
tx metadata than previous versions that are still
running, so we check the `get_txtype()` output to
decide which nVersion and nLockTime to use.
Also, the SNICKER locktime is reverted to zero as
according to draft spec.
* change offer type in test_coinjoin
* update docs for bech32 wallets
Also, ensure witness_utxo field is populated,
plus minor bugfixes related to presence of
NONWITNESS_UTXO field in provided payment PSBT.
Tested as being functional either with or without
NONWITNESS_UTXO field, for all-segwit inputs.
Also manually fire order creation in coinjoin tests.
This clarification and test change is required due
to the fact that LoopingCalls are designed to fire
immediately by default, before the reactor is
initialized (and therefore in a `running` state),
making it not possible to shutdown the reactor as
a result of events happening in that first call;
so we delay the first call of the maker's orderbook
populating code, so that if a no-coins error
occurs, it will actually shut down the reactor and
hence the whole yield generator program, as intended.
This commit implements a command line script and a GUI
dialog to receive a payment using the BIP78 protocol,
by setting up an ephemeral hidden service.
It also deprecates the pre-existing inter-Joinmarket
protocol for payjoin payments, since we now have
both sending and receiving support for BIP78. Thus,
much code in Maker, Taker and client-daemon protocol
is removed, as is some documentation in docs/PAYJOIN.md.
Also the script `sendpayment.py` is altered to support
only the BIP78 variant.
The test in jmclient/test/test_payjoin now implements
BIP78 over a TCP connection, while the custom tests in
test/payjoinserver.py can support hidden service based
tests, but the latter is not included in the test suite
and may not always work (it is only for manual
investigations).
The following features of BIP78 are supported:
minfeerate
additionalfeeoutputindex - but *only* for single
change output transactions
maxadditionalfeecontribution
The receiver does not have nor request payment
output substitution.
Utxo selection is no longer sophisticated, instead
we only choose a single utxo to keep the size
increase of the transaction minimal. Thus UIH is
not addressed at the moment.
Errors returned are in line with BIP78.
Sequence numbers are checked by receiver, and
kept identical if uniform, otherwise respected.
Receiver uses transaction monitor to shut down
when the payment is seen.
The workflow is almost entirely implemented in
jmclient/payjoin.py and the command line script
is in scripts/receive-payjoin.py. The setup, including
configuration changes for Tor, are documented in
docs/PAYJOIN.md, including a user guide video linked.
Upgrade python-bitcointx to 1.1.0:
Address requirements of python-bitcointx 1.1.0:
Specifically, the witness `utxo` field can no longer be
assumed to be of type CTxOut, so we should access the
CTxOut with the field witness_utxo and also when updating
the `utxo` field we now use `set_utxo()`.
Use PartiallySignedTransaction.get_fee() method.
Use PartiallySignedTransaction.set_utxo.
Additionally some minor typos/comment corrections and removal
of the now defunct `apply_freeze_signature`.
Add custom load location for libsecp where needed;
falls back to system installation if Joinmarket custom
installation is not found.
Decode error msg from server in payjoin
Cleanup test file test_proposals.txt (delete after test)
Human readable function names (names for human readable
conversions are now themselves human readable).
Remove unused get_*_vbyte functions and cleanup
Removes old unused files (electrum*.py).
Fixes core nohistory sync test to use both standard
wallet types, and fixes address import counter.
Fixes that test to use the right chain params so that
native segwit wallets can work in regtest with
nohistory mode.
Removes some now unneeded imports.
Fixes commontest.create_wallet_for_sync to hash all
parameters, including optional ones.
Replaces usage of binascii.hexlify with bintohex.
Note in particular that:
bitcoin.mktx in this PR now does support script
entries in outputs to account for nonstandard
destinations (as is needed for burn).
bitcoin.sign now supports p2wsh (as is needed
for timelocks).
Human readable representation for CTransaction
objects in jmbitcoin.secp256k1_transaction.py and for
PartiallySignedTransaction objects in jmclient.wallet.
PSBTWalletMixin, use of these in maker, taker, direct
send and in tests. Users should note that PSBT human
readable representations can in some cases be really
huge.
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
This commit uses the now created PSBTWalletMixin and additionally
creates a SNICKERWalletMixin, and adds a SNICKERReceiver object
to jmclient. A test of the end to end workflow of create and then
co-sign a SNICKER coinjoin as per the draft BIP is in test_snicker.
Additional changes:
updated python-bitcointx dependency to >=1.0.5
Minor refactoring of callbacks in tests and additional redeem
script checks to PSBTWalletMixin.sign_psbt.
Note that this work replaces #403 .
Subclassed btcointx.CKeyBase to create a privkey type
that uses our signing code.
These are then used to allow signing of inputs in
transactions owned by our wallet.
Created a PSBTWalletMixin to perform the above function,
and added it to our three wallet types.
Wrote a detailed creation, sign and broadcast test
for a psbt using utxos from within and outside the wallet.
Detailed tests across all 3 wallet types for psbt
Tests cover also mixed inputs of different
types and owned/unowned. Direct send now exported to be used
in tests rather than only script usage, also supports returning
a tx object rather than only a txid.
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
Fixes#442.
First, the CONNREFUSED socket error is handled in jsonrpc.
Second, we respond to this (but *not* to resets) with a reactor
shutdown in BitcoinCoreInterface.rpc(). This also necessitates
early-quitting in the calling function
(WalletService.transaction_monitor) since the reactor stop
will only stop future deferred calls, not the currently running
one. The obvious sys.exit approach is only used in startup,
because the reactor is not currently running at that point.
Also minor change to DummyBlockchainInterface for test.
chris-belcher
Adam Gibson
csH7KmCC9
Kristaps Kaupe
PulpCattel
dchoe
Jules Comte
jules23