This commit implements a command line script and a GUI
dialog to receive a payment using the BIP78 protocol,
by setting up an ephemeral hidden service.
It also deprecates the pre-existing inter-Joinmarket
protocol for payjoin payments, since we now have
both sending and receiving support for BIP78. Thus,
much code in Maker, Taker and client-daemon protocol
is removed, as is some documentation in docs/PAYJOIN.md.
Also the script `sendpayment.py` is altered to support
only the BIP78 variant.
The test in jmclient/test/test_payjoin now implements
BIP78 over a TCP connection, while the custom tests in
test/payjoinserver.py can support hidden service based
tests, but the latter is not included in the test suite
and may not always work (it is only for manual
investigations).
The following features of BIP78 are supported:
minfeerate
additionalfeeoutputindex - but *only* for single
change output transactions
maxadditionalfeecontribution
The receiver does not have nor request payment
output substitution.
Utxo selection is no longer sophisticated, instead
we only choose a single utxo to keep the size
increase of the transaction minimal. Thus UIH is
not addressed at the moment.
Errors returned are in line with BIP78.
Sequence numbers are checked by receiver, and
kept identical if uniform, otherwise respected.
Receiver uses transaction monitor to shut down
when the payment is seen.
The workflow is almost entirely implemented in
jmclient/payjoin.py and the command line script
is in scripts/receive-payjoin.py. The setup, including
configuration changes for Tor, are documented in
docs/PAYJOIN.md, including a user guide video linked.
Some instructions here will be redundant with the introductory [usage guide](USAGE.md); sections 1-3 are aimed at users who have not/ will not use Joinmarket for ordinary coinjoins.
So just skip those sections if you already know it.
("display" is optional because default; use `python wallet-tool.py -h` to see all possible methods).
Below is [an example](#sample-testnet-wallet-display-output) of what the wallet looks like (although
Below is [an example](#sample) of what the wallet looks like (although
yours will be mainnet, so the addresses will start with '3' not '2').
Joinmarket by default uses *5* accounts, not only 1 as some other wallets (e.g. Electrum), this is to help
@ -142,6 +150,8 @@ including the 12 word seed, although consider privacy concerns when sending addr
### Doing a PayJoin payment.
This section gives details on how to make payments with Payjoin. You might prefer to start with the video linked [here](#using-qt) to see how this works if you are using JoinmarketQt rather than the command line.
<aname="bip78"/>
#### Using BIP78 payjoins to pay a merchant.
@ -153,8 +163,8 @@ The process here is to use the syntax of sendpayment.py:
```
Notes on this:
* Payjoins BIP78 style are done using the `sendpayment` script (there is no Qt support yet, but it will come later).
* They are done using BIP21 URIs. These can be copy/pasted from a website (e.g. a btcpayserver invoice page), note that double quotes are required because the string contains special characters. Note also that you must see `pj=` in the URI, otherwise payjoin is not supported by that server.
* Payjoins BIP78 style are done using the `sendpayment` script, or by entering the BIP21 URI into the "Recipient" field in JoinmarketQt.
* They are done using BIP21 URIs. These can be copy/pasted from a website (e.g. a btcpayserver invoice page), note that double quotes are required (on the command line) because the string contains special characters. Note also that you must see `pj=` in the URI, otherwise payjoin is not supported by that server.
* If the url in `pj=` is `****.onion` it means you must be using Tor, remember to have Tor running on your system and change the configuration (see below) for sock5 port if necessary. If you are running the Tor browser the port is 9150 instead of 9050.
* Don't forget to specify the mixdepth you are spending from with `-m 0`. The payment amount is of course in the URI, along with the address.
* Pay attention to address type; this point is complicated, but: some servers will not be able to match the address type of the sender, and so won't be able to construct sensible Payjoin transactions. In that case they may fallback to the non-Payjoin payment (which is not a disaster). If you want to do a Payjoin with a server that only supports bech32, you will have to create a new Joinmarket wallet, specifying `native=true` in the `POLICY` section of `joinmarket.cfg` before you generate the wallet.
@ -213,144 +223,163 @@ bump the fee enough to add one input to the transaction, and this should be fine
#### Using Joinmarket-wallet-to-Joinmarket-wallet payjoins
(At the end of this file are full terminal outputs from a regtest run of the process,
you can read it after this to see that it makes sense; there's also a video
[here](https://joinmarket.me/blog/blog/payjoin-basic-demo) of the process running live with mainnet coins).
This is now deprecated; if you still want to use it, use Joinmarket(-clientserver) version 0.7.0 or lower, and see the corresponding older version of this document.
* Receiver needs to start: run (still in scripts/ directory):
Note : `-m 1` is choosing the *mixdepth* (see above) to *spend* coins from: in a payjoin,
the receiver also spends some coins, he just gets that amount back, as well as his payment.
In the joinmarket.cfg file, under `[POLICY]` section you should see a setting called `tx_fees`.
You can set this to any integer; if you set it to 1000 or less then it's treated as a "confirmation in N blocks target",
i.e. if you set it to 3 (the default), the fee is chosen from Bitcoin Core's estimator to target confirmation in 3 blocks.
So if you change it to e.g. 10, it will be cheaper but projected to get its first confirmation in 10 blocks on average.
If you funded into mixdepth 0 at the start, and you only have coins there, you must choose 0
here (which is the default). How much do you need? The code is fairly lenient and it doesn't
matter too much. But it needs to be more than zero!
If you set it to a number > 1000, though, it's a number of satoshis per kilobyte (technically, kilo-vbyte) that you want
to use. **Don't use less than about 1200 if you do this** - a typical figure might be 5000 or 10000, corresponding to
about 5-10 sats/byte, which nowadays is a reasonable fee. The exact amount is randomised by ~20% to avoid you inadvertently
watermarking all your transactions. So don't use <1200becausethenyoumightbeusinglessthan1sat/bytewhichis
difficult to relay on the Bitcoin network.
BIP78 itself has various controls around fees - essentially it tries to let the receiver bump the fee but *only slightly* to account for the fact that the receiver is adding at least one more input and so increasing the size of the transaction, and also ensure that low fees do not accidentally fall too low (even, below the relay limit). Joinmarket's receiver will only add one input and never more, for now, and it looks like this is the tradeoff that most wallets will make. If you want to learn more investigate the `maxadditionalfeecontribution`, `additionalfeeoutputindex` and `minfeerate` parameters described in the BIP.
As a spender in the BIP78 protocol, you will usually see the following: a small reduction in the size of your change output as a result of the extra 1 input. Unless the payment is very small, this probably won't be significant.
<aname="native"/>
#### What if I wanted bech32 native segwit addresses?
You can do this, but bear in mind: PayJoin only gives its full effect if you and your receiver are using the same kind of addresses; so do this only if you and your receiver(s)/sender(s) agree on it - most BIP78 receivers at least for now will only engage in the protocol if they can provide inputs of the same type as yours.
As was noted in the BIP78 section, it may be therefore that you *need* to do this (albeit that the worst that can happen is a fallback to non-payjoin payment, which isn't a disaster).
The receiver will be prompted to read/note the receiving address, amount and "ephemeral nick", and then
the script will just wait (indefinitely). It'll look something like this:
Also note: you *cannot* do Joinmarket coinjoins if you choose a bech32 wallet (this may change in future, see [this PR](https://github.com/JoinMarket-Org/joinmarket-clientserver/pull/656)).
In the configuration file `joinmarket.cfg` (which was created in the preparatory step above), go to the
POLICY section and set:
```
2019-01-16 16:11:40,018 [INFO] Your receiving address is: 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS
2019-01-16 16:11:40,018 [INFO] You will receive amount: 27000000 satoshis.
2019-01-16 16:11:40,018 [INFO] The sender also needs to know your ephemeral nickname: J5AFezpsuV95CBCH
2019-01-16 16:11:40,018 [INFO] This information has been stored in a file payjoin.txt; send it to your counterparty when you are ready.
[POLICY]
native = true
```
The "ephemeral nick" starts with J, version (5 currently), then a short base58 string. It's how the sender will find you in the
"joinmarket trading pit" (multiple IRC servers are used for this currently).
Note that this must be done *before* generating the wallet, as
the wallet internally, of course, stores which type of addresses it manages, and it can only be of two
types currently (ignoring legacy upgrades): bech32 or p2sh-segwit (starting with '3'), the latter being
the default (and the one used in Joinmarket itself).
* Receiver sends data to sender (amount, address, ephemeral nick).
Note that the bech32 style wallet is written to conform to [BIP84](https://github.com/bitcoin/bips/blob/master/bip-0084.mediawiki),
analogous to the BIP49 case for p2sh.
This data is stored in the file payjoin.txt but not currently using any encoding (that's a TODO).
<aname="receiving"/>
* Sender starts up the sendpayment script:
#### Receiving payments using BIP78 Payjoin
Joinmarket allows you to receive payments from any wallet that supports spending via the BIP78 Protocol, using a Tor hidden service.
This hidden service is "ephemeral" meaning it is set up specifically for the payment and discarded after you shut down the receiving process. The setup takes some few seconds but it isn't too slow.
To make this work, you will need to do some minor configuring Tor, the first time. This is explained in detail [below](#torconfig). If you fail to do this step, you will see some rather unintelligible errors about connection failures when trying to run the script described next.
Once that is ready, you can run the `receive-payjoin.py` script something like this:
Notice that the user has specified the three pieces of data given; using the `-T` flags this as a PayJoin; if you don't do this you will be
doing a Joinmarket coinjoin accidentally! (which wouldn't be the end of the world, the receiver would still get the money!).
The arguments provided are the wallet name and the exact amount you intend to receive (here it's 0.32 BTC (flagged as BTC because decimal), but you could also write `32000000` which will be interpreted as satoshis).
After a delay of 5-50 seconds (usually; Tor setup varies unpredictably), you will see a message like this:
As before -m 1 tells the wallet which mixdepth to source coins to spend from; it obviously needs to have at least the given amount
(in this case, 27 million satoshis or 0.27 btc).
```
Your hidden service is available. Please now pass this URI string to the sender to effect the payjoin payment:
Keep this process running until the payment is received.
```
* The two sides communicate
... which should be self explanatory. The sender may be using Joinmarket, or a different wallet that supports Payjoin, like Wasabi. As previously noted, Payjoins will not work if the address type in use by the two wallets is different, so for Wasabi it would be necessary to use a bech32 Joinmarket wallet (as was discussed [here](#native)).
This takes generally only a couple of seconds, not including a relatively slow startup (30s-60s or so depending on configuration),
during which time the sender acts similarly to any other Joinmarket participant (and so does the receiver, likewise), to have
a slightly better "blend in with the crowd" effect.
When the payment goes through you will see a chunk of logging text (most of which is serialized PSBTs being exchanged). If Payjoin was effected as intended, you will see:
Once the transaction is broadcast, you'll see a message to that effect and the script will shutdown. You can check the txid
on your favourite block explorer.
```
date string [INFO] Removed utxos= ...
date string [INFO] Added utxos= ...
date string [INFO] transaction seen on network: hex-txid
done
```
* What if something goes wrong and the payment fails to go through?
where hex-txid is of course the transaction id of the payjoin transaction which paid you.
First, if the sender succeeds in doing the first step, the receiver will have a non-coinjoin ordinary payment transaction
to fall back on. It'll look like this:
If you see:
```
2019-01-16 16:34:55,168 [INFO] Network transaction fee is: 1672 satoshis.
2019-01-16 16:34:55,175 [INFO] We'll use this serialized transaction to broadcast if your counterparty fails to broadcast the payjoin version:
2019-01-16 16:34:55,186 [INFO] The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing..
....
date string [INFO] Payment made without coinjoin. Transaction:
```
If the final step fails and we don't get a PayJoin, the receiver can just use his favorite broadcast method (e.g.
`sendrawtransaction` in Bitcoin Core) to send the signed transaction above (02000...).
followed by a detailed transaction output, it means that some incompatibility or error between the two wallets resulted in the normal non-payjoin (non-coinjoin) payment being sent; you still received your money, so DON'T ask to be paid again just because Payjoin failed! This is part of BIP78 - we recognize that things can go slightly wrong in the arrangement (for example, the wrong address type, or a fee requirement that cannot be met), so allowing normal payments instead is very much *intended behaviour*.
On the other hand, if you see at the end:
```
2020-09-12 13:01:15,887 [WARNING] Payment is not valid. Payment has NOT been made.
```
* Privacy and security controls
it means of course the other case. Double check with your counterparty, something more fundamental has gone wrong because they did not send you a valid non-coinjoin payment, as they were supposed to right at the start.
Just like Joinmarket, do note, that here the private information is communicated with **end-to-end encryption** - it's not like
the people/operators on the IRC servers are going to learn any information about your transaction.
<aname="torconfig"/>
Second, it's very recommended to use Tor connections to the messaging servers via their hidden services. See the
`[MESSAGING:serverN]` sections in the `joinmarket.cfg`. This keeps anyone from seeing your IP address origin.
#### Configuring Tor to setup a hidden service
This altogether means it's possible to have anonymity from your sender/receiver counterparty, if that's an issue.
You *do* still have to send them the payment information out of band, though. We could perhaps fold this in, although
it seems a bit tricky to do so without introducing a security issue.
(These steps were prepared using Ubuntu; you may have to adjust for your distro).
An additional minor feature is that the receiver "fakes" being a Joinmarket maker, so his bot looks the same as the other
ones in the Joinmarket trading pit (at least, to a crude extent). And the sender fakes being a Joinmarket taker, too.
First, ensure you have Tor installed:
Also we try to make the PayJoin look as much as possible as an ordinary payment. For example:
- we make the transaction version, locktime and sequence numbers look similar to those created by Core, Electrum.
- we try to avoid "UIH2", meaning we avoid a situation where one input is larger than any output, since that would
mean no other inputs are needed; wallet coin selection algorithms *usually* don't do that.
Security - since the receiver passively waits, what happens if a bad actor tries to connect to him? Nothing; an attacker
would fail to even start the process without knowing the payment amount and address, which the receiver is not broadcasting
around everywhere (especially not the amount and ephemeral nickname), and even if they knew that, the worst
they can do is learn at least 1 utxo of the receiver. The receiver won't pay attention to non-PayJoin messages, either.
```
sudo apt install tor
```
##### Controlling fees
Don't start the tor daemon yet though, since we need to do some setup. Edit Tor's config file with sudo:
**The fees are paid by the sender of funds; note that the fees are going to be a bit higher than a normal payment** (typically
about 2-3x higher); this may be changed to share the fee, in a future version. There are controls to make sure the fee
isn't *too* high.
```
sudo vim /etc/tor/torrc
```
In the joinmarket.cfg file, under `[POLICY]` section you should see a setting called `tx_fees`.
You can set this to any integer; if you set it to 1000 or less then it's treated as a "confirmation in N blocks target",
i.e. if you set it to 3 (the default), the fee is chosen from Bitcoin Core's estimator to target confirmation in 3 blocks.
So if you change it to e.g. 10, it will be cheaper but projected to get its first confirmation in 10 blocks on average.
and uncomment these two lines to enable hidden service startup:
If you set it to a number > 1000, though, it's a number of satoshis per kilobyte (technically, kilo-vbyte) that you want
to use. **Don't use less than about 1200 if you do this** - a typical figure might be 5000 or 10000, corresponding to
about 5-10 sats/byte, which nowadays is a reasonable fee. The exact amount is randomised by ~20% to avoid you inadvertently
watermarking all your transactions. So don't use <1200becausethenyoumightbeusinglessthan1sat/bytewhichis
difficult to relay on the Bitcoin network.
```
ControlPort 9051
CookieAuthentication 1
```
<aname="native"/>
However if you proceed at this point to try to run `receive-payjoin.py` as outlined above, you will almost certainly get an error like this:
#### What if I wanted bech32 native segwit addresses?
You can do this, but bear in mind: PayJoin only gives its full effect if you and your receiver are using the same kind of addresses; so do this only if you and your receiver(s)/sender(s) agree on it.
... because reading this file requires being a member of the group `debian-tor`. So add your user to this group:
As was noted in the BIP78 section, it may be therefore that you *need* to do this (albeit that the worst that can happen is a fallback to non-payjoin payment, which isn't a disaster).
```
sudo usermod -a -G debian-tor yourusername
```
Also note: you *cannot* do Joinmarket coinjoins if you choose a bech32 wallet (this may change in future).
... and then you must *restart the computer/server* for that change to take effect (check it with `groups yourusername`).
In the configuration file `joinmarket.cfg` (which was created in the preparatory step above), go to the
POLICY section and set:
Finally, after system restart, ensure Tor is started (it may be automatically, but anyway):
```
[POLICY]
native = true
sudo service tor start
```
Note that this must be done *before* generating the wallet, as
the wallet internally, of course, stores which type of addresses it manages, and it can only be of two
types currently (ignoring legacy upgrades): bech32 or p2sh-segwit (starting with '3'), the latter being
the default (and the one used in Joinmarket itself).
Once this is done, you should be able to run the BIP 78 receiver script, or [JoinmarketQt](#using-qt) and a hidden service will be automatically created for you from now on.
Note that the bech32 style wallet is written to conform to [BIP84](https://github.com/bitcoin/bips/blob/master/bip-0084.mediawiki),
analogous to the BIP49 case for p2sh.
<aname="using-qt"/>
### Using JoinmarketQt to send and receive Payjoins
All of the configuration details above apply to this scenario (for example, setting up Tor if you want to act as receiver.
But for the workflow on the GUI application, this video explains what to do:
2019-01-16 16:34:55,186 [INFO] The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing..
2019-01-16 16:34:55,187 [INFO] We selected inputs worth: 200000000
2019-01-16 16:34:55,195 [INFO] We estimated a fee of: 2585
2019-01-16 16:34:55,196 [INFO] We calculated a new change amount of: 172997415
2019-01-16 16:34:55,197 [INFO] We calculated a new destination amount of: 227000000
2019-01-16 16:35:00,232 [INFO] The transaction has been broadcast.
2019-01-16 16:35:00,232 [INFO] Txid is: 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae
2019-01-16 16:35:00,233 [INFO] Transaction in detail: {'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52',
Adam Gibson
5 years ago