Merge #279: Implement payjoin (p2ep) direct payment joins

28abddf Implement payjoin (p2ep) direct payment joins (AdamISZ)

README.md

@@ -14,6 +14,7 @@ Joinmarket is coinjoin software, and includes a wallet, which requires Bitcoin C
 * Can specify exact amount of coinjoin (figures from 0.01 to 30.0 btc and higher are practical), can choose time and number of counterparties
 * Can run passively to receive small payouts for taking part in coinjoins (see "Maker" and "yield-generator" in docs)
 * GUI to support Taker role, including tumbler/automated coinjoin sequence.
+* PayJoin - more economical and private payments between Joinmarket wallets.
 
 ### Quickstart
 
@@ -45,6 +46,10 @@ If you are running Joinmarket-Qt, you can instead use the [walkthrough](docs/JOI
 
 If you used the old version of Joinmarket, the notes in the [scripts readme](scripts/README.md) help to understand what has and hasn't changed about the scripts.
 
+### PayJoin
+
+If you want to use the PayJoin feature to pay/receive money to/from another Joinmarket wallet user, read [this guide](docs/PAYJOIN.md).
+
 ### Joinmarket-Qt
 
 Provides single join and multi-join/tumbler functionality (i.e. "Taker") only, in a GUI.

docs/PAYJOIN.md

@@ -0,0 +1,384 @@
+### PayJoin (aka P2EP) user guide.
+
+(You've installed using the `install.sh` as per instructions in the README before
+reading this).
+
+This document does **not** discuss why PayJoin is interesting or the general concept.
+For that, see [this](TODO-BLOGPOST) post.
+
+Some instructions here will be redundant with the introductory [usage guide](USAGE.md);
+this guide is aimed at users who have not/ will not use Joinmarket for ordinary coinjoins.
+So just skip redundant info if you already know it.
+
+### Preparatory step: configuring for Bitcoin Core.
+
+Joinmarket currently requires a Bitcoin Core full node, although it can be pruned.
+
+First thing to do: in `scripts/`, run:
+
+    python wallet-tool.py generate
+
+This *should* quit with an error, because the rpc is not configured. Open the newly created file `joinmarket.cfg`,
+and edit:
+
+    [BLOCKCHAIN]
+    rpc_user = yourusername-as-in-bitcoin.conf
+    rpc_password = yourpassword-as-in-bitcoin.conf
+    rpc_host = localhost #default usually correct 
+    rpc_port = 8332 # default for mainnet
+
+Note, you can also use a cookie file by setting, in this section, a variable `rpc_cookie_file` to the location of the file,
+as an alternative to using user/password.
+
+If you use Bitcoin Core's multiwallet feature, you can edit the value of `rpc_wallet_file` to your chosen wallet file.
+
+Then retry the same `generate` command; it should now not error - continue the generate process as per next step.
+
+If you still get rpc connection errors, make sure you can connect to your Core node using the command line first.
+
+### Make and fund the wallet
+
+Continue/complete the wallet generation with the above (`generate`) method.
+
+(Before you finish: want a bech32 wallet? you probably don't,
+but read [this](#what-if-i-wanted-bech32-native-segwit-addresses) in case.)
+
+The wallet you create is BIP49 by default, using BIP39 12 word seed,
+mnemonic extension optional (simplest to leave it out if you're not sure).
+
+Once the `generate` method run has completed, successfully, you need to look at the wallet contents. Use
+the `display` method which is the default:
+
+```
+python wallet-tool.py wallet-name-you-chose.dat [display]
+```
+
+("display" is optional because default; use `python wallet-tool.py -h` to see all possible methods).
+
+Below is [an example](#sample-testnet-wallet-display-output) of what the wallet looks like (although
+yours will be mainnet, so the addresses will start with '3' not '2').
+
+Joinmarket by default uses *5* accounts, not only 1 as some other wallets (e.g. Electrum), this is to help
+with coin isolation. Try to move coins from one account to another *only* via coinjoins; or, you can just
+use one or two accounts (called "mixdepths" in Joinmarket parlance) as if they were just one, understanding
+that if you don't bother to do anything special, the coins in those two mixdepths are linked.
+
+Fund the wallet by choosing one or more addresses in the *external* section of the first account (called
+"Mixdepth 0" here). When you fund, fund to the external addresses. The internals will be used for change.
+
+(The new standard (BIP49) *should* be compatible with TREZOR, Ledger, Electrum, Samourai and some others,
+including the 12 word seed, although consider privacy concerns when sending addresses to remote servers!).
+
+### Doing a PayJoin payment.
+
+(At the end of this file are full terminal outputs from a regtest run of the process,
+you can read it after this to see that it makes sense; there's also a video
+[here](https://joinmarket.me/blog/blog/payjoin-basic-demo) of the process running live with mainnet coins).
+
+* Receiver needs to start: run (still in scripts/ directory):
+
+```
+python receive-payjoin.py -m 1 receiver-wallet-name.jmdat amount-in-satoshis
+```
+
+Note : `-m 1` is choosing the *mixdepth* (see above) to *spend* coins from: in a payjoin,
+the receiver also spends some coins, he just gets that amount back, as well as his payment.
+
+If you funded into mixdepth 0 at the start, and you only have coins there, you must choose 0
+here (which is the default). How much do you need? The code is fairly lenient and it doesn't
+matter too much. But it needs to be more than zero!
+
+The receiver will be prompted to read/note the receiving address, amount and "ephemeral nick", and then
+the script will just wait (indefinitely). It'll look something like this:
+
+```
+2019-01-16 16:11:40,018 [INFO]  Your receiving address is: 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS
+2019-01-16 16:11:40,018 [INFO]  You will receive amount: 27000000 satoshis.
+2019-01-16 16:11:40,018 [INFO]  The sender also needs to know your ephemeral nickname: J5AFezpsuV95CBCH
+2019-01-16 16:11:40,018 [INFO]  This information has been stored in a file payjoin.txt; send it to your counterparty when you are ready.
+```
+
+The "ephemeral nick" starts with J, version (5 currently), then a short base58 string. It's how the sender will find you in the
+"joinmarket trading pit" (multiple IRC servers are used for this currently).
+
+* Receiver sends data to sender (amount, address, ephemeral nick).
+
+This data is stored in the file payjoin.txt but not currently using any encoding (that's a TODO).
+
+* Sender starts up the sendpayment script:
+
+```
+python sendpayment.py -m 1 receiver-wallet.jmdat 27000000 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS -T J5AFezpsuV95CBCH
+```
+
+Notice that the user has specified the three pieces of data given; using the `-T` flags this as a PayJoin; if you don't do this you will be
+doing a Joinmarket coinjoin accidentally! (which wouldn't be the end of the world, the receiver would still get the money!).
+
+As before -m 1 tells the wallet which mixdepth to source coins to spend from; it obviously needs to have at least the given amount
+(in this case, 27 million satoshis or 0.27 btc).
+
+* The two sides communicate
+
+This takes generally only a couple of seconds, not including a relatively slow startup (30s-60s or so depending on configuration),
+during which time the sender acts similarly to any other Joinmarket participant (and so does the receiver, likewise), to have
+a slightly better "blend in with the crowd" effect.
+
+Once the transaction is broadcast, you'll see a message to that effect and the script will shutdown. You can check the txid
+on your favourite block explorer.
+
+* What if something goes wrong and the payment fails to go through?
+
+First, if the sender succeeds in doing the first step, the receiver will have a non-coinjoin ordinary payment transaction
+to fall back on. It'll look like this:
+
+```
+2019-01-16 16:34:55,168 [INFO]  Network transaction fee is: 1672 satoshis.
+2019-01-16 16:34:55,175 [INFO]  We'll use this serialized transaction to broadcast if your counterparty fails to broadcast the payjoin version:
+2019-01-16 16:34:55,177 [INFO]  0200000000010152da645b5a2ec3a166ad8a933b1442fa38fc119faf7659d72033ba863fdd8d470000000017160014297b55001daa905a3552137ef19755cf4eae7babfeffffff02b8be4f0a0000000017a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87c0fc9b010000000017a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc8702483045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a65012102f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f42030000
+2019-01-16 16:34:55,186 [INFO]  The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing..
+....
+```
+
+If the final step fails and we don't get a PayJoin, the receiver can just use his favorite broadcast method (e.g.
+`sendrawtransaction` in Bitcoin Core) to send the signed transaction above (02000...).
+
+* Privacy and security controls
+
+Just like Joinmarket, do note, that here the private information is communicated with **end-to-end encryption** - it's not like
+the people/operators on the IRC servers are going to learn any information about your transaction.
+
+Second, it's very recommended to use Tor connections to the messaging servers via their hidden services. See the
+`[MESSAGING:serverN]` sections in the `joinmarket.cfg`. This keeps anyone from seeing your IP address origin.
+
+This altogether means it's possible to have anonymity from your sender/receiver counterparty, if that's an issue.
+You *do* still have to send them the payment information out of band, though. We could perhaps fold this in, although
+it seems a bit tricky to do so without introducing a security issue.
+
+An additional minor feature is that the receiver "fakes" being a Joinmarket maker, so his bot looks the same as the other
+ones in the Joinmarket trading pit (at least, to a crude extent). And the sender fakes being a Joinmarket taker, too.
+
+Also we try to make the PayJoin look as much as possible as an ordinary payment. For example:
+ - we make the transaction version, locktime and sequence numbers look similar to those created by Core, Electrum.
+ - we try to avoid "UIH2", meaning we avoid a situation where one input is larger than any output, since that would
+   mean no other inputs are needed; wallet coin selection algorithms *usually* don't do that.
+   
+Security - since the receiver passively waits, what happens if a bad actor tries to connect to him? Nothing; an attacker
+would fail to even start the process without knowing the payment amount and address, which the receiver is not broadcasting
+around everywhere (especially not the amount and ephemeral nickname), and even if they knew that, the worst
+they can do is learn at least 1 utxo of the receiver. The receiver won't pay attention to non-PayJoin messages, either.
+
+#### What if I wanted bech32 native segwit addresses?
+
+You can do this, but bear in mind: PayJoin only gives its full effect if you and your receiver are using
+the same kind of addresses; so do this only if you and your receiver(s)/sender(s) agree on it.
+
+Also note: you *cannot* do Joinmarket coinjoins if you choose a bech32 wallet (this may change in future).
+
+In the configuration file `joinmarket.cfg` (which was created in the preparatory step above), go to the
+POLICY section and set:
+
+```
+[POLICY]
+native = true
+```
+
+Note that this must be done *before* generating the wallet, as
+the wallet internally, of course, stores which type of addresses it manages, and it can only be of two
+types currently (ignoring legacy upgrades): bech32 or p2sh-segwit (starting with '3'), the latter being
+the default (and the one used in Joinmarket itself).
+
+Note that the bech32 style wallet is written to conform to [BIP84](https://github.com/bitcoin/bips/blob/master/bip-0084.mediawiki),
+analogous to the BIP49 case for p2sh.
+
+#### Sample testnet wallet display output
+
+```
+JM wallet
+mixdepth	0	tpubDC4qk8DsyiFYY85uktoKaiR1srWLSNRxZ2A4MXYg5LHy9XHKSTcF2tNtFpGz4LzXPcDH1kNkiww7MwipNqNSps6HSyjPTqTB18J7C4jrFMi
+external addresses	m/49'/1'/0'/0	tpubDFFCJfi4y6xPKquA6H6aP5EBZPupb6A9tJz8sErW8GN6D7D3s9MABPt1BczpQ3n8rBv7VLSVpu3ddvb7xEKMfNX2sMZ7XxiBD4J6kpfF7Ag
+m/49'/1'/0'/0/0     	2NBT6npWKxBEG8fkDjSFLDZJ7fNda4kYnAx	2.00000000	deposit
+m/49'/1'/0'/0/1     	2Mt7dBFikYwCQTDU129VTF9ahKWxeJjEUuF	0.00000000	new
+m/49'/1'/0'/0/2     	2N94bte8xWojSaX6yq2th3R3mUhvKzZqDJ9	0.00000000	new
+m/49'/1'/0'/0/3     	2MvSBTkCKwPHLdNPRXbovPXUM8oAfjUFPYc	0.00000000	new
+m/49'/1'/0'/0/4     	2MzVn3Nc3RRyN7shiVajA225xCTcaGn1PRw	0.00000000	new
+m/49'/1'/0'/0/5     	2MxJwv2dmkMupDuBLsEMa4HgG6GAieWHiXr	0.00000000	new
+m/49'/1'/0'/0/6     	2MwhdkeAcnCkam1LdVBQJ7un8syyoYB1HVH	0.00000000	new
+Balance:	2.00000000
+internal addresses	m/49'/1'/0'/1	
+Balance:	0.00000000
+Balance for mixdepth 0:	2.00000000
+mixdepth	1	tpubDC4qk8DsyiFYba4t8cSpadjoLYUdPwV5dAtBpzpPzgaDKPfSP42xNJq48QUKEVGHQRfFej6DCUjQqCKD8TtcqN932f27jmyjXaxVMpksos4
+external addresses	m/49'/1'/1'/0	tpubDEnijtftQiJpVgezdRNyKVVWGr9xKV9RgPQWHYDHpQ5utdHF7Sqh7xMUyNHcpdeKcKdQ753hFbyccRZNEHTUkHLnDaVqoXRo9XkATPtHhCp
+m/49'/1'/1'/0/0     	2NEsN45waxdkqjP5EnsP3K8YjMeZJh2RLx6	2.00000000	deposit
+m/49'/1'/1'/0/1     	2Mzhewfg6fr5jR122txdTShmLi7rH9ZscTm	2.00000000	deposit
+m/49'/1'/1'/0/2     	2N1vn44cv5m6PhRJMbZ1mR8dAJgHybnsT5u	2.00000000	deposit
+m/49'/1'/1'/0/3     	2NFwBsHkQ8mxCJWRJhkxAY28Tj8YuUJio4t	0.00000000	new
+m/49'/1'/1'/0/4     	2NA1YAa2VqHMSH9b2GYGBA8waUMxZTZwW5Z	0.00000000	new
+m/49'/1'/1'/0/5     	2MvLLp4cnP8ZVuWWDZzRCLFM2YcTfo9ALec	0.00000000	new
+m/49'/1'/1'/0/6     	2Mux3ZUHGmaMBiMsPDbQS56gRGGN26jMGzd	0.00000000	new
+m/49'/1'/1'/0/7     	2N3MWFiSHyRY3QLZgb63Vfcp4BGSFs6Y3bV	0.00000000	new
+m/49'/1'/1'/0/8     	2N3z5zKJPTfPc41esRRvMvjdKaSv6D6jqvY	0.00000000	new
+Balance:	6.00000000
+internal addresses	m/49'/1'/1'/1	
+Balance:	0.00000000
+Balance for mixdepth 1:	6.00000000
+mixdepth	2	tpubDC4qk8DsyiFYdQBNNzo3vHq63Gag4eUJT29UaTVNHM89hJk6CshZ9WGemQNDh2LGDXCud8anAQ4UR7n7tSWiJtviR8WJuTB78ZbEHpFNcLH
+external addresses	m/49'/1'/2'/0	tpubDEY4sVvs1TX82DftUUB51Agg4Ln7BksoRGESNWtrWTDntV2fCs5wNrqiPENcXBAEHtnaY9ZaK48PRFEw1GLhcTxdDNHUyuqDd2YyNYKoVAo
+m/49'/1'/2'/0/0     	2NBSnSB3nVN4TA7EcNkhcRsmrdyXSQepDFE	0.00000000	new
+m/49'/1'/2'/0/1     	2NFtrtpdvmCRXMy1fV8w1eLWpF8MC3nre7n	0.00000000	new
+m/49'/1'/2'/0/2     	2NFHJvGLWU7KuNkbo8rzPwfGtCS5RtNDw7c	0.00000000	new
+m/49'/1'/2'/0/3     	2NGJcmeScFnTSzZzSRNHiLL8zjYeA9ngx5A	0.00000000	new
+m/49'/1'/2'/0/4     	2NBUAxKrNQtp49xYYHh9f6YHfo2FvYBP1NL	0.00000000	new
+m/49'/1'/2'/0/5     	2MyQfzKyPTfYT4vTe2d3fyGvQMoGX6GAhcr	0.00000000	new
+Balance:	0.00000000
+internal addresses	m/49'/1'/2'/1	
+Balance:	0.00000000
+Balance for mixdepth 2:	0.00000000
+mixdepth	3	tpubDC4qk8DsyiFYgXoNb3UmiaG2veSTdrLCxEBUsTEQnDgQLCaC8Yg1z1Bcwn4ZQivNhgBxHEWH7j8hbRx7rab2kYLy4r4PXxor4Ho3A5AJvVH
+external addresses	m/49'/1'/3'/0	tpubDEnwog2atqaSLe9xVTTdqxY5ynysqeQPsXuaKrZ6HaYCJcFPY7LmhepmwFTJYTkqf1w5jgLQmoZvREyk9qiq4P2A2fSGTyj62WUE4VjXK76
+m/49'/1'/3'/0/0     	2MtckYQLD6bJZiPXffW7m5rryMao7u4ktng	0.00000000	new
+m/49'/1'/3'/0/1     	2N9KrH5Bi35ZH3DVUwD67eh8MtQD1LLhgCa	0.00000000	new
+m/49'/1'/3'/0/2     	2N6zLabMeXTXfyzQttr6PBV2JdbZPrVd9i8	0.00000000	new
+m/49'/1'/3'/0/3     	2MzYTWMjXcv4NBce5PKcDAceT7gMuYrGovc	0.00000000	new
+m/49'/1'/3'/0/4     	2N6tYykyCZLtvP1RJZvgZ9a7c6xbQaqpE66	0.00000000	new
+m/49'/1'/3'/0/5     	2N2o3eJ9h2BC5q3TzPVqhn9gmgWBjaL67Hu	0.00000000	new
+Balance:	0.00000000
+internal addresses	m/49'/1'/3'/1	
+Balance:	0.00000000
+Balance for mixdepth 3:	0.00000000
+mixdepth	4	tpubDC4qk8DsyiFYj7GF4LV97c9f7Yff1mrwnxkpi5twGSLmesmPyM4xXBWsxMw9ZLFycVVC4TeeX1ESjNP4rVVrJEDVCm7C3UMvZH9vs6srsAi
+external addresses	m/49'/1'/4'/0	tpubDFA3XKgf2ZiusZHr4we5utkrQ9toN5s7QGKndNMrdQFQfjQU6yiiMT65tmXFCPduSc7muLFegAi36pv4LCdRnhpRYp2QUpm1izyrboWSjzV
+m/49'/1'/4'/0/0     	2N18HGRJyaaUwLQFTyfjqZoXGCV8Yv5rbwD	0.00000000	new
+m/49'/1'/4'/0/1     	2Mt94Y4iguYLkBAhsXT5a1P8VMTQ4kxdZJD	0.00000000	new
+m/49'/1'/4'/0/2     	2N5vYyG1gx8ht2JCKkuVqow3Qn51cHrwaxh	0.00000000	new
+m/49'/1'/4'/0/3     	2N44ZtKYu21p1qN4DBoTCYZL2sms6YBeWVW	0.00000000	new
+m/49'/1'/4'/0/4     	2NF9p2b2PfHvXiPmPP5JPq4CRxKn47LPZrW	0.00000000	new
+m/49'/1'/4'/0/5     	2MwgfAvbPc4ASD84WdBYo2FM5bXBVG1rRG9	0.00000000	new
+Balance:	0.00000000
+internal addresses	m/49'/1'/4'/1	
+Balance:	0.00000000
+Balance for mixdepth 4:	0.00000000
+Total balance:	8.00000000
+```
+
+#### Full sender-side log of a regtest run of PayJoin
+
+(Note that the "Received offers from joinmarket pit" message is a privacy feature, we don't actually respond to any offers).
+
+```
+(jmvenv) me@here:~/jm/scripts$ python sendpayment.py -m1 b80d142a466dbf56f518a3a8c017ab85 27000000 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS -T J5AFezpsuV95CBCH
+2019-01-16 16:34:44,877 [WARNING]  Cannot listen on port 27183, trying next port
+2019-01-16 16:34:44,877 [WARNING]  Cannot listen on port 27184, trying next port
+2019-01-16 16:34:44,877 [WARNING]  Cannot listen on port 27185, trying next port
+2019-01-16 16:34:44,878 [WARNING]  Cannot listen on port 27186, trying next port
+2019-01-16 16:34:44,878 [INFO]  Listening on port 27187
+2019-01-16 16:34:47,495 [INFO]  Could not connect to *ALL* servers yet, waiting up to 60 more seconds.
+2019-01-16 16:34:47,496 [INFO]  All IRC servers connected, starting execution.
+2019-01-16 16:34:47,501 [INFO]  JM daemon setup complete
+2019-01-16 16:34:52,508 [INFO]  INFO:Received offers from joinmarket pit
+2019-01-16 16:34:52,531 [INFO]  total estimated amount spent = 27020000
+2019-01-16 16:34:52,594 [INFO]  Makers responded with: ["J5AFezpsuV95CBCH"]
+2019-01-16 16:34:52,600 [INFO]  Obtained proposed payjoin tx
+{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52',
+                       'index': 0},
+          'script': '',
+          'sequence': 4294967294}],
+ 'locktime': 834,
+ 'outs': [{'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87',
+           'value': 172998328},
+          {'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87',
+           'value': 27000000}],
+ 'version': 2}
+2019-01-16 16:34:52,602 [INFO]  INFO:Built tx proposal, sending to receiver.
+2019-01-16 16:34:56,622 [INFO]  Obtained tx from receiver:
+{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52',
+                       'index': 0},
+          'script': '',
+          'sequence': 4294967294,
+          'txinwitness': []},
+         {'outpoint': {'hash': '5e19edacc10298e7761be7231db5fa44ad73903c3690c4b71b443355267d346a',
+                       'index': 0},
+          'script': '160014a5b07a64bff72b442c71761599e0b627c687661f',
+          'sequence': 4294967294,
+          'txinwitness': ['3044022077a811939910a935934f68ac8a70439bab665ce9ad4cd02e42289c3046d606e1022054ef241980733caa7c7c5b6bb1babf96caba83a9fb0a8e091d8358b1cb35fcdb01',
+                          '02c7ceecf0783ecc7530e018397db434af63eda63765c79816f023699a22663926']}],
+ 'locktime': 834,
+ 'outs': [{'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87',
+           'value': 227000000},
+          {'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87',
+           'value': 172997415}],
+ 'version': 2}
+2019-01-16 16:34:56,630 [INFO]  INFO:Network transaction fee is: 2585 satoshis.
+2019-01-16 16:34:56,641 [INFO]  txid = 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae
+2019-01-16 16:34:56,661 [INFO]  Transaction broadcast OK.
+2019-01-16 16:35:01,651 [INFO]  Transaction seen on network, shutting down.
+2019-01-16 16:35:01,652 [INFO]  Txid was: 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae
+```
+
+#### Full receiver-side log of a regtest run of PayJoin
+
+```
+(jmvenv) me@here:~/jm/scripts$ python receive-payjoin.py -m1 84735f364c2cf4c8ddaa614315aeae14 27000000
+2019-01-16 16:11:39,952 [INFO]  offerlist=[{'maxsize': 112412265, 'ordertype': 'swreloffer', 'cjfee': '0.00035792', 'oid': 0, 'minsize': 1095839, 'txfee': 493}]
+2019-01-16 16:11:39,952 [INFO]  starting receive-payjoin
+2019-01-16 16:11:40,011 [WARNING]  Cannot listen on port 27183, trying next port
+2019-01-16 16:11:40,011 [WARNING]  Cannot listen on port 27184, trying next port
+2019-01-16 16:11:40,011 [WARNING]  Cannot listen on port 27185, trying next port
+2019-01-16 16:11:40,012 [INFO]  Listening on port 27186
+2019-01-16 16:11:40,018 [INFO]  Your receiving address is: 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS
+2019-01-16 16:11:40,018 [INFO]  You will receive amount: 27000000 satoshis.
+2019-01-16 16:11:40,018 [INFO]  The sender also needs to know your ephemeral nickname: J5AFezpsuV95CBCH
+2019-01-16 16:11:40,018 [INFO]  This information has been stored in a file payjoin.txt; send it to your counterparty when you are ready.
+2019-01-16 16:35:00+0100 [-] Enter 'y' to wait for the payment:y
+2019-01-16 16:34:32,449 [INFO]  Could not connect to *ALL* servers yet, waiting up to 60 more seconds.
+2019-01-16 16:34:32,450 [INFO]  All IRC servers connected, starting execution.
+2019-01-16 16:34:32,453 [INFO]  JM daemon setup complete
+
+(note time gap here; just waiting)
+
+2019-01-16 16:34:55,163 [INFO]  obtained tx proposal from sender:
+{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52',
+                       'index': 0},
+          'script': '160014297b55001daa905a3552137ef19755cf4eae7bab',
+          'sequence': 4294967294,
+          'txinwitness': ['3045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a6501',
+                          '02f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f']}],
+ 'locktime': 834,
+ 'outs': [{'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87',
+           'value': 172998328},
+          {'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87',
+           'value': 27000000}],
+ 'version': 2}
+2019-01-16 16:34:55,165 [WARNING]  Connection had broken pipe, attempting reconnect.
+2019-01-16 16:34:55,168 [INFO]  Network transaction fee is: 1672 satoshis.
+2019-01-16 16:34:55,175 [INFO]  We'll use this serialized transaction to broadcast if your counterparty fails to broadcast the payjoin version:
+2019-01-16 16:34:55,177 [INFO]  0200000000010152da645b5a2ec3a166ad8a933b1442fa38fc119faf7659d72033ba863fdd8d470000000017160014297b55001daa905a3552137ef19755cf4eae7babfeffffff02b8be4f0a0000000017a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87c0fc9b010000000017a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc8702483045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a65012102f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f42030000
+2019-01-16 16:34:55,186 [INFO]  The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing..
+2019-01-16 16:34:55,187 [INFO]  We selected inputs worth: 200000000
+2019-01-16 16:34:55,195 [INFO]  We estimated a fee of: 2585
+2019-01-16 16:34:55,196 [INFO]  We calculated a new change amount of: 172997415
+2019-01-16 16:34:55,197 [INFO]  We calculated a new destination amount of: 227000000
+2019-01-16 16:35:00,232 [INFO]  The transaction has been broadcast.
+2019-01-16 16:35:00,232 [INFO]  Txid is: 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae
+2019-01-16 16:35:00,233 [INFO]  Transaction in detail: {'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52',
+                       'index': 0},
+          'script': '160014297b55001daa905a3552137ef19755cf4eae7bab',
+          'sequence': 4294967294,
+          'txinwitness': ['3045022100c1d579fb19f5b8710b407e3995657313d0b1c9fe1180b852834b2210ba19b5e30220464f034eef8d8975b4fe10db7edbf14fcfdc6d71ada3d2063d34fa5c04a38cb801',
+                          '02f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f']},
+         {'outpoint': {'hash': '5e19edacc10298e7761be7231db5fa44ad73903c3690c4b71b443355267d346a',
+                       'index': 0},
+          'script': '160014a5b07a64bff72b442c71761599e0b627c687661f',
+          'sequence': 4294967294,
+          'txinwitness': ['3044022077a811939910a935934f68ac8a70439bab665ce9ad4cd02e42289c3046d606e1022054ef241980733caa7c7c5b6bb1babf96caba83a9fb0a8e091d8358b1cb35fcdb01',
+                          '02c7ceecf0783ecc7530e018397db434af63eda63765c79816f023699a22663926']}],
+ 'locktime': 834,
+ 'outs': [{'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87',
+           'value': 227000000},
+          {'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87',
+           'value': 172997415}],
+ 'version': 2}
+2019-01-16 16:35:00,234 [INFO]  shutting down.
+```
+
+

jmbitcoin/jmbitcoin/secp256k1_main.py

@@ -584,6 +584,13 @@ def ecdsa_raw_verify(msg, pub, sig, usehex, rawmsg=False):
 
 def estimate_tx_size(ins, outs, txtype='p2pkh'):
     '''Estimate transaction size.
+    The txtype field as detailed below is used to distinguish
+    the type, but there is at least one source of meaningful roughness:
+    we assume the output types are the same as the input (to be fair,
+    outputs only contribute a little to the overall total). This combined
+    with a few bytes variation in signature sizes means we will expect,
+    say, 10% inaccuracy here.
+
     Assuming p2pkh:
     out: 8+1+3+2+20=34, in: 1+32+4+1+1+~73+1+1+33=147,
     ver:4,seq:4, +2 (len in,out)
@@ -592,6 +599,16 @@ def estimate_tx_size(ins, outs, txtype='p2pkh'):
     "ins" must contain M, N so ins= (numins, M, N) (crude assuming all same)
     74*M + 34*N + 45 per input, so total ins ~ len_ins * (45+74M+34N)
     so total ~ 34*len_out + (45+74M+34N)*len_in + 10
+    Assuming p2sh-p2wpkh:
+    witness are roughly 3+~73+33 for each input
+    (txid, vin, 4+20 for witness program encoded as scriptsig, 4 for sequence)
+    non-witness input fields are roughly 32+4+4+20+4=64, so total becomes
+    n_in * 64 + 4(ver) + 4(locktime) + n_out*34
+    Assuming p2wpkh native:
+    witness as previous case
+    non-witness loses the 24 witnessprogram, replaced with 1 zero,
+    in the scriptSig, so becomes:
+    n_in * 41 + 4(ver) + 4(locktime) +2 (len in, out) + n_out*34
     '''
     if txtype == 'p2pkh':
         return 10 + ins * 147 + 34 * outs
@@ -599,12 +616,15 @@ def estimate_tx_size(ins, outs, txtype='p2pkh'):
         #return the estimate for the witness and non-witness
         #portions of the transaction, assuming that all the inputs
         #are of segwit type p2sh-p2wpkh
-        #witness are roughly 3+~73+33 for each input
+        # Note as of Jan19: this misses 2 bytes (trivial) for len in, out
-        #non-witness input fields are roughly 32+4+4+20+4=64, so total becomes
+        # and also overestimates output size by 2 bytes.
-        #n_in * 64 + 4(ver) + 4(locktime) + n_out*34 + n_in * 109
         witness_estimate = ins*109
         non_witness_estimate = 4 + 4 + outs*34 + ins*64
         return (witness_estimate, non_witness_estimate)
+    elif txtype == 'p2wpkh':
+        witness_estimate = ins*109
+        non_witness_estimate = 4 + 4 + 2 + outs*31 + ins*41
+        return (witness_estimate, non_witness_estimate)
     elif txtype == 'p2shMofN':
         ins, M, N = ins
         return 10 + (45 + 74*M + 34*N) * ins + 34 * outs

jmbitcoin/jmbitcoin/secp256k1_transaction.py

@@ -438,7 +438,7 @@ def is_segwit_native_script(script):
         return True
     return False
 
-def script_to_address(script, vbyte=0, witver=0):
+def script_to_address(script, vbyte=b'\x00', witver=0):
     """ Given a hex or bytes script, and optionally a version byte
     (for P2SH) and/or a witness version (for native segwit witness
     programs), convert to a valid address (either bech32 or Base58CE).
@@ -452,7 +452,7 @@ def script_to_address(script, vbyte=0, witver=0):
         script = binascii.unhexlify(script)
     if is_segwit_native_script(script):
         #hrp interpreted from the vbyte entry, TODO: better way?
-        if vbyte in [0, 5]:
+        if vbyte in [b'\x00', b'\x05']:
             hrp = 'bc'
         elif vbyte == 100:
             hrp = 'bcrt'
@@ -465,7 +465,7 @@ def script_to_address(script, vbyte=0, witver=0):
         return bin_to_b58check(script[3:-2], vbyte)
     else:
         # BIP0016 scripthash addresses: requires explicit vbyte set
-        if vbyte == 0: raise Exception("Invalid version byte for P2SH")
+        if vbyte == b'\x00': raise Exception("Invalid version byte for P2SH")
         return bin_to_b58check(script[2:-1], vbyte)
 
 def pubkey_to_script(pubkey, script_pre, script_post=b'',
@@ -820,8 +820,23 @@ def apply_multisignatures(*args):
     txobj["ins"][i]["script"] = serialize_script([None] + sigs + [script])
     return serialize(txobj)
 
-def mktx(ins, outs, version=1):
+def mktx(ins, outs, version=1, locktime=0):
-    txobj = {"locktime": 0, "version": version, "ins": [], "outs": []}
+    """ Given a list of input dicts with key "output"
+    which are txid:n strings in hex, and a list of outputs
+    which are dicts with keys "address", "value", outputs
+    a hex serialized tranasction encoding this data.
+    Tx version and locktime are optionally set, for non-default
+    locktimes, inputs are given nSequence as per below comment.
+    """
+    txobj = {"locktime": locktime, "version": version, "ins": [], "outs": []}
+    # This does NOT trigger rbf and mimics Core's standard behaviour as of
+    # Jan 2019.
+    # Tx creators wishing to use rbf will need to set it explicitly outside
+    # of this function.
+    if locktime != 0:
+        sequence = 0xffffffff - 1
+    else:
+        sequence = 0xffffffff
     for i in ins:
         if isinstance(i, dict) and "outpoint" in i:
             txobj["ins"].append(i)
@@ -832,7 +847,7 @@ def mktx(ins, outs, version=1):
                 "outpoint": {"hash": i[:64],
                              "index": int(i[65:])},
                 "script": "",
-                "sequence": 4294967295
+                "sequence": sequence
             })
     for o in outs:
         if isinstance(o, str):

jmclient/jmclient/__init__.py

@@ -8,21 +8,22 @@ from .support import (calc_cj_fee, choose_sweep_orders, choose_orders,
                       cheapest_order_choose, weighted_order_choose,
                       rand_norm_array, rand_pow_array, rand_exp_array, select,
                       select_gradual, select_greedy, select_greediest,
-                      get_random_bytes, random_under_max_order_choose)
+                      get_random_bytes, random_under_max_order_choose,
+                      select_one_utxo)
 from .jsonrpc import JsonRpcError, JsonRpcConnectionError, JsonRpc
 from .old_mnemonic import mn_decode, mn_encode
-from .taker import Taker
+from .taker import Taker, P2EPTaker
 from .wallet import (Mnemonic, estimate_tx_fee, WalletError, BaseWallet, ImportWalletMixin,
                      BIP39WalletMixin, BIP32Wallet, BIP49Wallet, LegacyWallet,
                      SegwitWallet, SegwitLegacyWallet, UTXOManager,
-                     WALLET_IMPLEMENTATIONS)
+                     WALLET_IMPLEMENTATIONS, make_shuffled_tx)
 from .storage import (Argon2Hash, Storage, StorageError,
                       StoragePasswordError, VolatileStorage)
 from .cryptoengine import BTCEngine, BTC_P2PKH, BTC_P2SH_P2WPKH, EngineError
 from .configure import (
     load_program_config, get_p2pk_vbyte, jm_single, get_network,
     validate_address, get_irc_mchannels, get_blockchain_interface_instance,
-    get_p2sh_vbyte, set_config, is_segwit_mode)
+    get_p2sh_vbyte, set_config, is_segwit_mode, is_native_segwit_mode)
 from .blockchaininterface import (BlockchainInterface, sync_wallet,
                                   RegtestBitcoinCoreInterface, BitcoinCoreInterface)
 from .electruminterface import ElectrumInterface
@@ -45,7 +46,7 @@ from .wallet_utils import (
     wallet_tool_main, wallet_generate_recover_bip39, open_wallet,
     open_test_wallet_maybe, create_wallet, get_wallet_cls, get_wallet_path,
     wallet_display)
-from .maker import Maker
+from .maker import Maker, P2EPMaker
 from .yieldgenerator import YieldGenerator, YieldGeneratorBasic, ygmain
 # Set default logging handler to avoid "No handler found" warnings.
 

jmclient/jmclient/client_protocol.py

@@ -72,6 +72,9 @@ class JMClientProtocol(amp.AMP):
         #The constructed length will be 1 + 1 + NICK_MAX_ENCODED
         self.nick = self.nick_header + str(self.jm_version) + self.nick_pkh
         jm_single().nickname = self.nick
+        informuser = getattr(self.client, "inform_user_details", None)
+        if callable(informuser):
+            informuser()
 
     @commands.JMInitProto.responder
     def on_JM_INIT_PROTO(self, nick_hash_length, nick_max_encoded,
@@ -126,6 +129,41 @@ class JMClientProtocol(amp.AMP):
         self.defaultCallbacks(d)
         return {'accepted': True}
 
+    def make_tx(self, nick_list, txhex):
+        d = self.callRemote(commands.JMMakeTx,
+                            nick_list= json.dumps(nick_list),
+                            txhex=txhex)
+        self.defaultCallbacks(d)
+
+    def on_p2ep_tx_received(self, nick, txhex):
+        """ This is called by both "maker" and "taker"
+        in the p2ep protocol; taker sends signed fallback
+        non-coinjoin transaction to maker, then maker sends
+        partially signed payjoin to taker.
+        Processing at client protocol level is the
+        same, business logic deferred to the P2EPMaker,Taker
+        classes.
+        """
+        retval = self.client.on_tx_received(nick, txhex)
+        if not retval[0]:
+            if retval[1] == "OK":
+                jlog.info("Transaction broadcast OK.")
+            else:
+                jlog.info("We refused to continue on receiving tx")
+                jlog.info("Reason: " + retval[1])
+        else:
+            # This will only be called on the "maker"/receiver
+            # side, who will pass back the new partially signed
+            # version; on Taker side, we stop above, since we
+            # just sign and push within the P2EPTaker.
+            nick, txhex = retval[1:]
+            # make_tx is slightly misnamed; it just transfers
+            # the transaction to the given counterparties (here
+            # only one) over the message channel, via the
+            # AMP command JMMakeTx.
+            self.make_tx([nick], txhex)
+        return {"accepted": True}
+
 class JMMakerClientProtocol(JMClientProtocol):
     def __init__(self, factory, maker, nick_priv=None):
         self.factory = factory
@@ -209,6 +247,11 @@ class JMMakerClientProtocol(JMClientProtocol):
 
     @commands.JMTXReceived.responder
     def on_JM_TX_RECEIVED(self, nick, txhex, offer):
+        # "none" flags p2ep protocol; pass through to the generic
+        # on_tx handler for that:
+        if offer == "none":
+            return self.on_p2ep_tx_received(nick, txhex)
+
         offer = json.loads(offer)
         retval = self.client.on_tx_received(nick, txhex, offer)
         if not retval[0]:
@@ -437,16 +480,20 @@ class JMTakerClientProtocol(JMClientProtocol):
             self.push_tx(nick_to_use, txhex)
         return {'accepted': True}
 
+    @commands.JMTXReceived.responder
+    def on_JM_TX_RECEIVED(self, nick, txhex, offer):
+        """ Only used in the P2EP protocol.
+        """
+        if not offer == "none":
+            # Protocol error; this message should only ever
+            # be received, Taker side, in the P2EP protocol.
+            raise JMProtocolError("Taker received unexpected JMTXReceived")
+        return self.on_p2ep_tx_received(nick, txhex)
+
     def get_offers(self):
         d = self.callRemote(commands.JMRequestOffers)
         self.defaultCallbacks(d)
 
-    def make_tx(self, nick_list, txhex):
-        d = self.callRemote(commands.JMMakeTx,
-                            nick_list= json.dumps(nick_list),
-                            txhex=txhex)
-        self.defaultCallbacks(d)
-
     def push_tx(self, nick_to_push, txhex_to_push):
         d = self.callRemote(commands.JMPushTx, nick=str(nick_to_push),
                             txhex=str(txhex_to_push))
@@ -470,21 +517,26 @@ class JMClientProtocolFactory(protocol.ClientFactory):
     def buildProtocol(self, addr):
         return self.protocol(self, self.client)
 
-def start_reactor(host, port, factory, ish=True, daemon=False, rs=True, gui=False): #pragma: no cover
+def start_reactor(host, port, factory, ish=True, daemon=False, rs=True,
+                  gui=False, p2ep=False): #pragma: no cover
     #(Cannot start the reactor in tests)
     #Not used in prod (twisted logging):
     #startLogging(stdout)
     usessl = True if jm_single().config.get("DAEMON", "use_ssl") != 'false' else False
     if daemon:
         try:
-            from jmdaemon import JMDaemonServerProtocolFactory, start_daemon
+            from jmdaemon import JMDaemonServerProtocolFactory, start_daemon,\
+                 P2EPDaemonServerProtocolFactory
         except ImportError:
             jlog.error("Cannot start daemon without jmdaemon package; "
                        "either install it, and restart, or, if you want "
                        "to run the daemon separately, edit the DAEMON "
                        "section of the config. Quitting.")
             return
-        dfactory = JMDaemonServerProtocolFactory()
+        if not p2ep:
+            dfactory = JMDaemonServerProtocolFactory()
+        else:
+            dfactory = P2EPDaemonServerProtocolFactory()
         orgport = port
         while True:
             try:

jmclient/jmclient/configure.py

@@ -175,9 +175,15 @@ unconfirm_timeout_sec = 180
 confirm_timeout_hours = 6
 
 [POLICY]
-#Use segwit style wallets and transactions
+# Use segwit style wallets and transactions
+# Only set to false for old wallets, Joinmarket is now segwit only.
 segwit = true
 
+# Use native segwit (bech32) wallet. This is NOT
+# currently supported in Joinmarket coinjoins. Only set to "true"
+# if specifically advised to do so.
+native = false
+
 # for dust sweeping, try merge_algorithm = gradual
 # for more rapid dust sweeping, try merge_algorithm = greedy
 # for most rapid dust sweeping, try merge_algorithm = greediest
@@ -352,7 +358,12 @@ def validate_address(addr):
     try:
         assert len(addr) > 2
         if addr[:2].lower() in ['bc', 'tb']:
-            #Enforce testnet/mainnet per config
+            # Regtest special case
+            if addr[:4] == 'bcrt':
+                if btc.bech32addr_decode('bcrt', addr)[1]:
+                    return True, 'address validated'
+                return False, 'Invalid bech32 regtest address'
+            #Else, enforce testnet/mainnet per config
             if get_network() == "testnet":
                 hrpreq = 'tb'
             else:
@@ -526,3 +537,8 @@ def get_blockchain_interface_instance(_config):
 
 def is_segwit_mode():
     return jm_single().config.get('POLICY', 'segwit') != 'false'
+
+def is_native_segwit_mode():
+    if not is_segwit_mode():
+        return False
+    return jm_single().config.get('POLICY', 'native') != 'false'

jmclient/jmclient/cryptoengine.py

@@ -11,7 +11,7 @@ import jmbitcoin as btc
 from .configure import get_network
 
 
-TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH = range(3)
+TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, TYPE_P2SH_M_N = range(4)
 NET_MAINNET, NET_TESTNET = range(2)
 NET_MAP = {'mainnet': NET_MAINNET, 'testnet': NET_TESTNET}
 WIF_PREFIX_MAP = {'mainnet': b'\x80', 'testnet': b'\xef'}

jmclient/jmclient/maker.py

@@ -2,9 +2,10 @@
 from __future__ import (absolute_import, division,
                         print_function, unicode_literals)
 from builtins import * # noqa: F401
-
+from future.utils import iteritems
 import base64
 import pprint
+import random
 import sys
 import abc
 from binascii import unhexlify
@@ -12,11 +13,12 @@ from binascii import unhexlify
 
 from jmbitcoin import SerializationError, SerializationTruncationError
 import jmbitcoin as btc
+from jmclient.wallet import estimate_tx_fee, make_shuffled_tx
 from jmclient.configure import jm_single
 from jmbase.support import get_log
-from jmclient.support import (calc_cj_fee)
+from jmclient.support import calc_cj_fee, select_one_utxo
 from jmclient.podle import verify_podle, PoDLE, PoDLEError
-from twisted.internet import task
+from twisted.internet import task, reactor
 from .cryptoengine import EngineError
 
 jlog = get_log()
@@ -284,3 +286,374 @@ class Maker(object):
         """Performs actions on receipt of 1st confirmation of
         a transaction into a block (e.g. announce orders)
         """
+
+class P2EPMaker(Maker):
+    """ The P2EP Maker object is instantiated for a specific payment,
+    with a specific address and expected payment amount. It inherits
+    normal Maker behaviour on startup and makes fake offers, which
+    it does not follow up in direct peer interaction (to be specific:
+    `!fill` requests in privmsg are simply ignored). Under the hood,
+    the daemon protocol will allow pubkey exchange with any counterparty,
+    but only after the Taker makes a !tx proposal matching our intended
+    address and payment amount, which were agreed out of band with the
+    sender(Taker) counterparty, do we pass over our intended inputs
+    and partially signed transaction, thus information leak to snoopers
+    is not possible.
+    """
+    def __init__(self, wallet, mixdepth, amount):
+        self.receiving_amount = amount
+        self.mixdepth = mixdepth
+        # destination mixdepth must be different from that
+        # which we source coins from; use the standard "next"
+        dest_mixdepth = (self.mixdepth + 1) % wallet.max_mixdepth
+        # Select an unused destination in the external branch
+        self.destination_addr = wallet.get_new_addr(dest_mixdepth, 0)
+        super(P2EPMaker, self).__init__(wallet)
+        # Callback to request user permission (for e.g. GUI)
+        # args: (1) message, as string
+        # returns: True or False
+        self.user_check = self.default_user_check
+        self.user_info = self.default_user_info_callback
+
+    def default_user_check(self, message):
+        if input(message) == 'y':
+            return True
+        return False
+
+    def default_user_info_callback(self, message):
+        """ TODO this is basically the same function
+        as taker_info_callback (currently used for GUI);
+        fold this and some other convenience functions together
+        and use a root CJPeer class in jmbase to avoid code
+        duplication.
+        """
+        jlog.info(message)
+
+    def inform_user_details(self):
+        self.user_info("Your receiving address is: " + self.destination_addr)
+        self.user_info("You will receive amount: " + str(
+            self.receiving_amount) + " satoshis.")
+        self.user_info("The sender also needs to know your ephemeral "
+                  "nickname: " + jm_single().nickname)
+        self.user_info("This information has been stored in a file payjoin.txt;"
+                  " send it to your counterparty when you are ready.")
+        with open("payjoin.txt", "w") as f:
+            f.write("Payjoin transfer details:\n\n")
+            f.write("Address: " + self.destination_addr + "\n")
+            f.write("Amount (in sats): " + str(self.receiving_amount) + "\n")
+            f.write("Receiver nick: " + jm_single().nickname + "\n")
+        if not self.user_check("Enter 'y' to wait for the payment:"):
+            sys.exit(0)
+
+    def create_my_orders(self):
+        """ Fake offer for public consumption.
+        Requests to fill will be ignored.
+        """
+        ordertype = random.choice(("swreloffer", "swabsoffer"))
+        minsize = random.randint(100000, 10000000)
+        maxsize = random.randint(100000, 1000000000) + minsize
+        txfee = random.randint(0, 1000)
+        if ordertype == "swreloffer":
+            cjfee = str(random.randint(0, 100000)/100000000.0)
+        else:
+            cjfee = random.randint(0, 10000)
+        order = {'oid': 0,
+                 'ordertype': ordertype,
+                 'minsize': minsize,
+                 'maxsize': maxsize,
+                 'txfee': txfee,
+                 'cjfee': cjfee}
+
+        # sanity check
+        assert order['minsize'] >= 0
+        assert order['maxsize'] > 0
+        if order['minsize'] > order['maxsize']:
+            jlog.info('minsize (' + str(order['minsize']) + ') > maxsize (' + str(
+                order['maxsize']) + ')')
+            return []
+
+        return [order]
+
+    def oid_to_order(self, offer, amount):
+        # unreachable; only here to satisy abc.
+        pass
+
+    def on_tx_unconfirmed(self, txd, txid):
+        self.user_info("The transaction has been broadcast.")
+        self.user_info("Txid is: " + txid)
+        self.user_info("Transaction in detail: " + pprint.pformat(txd))
+        self.user_info("shutting down.")
+        reactor.stop()
+
+    def on_tx_confirmed(self, offer, confirmations, txid):
+        # will not be reached except in testing
+        self.on_tx_unconfirmed(offer, confirmations)
+
+    def on_tx_received(self, nick, txhex):
+        """ Called when the sender-counterparty has sent a transaction proposal.
+        1. First we check for the expected destination and amount (this is
+           sufficient to identify our cp, as this info was presumably passed
+           out of band, as for any normal payment).
+        2. Then we verify the validity of the proposed non-coinjoin
+           transaction; if not, reject, otherwise store this as a
+           fallback transaction in case the protocol doesn't complete.
+        3. Next, we select utxos from our wallet, to add into the
+           payment transaction as input. Try to select so as to not
+           trigger the UIH2 condition, but continue (and inform user)
+           even if we can't (if we can't select any coins, broadcast the
+           non-coinjoin payment, if the user agrees).
+           Proceeding with payjoin:
+        4. We update the output amount at the destination address.
+        5. We modify the change amount in the original proposal (which
+           will be the only other output other than the destination),
+           reducing it to account for the increased transaction fee
+           caused by our additional proposed input(s).
+        6. Finally we sign our own input utxo(s) and re-serialize the
+           tx, allowing it to be sent back to the counterparty.
+        7. If the transaction is not fully signed and broadcast within
+           the time unconfirm_timeout_sec as specified in the joinmarket.cfg,
+           we broadcast the non-coinjoin fallback tx instead.
+        """
+        try:
+            tx = btc.deserialize(txhex)
+        except (IndexError, SerializationError, SerializationTruncationError) as e:
+            return (False, 'malformed txhex. ' + repr(e))
+        self.user_info('obtained tx proposal from sender:\n' + pprint.pformat(tx))
+
+        if len(tx["outs"]) != 2:
+            return (False, "Transaction has more than 2 outputs; not supported.")
+        dest_found = False
+        destination_index = -1
+        change_index = -1
+        proposed_change_value = 0
+        for index, out in enumerate(tx["outs"]):
+            if out["script"] == btc.address_to_script(self.destination_addr):
+                # we found the expected destination; is the amount correct?
+                if not out["value"] == self.receiving_amount:
+                    return (False, "Wrong payout value in proposal from sender.")
+                dest_found = True
+                destination_index = index
+            else:
+                change_found = True
+                proposed_change_out = out["script"]
+                proposed_change_value = out["value"]
+                change_index = index
+
+        if not dest_found:
+            return (False, "Our expected destination address was not found.")
+
+        # Verify valid input utxos provided and check their value.
+        # batch retrieval of utxo data
+        utxo = {}
+        ctr = 0
+        for index, ins in enumerate(tx['ins']):
+            utxo_for_checking = ins['outpoint']['hash'] + ':' + str(ins[
+                'outpoint']['index'])
+            utxo[ctr] = [index, utxo_for_checking]
+            ctr += 1
+
+        utxo_data = jm_single().bc_interface.query_utxo_set(
+            [x[1] for x in utxo.values()])
+
+        total_sender_input = 0
+        for i, u in iteritems(utxo):
+            if utxo_data[i] is None:
+                return (False, "Proposed transaction contains invalid utxos")
+            total_sender_input += utxo_data[i]["value"]
+
+        # Check that the transaction *as proposed* balances; check that the
+        # included fee is within 0.3-3x our own current estimates, if not user
+        # must decide.
+        btc_fee = total_sender_input - self.receiving_amount - proposed_change_value
+        self.user_info("Network transaction fee is: " + str(btc_fee) + " satoshis.")
+        fee_est = estimate_tx_fee(len(tx['ins']), len(tx['outs']),
+                                  txtype=self.wallet.get_txtype())
+        fee_ok = False
+        if btc_fee > 0.3 * fee_est and btc_fee < 3 * fee_est:
+            fee_ok = True
+        else:
+            if self.user_check("Is this transaction fee acceptable? (y/n):"):
+                fee_ok = True
+        if not fee_ok:
+            return (False,
+                    "Proposed transaction fee not accepted due to tx fee: " + str(
+                            btc_fee))
+
+        # This direct rpc call currently assumes Core 0.17, so not using now.
+        # It has the advantage of (a) being simpler and (b) allowing for any
+        # non standard coins.
+        #
+        #res = jm_single().bc_interface.rpc('testmempoolaccept', [txhex])
+        #print("Got this result from rpc call: ", res)
+        #if not res["accepted"]:
+        #    return (False, "Proposed transaction was rejected from mempool.")
+
+        # Manual verification of the transaction signatures. Passing this
+        # test does imply that the transaction is valid (unless there is
+        # a double spend during the process), but is restricted to standard
+        # types: p2pkh, p2wpkh, p2sh-p2wpkh only. Double spend is not counted
+        # as a risk as this is a payment.
+        for i, u in iteritems(utxo):
+            if "txinwitness" in tx["ins"][u[0]]:
+                ver_amt = utxo_data[i]["value"]
+                try:
+                    ver_sig, ver_pub = tx["ins"][u[0]]["txinwitness"]
+                except Exception as e:
+                    self.user_info("Segwit error: " + repr(e))
+                    return (False, "Segwit input not of expected type, "
+                            "either p2sh-p2wpkh or p2wpkh")
+                # note that the scriptCode is the same whether nested or not
+                # also note that the scriptCode has to be inferred if we are
+                # only given a transaction serialization.
+                scriptCode = "76a914" + btc.hash160(unhexlify(ver_pub)) + "88ac"
+            else:
+                scriptCode = None
+                ver_amt = None
+                scriptSig = btc.deserialize_script(tx["ins"][u[0]]["script"])
+                if len(scriptSig) != 2:
+                    return (False,
+                        "Proposed transaction contains unsupported input type")
+                ver_sig, ver_pub = scriptSig
+            if not btc.verify_tx_input(txhex, u[0],
+                                           utxo_data[i]['script'],
+                                           ver_sig, ver_pub,
+                                           scriptCode=scriptCode,
+                                           amount=ver_amt):
+                return (False, "Proposed transaction is not correctly signed.")
+
+        # At this point we are satisfied with the proposal. Record the fallback
+        # in case the sender disappears and the payjoin tx doesn't happen:
+        self.user_info("We'll use this serialized transaction to broadcast if your"
+                  " counterparty fails to broadcast the payjoin version:")
+        self.user_info(txhex)
+        # Keep a local copy for broadcast fallback:
+        self.fallback_tx = txhex
+
+        # Now we add our own inputs:
+        # See the gist comment here:
+        # https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e#gistcomment-2799709
+        # which sets out the decision Bob must make.
+        # In cases where Bob can add any amount, he selects one utxo
+        # to keep it simple.
+        # In cases where he must choose at least X, he selects one utxo
+        # which provides X if possible, otherwise defaults to a normal
+        # selection algorithm.
+        # In those cases where he must choose X but X is unavailable,
+        # he selects all coins, and proceeds anyway with payjoin, since
+        # it has other advantages (CIOH and utxo defrag).
+        my_utxos = {}
+        largest_out = max(self.receiving_amount, proposed_change_value)
+        max_sender_amt = max([u['value'] for u in utxo_data])
+        not_uih2 = False
+        if max_sender_amt < largest_out:
+            # just select one coin.
+            # have some reasonable lower limit but otherwise choose
+            # randomly; note that this is actually a great way of
+            # sweeping dust ...
+            self.user_info("Choosing one coin at random")
+            try:
+                my_utxos = self.wallet.select_utxos(
+                    self.mixdepth,  jm_single().DUST_THRESHOLD,
+                    select_fn=select_one_utxo)
+            except:
+                return self.no_coins_fallback()
+            not_uih2 = True
+        else:
+            # get an approximate required amount assuming 4 inputs, which is
+            # fairly conservative (but guess by necessity).
+            fee_for_select = estimate_tx_fee(len(tx['ins']) + 4, 2,
+                                             txtype=self.wallet.get_txtype())
+            approx_sum = max_sender_amt - largest_out + fee_for_select
+            try:
+                my_utxos = self.wallet.select_utxos(self.mixdepth, approx_sum)
+                not_uih2 = True
+            except Exception:
+                # TODO probably not logical to always sweep here.
+                self.user_info("Sweeping all coins in this mixdepth.")
+                try:
+                    my_utxos = self.wallet.get_utxos_by_mixdepth()[self.mixdepth]
+                except:
+                    return self.no_coins_fallback()
+        if not_uih2:
+            self.user_info("The proposed tx does not trigger UIH2, which "
+                      "means it is indistinguishable from a normal "
+                      "payment. This is the ideal case. Continuing..")
+        else:
+            self.user_info("The proposed tx does trigger UIH2, which it makes "
+                      "it somewhat distinguishable from a normal payment,"
+                      " but proceeding with payjoin..")
+
+        my_total_in = sum([va['value'] for va in my_utxos.values()])
+        self.user_info("We selected inputs worth: " + str(my_total_in))
+        # adjust the output amount at the destination based on our contribution
+        new_destination_amount = self.receiving_amount + my_total_in
+        # estimate the required fee for the new version of the transaction
+        total_ins = len(tx["ins"]) + len(my_utxos.keys())
+        est_fee = estimate_tx_fee(total_ins, 2, txtype=self.wallet.get_txtype())
+        self.user_info("We estimated a fee of: " + str(est_fee))
+        new_change_amount = total_sender_input + my_total_in - \
+            new_destination_amount - est_fee
+        self.user_info("We calculated a new change amount of: " + str(new_change_amount))
+        self.user_info("We calculated a new destination amount of: " + str(new_destination_amount))
+        # now reconstruct the transaction with the new inputs and the
+        # amount-changed outputs
+        new_outs = [{"address": self.destination_addr,
+                     "value": new_destination_amount}]
+        if new_change_amount >= jm_single().BITCOIN_DUST_THRESHOLD:
+            new_outs.append({"script": proposed_change_out,
+                     "value": new_change_amount})
+        new_ins = [x[1] for x in utxo.values()]
+        new_ins.extend(my_utxos.keys())
+        # set locktime for best anonset (Core, Electrum) - most recent block.
+        # this call should never fail so no catch here.
+        currentblock = jm_single().bc_interface.rpc(
+            "getblockchaininfo", [])["blocks"]
+        new_tx = make_shuffled_tx(new_ins, new_outs, False, 2, currentblock)
+        new_tx_deser = btc.deserialize(new_tx)
+
+        # sign our inputs before transfer
+        our_inputs = {}
+        for index, ins in enumerate(new_tx_deser['ins']):
+            utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index'])
+            if utxo not in my_utxos:
+                continue
+            script = self.wallet.addr_to_script(my_utxos[utxo]['address'])
+            amount = my_utxos[utxo]['value']
+            our_inputs[index] = (script, amount)
+
+        txs = self.wallet.sign_tx(btc.deserialize(new_tx), our_inputs)
+        jm_single().bc_interface.add_tx_notify(txs,
+            self.on_tx_unconfirmed, self.on_tx_confirmed,
+            self.destination_addr,
+            wallet_name=jm_single().bc_interface.get_wallet_name(self.wallet),
+            txid_flag=False, vb=self.wallet._ENGINE.VBYTE)
+        # The blockchain interface just abandons monitoring if the transaction
+        # is not broadcast before the configured timeout; we want to take
+        # action in this case, so we add an additional callback to the reactor:
+        reactor.callLater(jm_single().config.getint("TIMEOUT",
+                            "unconfirm_timeout_sec"), self.broadcast_fallback)
+        return (True, nick, btc.serialize(txs))
+
+    def no_coins_fallback(self):
+        """ Broadcast, optionally, the fallback non-coinjoin transaction
+        because we were not able to select coins to contribute.
+        """
+        self.user_info("Unable to select any coins; this mixdepth is empty.")
+        if self.user_check("Would you like to broadcast the non-coinjoin payment?"):
+            self.broadcast_fallback()
+            return (False, "Coinjoin unsuccessful, fallback attempted.")
+        else:
+            self.user_info("You chose not to broadcast; the payment has NOT been made.")
+            return (False, "No transaction made.")
+
+    def broadcast_fallback(self):
+        self.user_info("Broadcasting non-coinjoin fallback transaction.")
+        txid = btc.txhash(self.fallback_tx)
+        success = jm_single().bc_interface.pushtx(self.fallback_tx)
+        if not success:
+            self.user_info("ERROR: the fallback transaction did not broadcast. "
+                      "The payment has NOT been made.")
+        else:
+            self.user_info("Payment received successfully, but it was NOT a coinjoin.")
+            self.user_info("Txid: " + txid)
+        reactor.stop()

jmclient/jmclient/support.py

@@ -2,7 +2,6 @@ from __future__ import (absolute_import, division,
                         print_function, unicode_literals)
 from builtins import * # noqa: F401
 from functools import reduce
-
 import random
 from jmbase.support import get_log
 from decimal import Decimal
@@ -163,6 +162,10 @@ def select_greediest(unspent, value):
             end += 1
         return low[0:end]
 
+def select_one_utxo(unspent, value):
+    key = lambda u: u['value']
+    return [random.choice([u for u in unspent if key(u) >= value])]
+
 
 def calc_cj_fee(ordertype, cjfee, cj_amount):
     if ordertype in ['swabsoffer', 'absoffer']:

jmclient/jmclient/taker.py

@@ -7,14 +7,16 @@ from future.utils import iteritems
 import base64
 import pprint
 import random
+from twisted.internet import reactor
 from binascii import hexlify, unhexlify
 
+from jmbitcoin import SerializationError, SerializationTruncationError
 import jmbitcoin as btc
 from jmclient.configure import get_p2sh_vbyte, jm_single, validate_address
 from jmbase.support import get_log
 from jmclient.support import (calc_cj_fee, weighted_order_choose, choose_orders,
                               choose_sweep_orders)
-from jmclient.wallet import estimate_tx_fee
+from jmclient.wallet import estimate_tx_fee, make_shuffled_tx
 from jmclient.podle import generate_podle, get_podle_commitments, PoDLE
 from .output import generate_podle_error_string
 from .cryptoengine import EngineError
@@ -491,9 +493,7 @@ class Taker(object):
                         for u in sum(self.utxos.values(), [])]
         self.outputs.append({'address': self.coinjoin_address(),
                              'value': self.cjamount})
-        random.shuffle(self.utxo_tx)
+        tx = make_shuffled_tx(self.utxo_tx, self.outputs, False)
-        random.shuffle(self.outputs)
-        tx = btc.mktx(self.utxo_tx, self.outputs)
         jlog.info('obtained tx\n' + pprint.pformat(btc.deserialize(tx)))
 
         self.latest_tx = btc.deserialize(tx)
@@ -842,3 +842,344 @@ class Taker(object):
             return
         assert hasattr(bci, 'get_wallet_name')
         bci.import_addresses(addr_list, bci.get_wallet_name(self.wallet))
+
+class P2EPTaker(Taker):
+    """ The P2EP Taker will initialize its protocol directly
+    with the prescribed counterparty (see -T argument to
+    sendpayment). It inherits the normal behaviour of requesting
+    an orderbook on startup, but does nothing with it; this
+    improves the privacy of the operation.
+    """
+
+    def __init__(self, counterparty, wallet, schedule, callbacks):
+        super(P2EPTaker, self).__init__(wallet, schedule, callbacks=callbacks)
+        self.p2ep_receiver_nick = counterparty
+        # Callback to request user permission (for e.g. GUI)
+        # args: (1) message, as string
+        # returns: True or False
+        self.user_check = self.default_user_check
+
+    def default_user_check(self, message):
+        if input(message) == 'y':
+            return True
+        return False
+
+    def register_user_check_callback(self, user_check):
+        self.user_check = user_check
+
+    def unconfirm_callback(self, txd, txid):
+        jlog.info("Transaction seen on network, shutting down.")
+        jlog.info("Txid was: " + txid)
+        # In P2EP we stop the protocol here.
+        reactor.stop()
+
+    def confirm_callback(self, txd, txid, confirmations):
+        # Will never trigger except in testing
+        self.unconfirm_callback(txd, txid)
+
+    def initialize(self, orderbook):
+        """ Note that the orderbook parameter is ignored.
+        Here the schedule data (the standard format for coinjoin
+        request specification) passes in the amount, destination
+        and source mixdepth information. We then select coins
+        using the inherited Taker method to do so.
+        """
+        if self.aborted:
+            return (False,)
+        self.taker_info_callback("INFO", "Received offers from joinmarket pit")
+        # only one schedule item has been allowed; parse from it.
+        self.schedule_index += 1
+        si = self.schedule[0]
+        self.mixdepth = si[0]
+        self.cjamount = si[1]
+        # For the p2ep taker, the variable 'my_cj_addr' is the destination:
+        self.my_cj_addr = si[3]
+        if isinstance(self.cjamount, float):
+            raise JMTakerError("P2EP coinjoin must use amount in satoshis")
+        if self.cjamount == 0:
+            # Note that we don't allow sweep, currently, since the coin
+            # choosing algo would not apply in that case (we'd have to rewrite
+            # prepare_my_bitcoin_data for that case).
+            raise JMTakerError("P2EP coinjoin does not currently support sweep")
+
+        # Next we prepare our coins with the inherited method
+        # for this purpose; for this we must set the
+        # number of counterparties and fee per cpy, for fee estimation;
+        # the estimates will be rather rough, but that's fine. Here we
+        # will end up selecting 20k sats more than the destination amount,
+        # which will make the already ultra-rare edge case of not selecting
+        # enough for fees, even more rare. "Stuck" coins due to edge cases
+        # are not an issue since the wallet has direct-send sweep.
+        self.n_counterparties = 1
+        self.txfee_default = 10000
+        self.total_cj_fee = 0
+        # Preparing bitcoin data here includes choosing utxos/coins.
+        # We don't trust the user on the selection algo choice; we want it
+        # to be fairly greedy for technical reasons explained in the comment
+        # thread to this gist:
+        # https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e
+        jm_single().config.set("POLICY", "merge_algorithm", "greedy")
+        self.noncj_fee_est = 0
+        if not self.prepare_my_bitcoin_data():
+            return (False, )
+        self.outputs = []
+        self.cjfee_total = 0
+        self.txfee_default = 5000
+        self.latest_tx = None
+        self.txid = None
+
+        # we return dummy values for commitment and revelation,
+        # and the offer dict only signals the nick of the counterparty.
+        return (True, self.cjamount, "p2ep", "p2ep", {self.p2ep_receiver_nick:{}})
+
+    def receive_utxos(self, ioauth_data):
+        """ TODO this function is misnamed for the
+        purpose of code reuse; fix it(e.g. 'make_tx_proposal').
+
+        The ioauth_data field will be a list containing the single
+        nick which we intend to send to (which fact we should check),
+        and then construct a transaction with the intended amount
+        to the intended destination. This transaction will not, in
+        normal operation, be broadcast; because the Maker (receiver)
+        will add his own inputs and change the total to be received at
+        that address. Thus we are functionally only sending our own input
+        utxos - the signed tx may be used as a fallback by the counterparty
+        in case we disappear - however this also serves the purpose of
+        signalling that we are the right counterparty.
+        """
+        if not ioauth_data[0] == self.p2ep_receiver_nick:
+            return (False, "Wrong counterparty IRC nick: " + ioauth_data[0])
+        # Transaction construction: use inputs as per `prepare_my_bitcoin_data`,
+        # use output destination self.my_cj_addr and use amount self.amount
+        self.outputs.append({'address': self.my_cj_addr,
+                                     'value': self.cjamount})
+        my_total_in = sum([va['value'] for u, va in iteritems(self.input_utxos)])
+        # estimate the fee for the version of the transaction which is
+        # not coinjoined:
+        est_fee = estimate_tx_fee(len(self.input_utxos), 2,
+                                  txtype=self.wallet.get_txtype())
+        my_change_value = my_total_in - self.cjamount - est_fee
+        if my_change_value <= 0:
+            # as discussed in initialize(), this should be an extreme edge case.
+            raise ValueError("Wallet utxo selection chose too few coins")
+        elif self.my_change_addr and my_change_value <= jm_single(
+            ).BITCOIN_DUST_THRESHOLD:
+            jlog.info("Dynamically calculated change lower than dust: " + str(
+                my_change_value) + "; dropping.")
+            self.my_change_addr = None
+            my_change_value = 0
+        # Note that the sweep case (my_change_addr is None, but not due to dust)
+        # is not currently allowed here.
+        if self.my_change_addr is not None:
+            self.outputs.append({'address': self.my_change_addr,
+                                 'value': my_change_value})
+        # set locktime for best anonset (Core, Electrum) - most recent block.
+        # this call should never fail so no catch here.
+        currentblock = jm_single().bc_interface.rpc(
+            "getblockchaininfo", [])["blocks"]
+        # As for JM coinjoins, the `None` key is used for our own inputs
+        # to the transaction; this preparatory version contains only those.
+        tx = make_shuffled_tx(self.utxos[None], self.outputs,
+                              False, 2, currentblock)
+        jlog.info('Obtained proposed payjoin tx\n' + pprint.pformat(
+            btc.deserialize(tx)))
+        # We now sign as a courtesy, because if we disappear the recipient
+        # can still claim his coins with this.
+        # sign our inputs before transfer
+        our_inputs = {}
+        dtx = btc.deserialize(tx)
+        for index, ins in enumerate(dtx['ins']):
+            utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index'])
+            script = self.wallet.addr_to_script(self.input_utxos[utxo]['address'])
+            amount = self.input_utxos[utxo]['value']
+            our_inputs[index] = (script, amount)
+        self.signed_noncj_tx = btc.serialize(self.wallet.sign_tx(dtx, our_inputs))
+        self.taker_info_callback("INFO", "Built tx proposal, sending to receiver.")
+        return (True, [self.p2ep_receiver_nick], self.signed_noncj_tx)
+
+    def on_tx_received(self, nick, txhex):
+        """ Here the taker (payer) retrieves a version of the
+        transaction from the maker (receiver) which should have
+        the following properties:
+        * Destination address as previously agreed.
+        * Our correct change output with amount corresponding to fee.
+        * Net of (destination amount) - (receiver input amount)
+          must be equal to the original amount self.cjamount.
+        * Our inputs must be unchanged from original proposal.
+        * Counterparty should not provide more than 5 utxos; this
+          is a crude avoidance of over-paying fees, but note that
+          the maker selection should mean this almost never happens.
+        * Counterparties' transaction signatures must be valid.
+        If all conditions are met, we sign each of our inputs
+        and then broadcast (TODO broadcast delay or don't broadcast).
+        """
+        try:
+            tx = btc.deserialize(txhex)
+        except (IndexError, SerializationError, SerializationTruncationError) as e:
+            return (False, "malformed txhex. " + repr(e))
+        jlog.info("Obtained tx from receiver:\n" + pprint.pformat(tx))
+        cjaddr_script = btc.address_to_script(self.my_cj_addr)
+        changeaddr_script = btc.address_to_script(self.my_change_addr)
+
+        # We ensure that the coinjoin address and our expected change
+        # address are still in the outputs, once (with the caveat that
+        # the change address is allowed to be absent in a special case
+        # of dust change, which we assess after).
+        times_seen_cj_addr = 0
+        times_seen_change_addr = 0
+        for outs in tx['outs']:
+            if outs['script'] == cjaddr_script:
+                times_seen_cj_addr += 1
+                new_cj_amount = outs['value']
+                if new_cj_amount < self.cjamount:
+                    # This is a violation of protocol;
+                    # receiver must be providing extra bitcoin
+                    # as input, so his receiving amount should have increased.
+                    return (False,
+                    'Wrong cj_amount. I expect at least' + str(self.cjamount))
+            if outs['script'] == changeaddr_script:
+                times_seen_change_addr += 1
+                new_change_amount = outs['value']
+        if times_seen_cj_addr != 1:
+            fmt = ('cj addr not in tx outputs once, #cjaddr={}').format
+            return (False, (fmt(times_seen_cj_addr)))
+        if times_seen_change_addr != 1:
+            if times_seen_change_addr > 1:
+                return (False, "proposed tx has change address duplicated")
+            # Otherwise change has been ditched; will check this later.
+            new_change_amount = 0
+
+        # Check that our inputs are present.
+        tx_utxo_set = set(ins['outpoint']['hash'] + ':' + str(
+            ins['outpoint']['index']) for ins in tx['ins'])
+        if not tx_utxo_set.issuperset(set(self.utxos[None])):
+            return (False, "my utxos are not contained")
+
+        # Before even starting fee calculations, reject  > 5
+        # inputs from counterparty as an abuse (accidental or
+        # not) of PayJoin to sweep utxos at no cost.
+        # (TODO This is very kludgy, more sophisticated approach
+        # should be used in future):
+        if len(tx["ins"]) - len (self.utxos[None]) > 5:
+            return (False,
+                    "proposed tx has more than 5 inputs from "
+                    "the recipient, which is too expensive.")
+
+        # If we ignored fees, we would only need to check that
+        # the difference between our inputs and outputs was equal
+        # to the expected payment; but this difference will include
+        # the bitcoin transaction fee.
+        # Hence, we retrieve the counterparty's input amount,
+        # and find the overall bitcoin network fee, and decide
+        # from this whether the change value is as expected
+        # (our inputs - expected payment - network fee);
+        # and if that is dusty, agree to sign without change.
+
+        # batch retrieval of utxo data; we collect the utxos
+        # in the inputs which do not belong to us, and put
+        # them into a dict (`retrieve_utxos`), keyed by their
+        # index in the inputs, so we can use the collected
+        # script and amount data when we do the next stages,
+        # checking input validity and transaction balance.
+        retrieve_utxos = {}
+        ctr = 0
+        for index, ins in enumerate(tx['ins']):
+            utxo_for_checking = ins['outpoint']['hash'] + ':' + str(
+                ins['outpoint']['index'])
+            if utxo_for_checking in self.utxos[None]:
+                continue
+            retrieve_utxos[ctr] = [index, utxo_for_checking]
+            ctr += 1
+        # we always accept unconf utxos from receiver; it's their payment:
+        utxo_data = jm_single().bc_interface.query_utxo_set(
+            [x[1] for x in retrieve_utxos.values()], includeunconf=True)
+
+        # Next we'll verify each of the counterparty's inputs,
+        # while at the same time gathering the total they spent.
+        total_receiver_input = 0
+        for i, u in iteritems(retrieve_utxos):
+            if utxo_data[i] is None:
+                return (False, "Proposed transaction contains invalid utxos")
+            total_receiver_input += utxo_data[i]["value"]
+            scriptCode = None
+            ver_amt = None
+            idx = retrieve_utxos[i][0]
+            if "txinwitness" in tx["ins"][idx]:
+                ver_amt = utxo_data[i]["value"]
+                try:
+                    ver_sig, ver_pub = tx["ins"][idx]["txinwitness"]
+                except Exception as e:
+                    print("Segwit error: ", repr(e))
+                    return (False, "Segwit input not of expected type, "
+                            "either p2sh-p2wpkh or p2wpkh")
+                # note that the scriptCode is the same whether nested or not
+                # also note that the scriptCode has to be inferred if we are
+                # only given a transaction serialization.
+                scriptCode = "76a914" + btc.hash160(unhexlify(ver_pub)) + "88ac"
+            else:
+                scriptSig = btc.deserialize_script(tx["ins"][idx]["script"])
+                if len(scriptSig) != 2:
+                    return (False,
+                    "Proposed transaction contains unsupported input type")
+                ver_sig, ver_pub = scriptSig
+            if not btc.verify_tx_input(txhex, idx,
+                                       utxo_data[i]['script'],
+                                       ver_sig, ver_pub,
+                                       scriptCode=scriptCode,
+                                       amount=ver_amt):
+                return (False,
+                        "Proposed transaction is not correctly signed.")
+        payment = new_cj_amount - total_receiver_input
+        if payment != self.cjamount:
+            return (False, "Proposed transaction has wrong payment amount: " +\
+                    str(payment) + ", should be: " + str(self.cjamount))
+        # reminder: the keys of the input_utxos dict == self.utxos[None]
+        total_sender_input =  sum([va['value'] for va in self.input_utxos.values()])
+        # check full transaction balance
+        btc_fee = total_receiver_input + total_sender_input - new_cj_amount - new_change_amount
+        self.taker_info_callback("INFO",
+                                 "Network transaction fee is: " + str(btc_fee) + " satoshis.")
+        if btc_fee <= 0:
+            return (False, "Proposed transaction has no bitcoin fee")
+        # To validate the fee, we need to check the size, but this can only be estimated
+        # until it's fully signed; we now know the number of inputs, so we can use
+        # our fee estimator. Its return value will be governed by our own fee settings
+        # in joinmarket.cfg; allow either (a) automatic agreement for any value within
+        # a range of 0.3 to 3x this figure, or (b) user to agree on prompt.
+        fee_est = estimate_tx_fee(len(tx['ins']), len(tx['outs']),
+                                  txtype=self.wallet.get_txtype())
+        fee_ok = False
+        if btc_fee > 0.3 * fee_est and btc_fee < 3 * fee_est:
+            fee_ok = True
+        else:
+            if self.user_check("Is this transaction fee acceptable? (y/n):"):
+                fee_ok = True
+        if not fee_ok:
+            return (False,
+                    "Proposed transaction fee not accepted due to tx fee: " + str(
+                        btc_fee))
+
+        self.total_txfee = btc_fee
+        # now that the fee is known and accepted, we can check our change
+        if new_change_amount == 0:
+            # calculate what the change would be, after subtracting the agreed fee;
+            # if it's dusty, then we continue with no change; otherwise we prompt/
+            # reject.
+            hyp_change_amount = total_receiver_input - self.cjamount - btc_fee
+            if hyp_change_amount <= jm_single().BITCOIN_DUST_THRESHOLD:
+                jlog.info("Counterparty correctly removed dusty change value:"\
+                          + str(hyp_change_amount))
+            else:
+                jlog.info(('WARNING CHANGE NOT BEING '
+                                       'USED\nCHANGEVALUE = {}').format(
+                                           hyp_change_amount))
+                if not self.user_check("OK to broadcast with this change spent "
+                         "to miner fee? (y/n):"):
+                    return (False, "Proposed transaction not accepted due to "
+                            "absent change.")
+
+        # All checks have passed; we sign and broadcast
+        self.latest_tx = tx
+        self.self_sign_and_push()
+        # returning False here is not an error condition, only stops processing.
+        return (False, "OK")

jmclient/jmclient/taker_utils.py

@@ -7,7 +7,6 @@ import pprint
 import os
 import time
 import numbers
-from binascii import unhexlify
 from jmbase import get_log, jmprint
 from .configure import jm_single, validate_address
 from .schedule import human_readable_schedule_entry, tweak_tumble_schedule,\
@@ -113,7 +112,7 @@ def sign_tx(wallet, tx, utxos):
         script = wallet.addr_to_script(utxos[utxo]['address'])
         amount = utxos[utxo]['value']
         our_inputs[index] = (script, amount)
-    return wallet.sign_tx(deserialize(unhexlify(serialize(stx))), our_inputs)
+    return wallet.sign_tx(stx, our_inputs)
 
 def import_new_addresses(wallet, addr_list):
     # FIXME: same code as in maker.py and taker.py

jmclient/jmclient/wallet.py

@@ -4,6 +4,7 @@ from builtins import * # noqa: F401
 
 from configparser import NoOptionError
 import warnings
+import random
 import functools
 import collections
 import numbers
@@ -66,6 +67,18 @@ class Mnemonic(MnemonicParent):
     def detect_language(cls, code):
         return "english"
 
+def make_shuffled_tx(ins, outs, deser=True, version=1, locktime=0):
+    """ Simple utility to ensure transaction
+    inputs and outputs are randomly ordered.
+    Can possibly be replaced by BIP69 in future
+    """
+    random.shuffle(ins)
+    random.shuffle(outs)
+    tx = btc.mktx(ins, outs, version=version, locktime=locktime)
+    if deser:
+        return btc.deserialize(tx)
+    else:
+        return tx
 
 def estimate_tx_fee(ins, outs, txtype='p2pkh'):
     '''Returns an estimate of the number of satoshis required
@@ -82,9 +95,9 @@ def estimate_tx_fee(ins, outs, txtype='p2pkh'):
     if txtype in ['p2pkh', 'p2shMofN']:
         tx_estimated_bytes = btc.estimate_tx_size(ins, outs, txtype)
         return int((tx_estimated_bytes * fee_per_kb)/Decimal(1000.0))
-    elif txtype=='p2sh-p2wpkh':
+    elif txtype in ['p2wpkh', 'p2sh-p2wpkh']:
         witness_estimate, non_witness_estimate = btc.estimate_tx_size(
-            ins, outs, 'p2sh-p2wpkh')
+            ins, outs, txtype)
         return int(int((
         non_witness_estimate + 0.25*witness_estimate)*fee_per_kb)/Decimal(1000.0))
     else:
@@ -168,12 +181,13 @@ class UTXOManager(object):
 
         self._utxo[mixdepth][(txid, index)] = (path, value)
 
-    def select_utxos(self, mixdepth, amount, utxo_filter=()):
+    def select_utxos(self, mixdepth, amount, utxo_filter=(), select_fn=None):
         assert isinstance(mixdepth, numbers.Integral)
         utxos = self._utxo[mixdepth]
         available = [{'utxo': utxo, 'value': val}
             for utxo, (addr, val) in utxos.items() if utxo not in utxo_filter]
-        selected = self.selector(available, amount)
+        selector = select_fn or self.selector
+        selected = selector(available, amount)
         return {s['utxo']: {'path': utxos[s['utxo']][0],
                             'value': utxos[s['utxo']][1]}
                 for s in selected}
@@ -315,6 +329,8 @@ class BaseWallet(object):
             return 'p2pkh'
         elif self.TYPE == TYPE_P2SH_P2WPKH:
             return 'p2sh-p2wpkh'
+        elif self.TYPE == TYPE_P2WPKH:
+            return 'p2wpkh'
         assert False
 
     def sign_tx(self, tx, scripts, **kwargs):
@@ -549,12 +565,12 @@ class BaseWallet(object):
         self._utxos.add_utxo(txid, index, path, value, mixdepth)
 
     @deprecated
-    def select_utxos(self, mixdepth, amount, utxo_filter=None):
+    def select_utxos(self, mixdepth, amount, utxo_filter=None, select_fn=None):
         utxo_filter_new = None
         if utxo_filter:
             utxo_filter_new = [(unhexlify(utxo[:64]), int(utxo[65:]))
                                for utxo in utxo_filter]
-        ret = self.select_utxos_(mixdepth, amount, utxo_filter_new)
+        ret = self.select_utxos_(mixdepth, amount, utxo_filter_new, select_fn)
         ret_conv = {}
         for utxo, data in ret.items():
             addr = self.get_addr_path(data['path'])
@@ -562,7 +578,8 @@ class BaseWallet(object):
             ret_conv[utxo_txt] = {'address': addr, 'value': data['value']}
         return ret_conv
 
-    def select_utxos_(self, mixdepth, amount, utxo_filter=None):
+    def select_utxos_(self, mixdepth, amount, utxo_filter=None,
+                      select_fn=None):
         """
         Select a subset of available UTXOS for a given mixdepth whose value is
         greater or equal to amount.
@@ -585,7 +602,8 @@ class BaseWallet(object):
             assert len(i) == 2
             assert isinstance(i[0], bytes)
             assert isinstance(i[1], numbers.Integral)
-        ret = self._utxos.select_utxos(mixdepth, amount, utxo_filter)
+        ret = self._utxos.select_utxos(
+            mixdepth, amount, utxo_filter, select_fn)
 
         for data in ret.values():
             data['script'] = self.get_script_path(data['path'])

jmclient/jmclient/wallet_utils.py

@@ -14,10 +14,10 @@ from collections import Counter
 from itertools import islice
 from jmclient import (get_network, WALLET_IMPLEMENTATIONS, Storage, podle,
     jm_single, BitcoinCoreInterface, JsonRpcError, sync_wallet, WalletError,
-    VolatileStorage, StoragePasswordError,
+    VolatileStorage, StoragePasswordError, is_segwit_mode, SegwitLegacyWallet,
-    is_segwit_mode, SegwitLegacyWallet, LegacyWallet)
+    LegacyWallet, SegwitWallet, is_native_segwit_mode)
 from jmbase.support import get_password, jmprint
-from .cryptoengine import TYPE_P2PKH, TYPE_P2SH_P2WPKH
+from .cryptoengine import TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH
 import jmbitcoin as btc
 
 
@@ -897,6 +897,8 @@ def wallet_signmessage(wallet, hdpath, message):
 
 def get_wallet_type():
     if is_segwit_mode():
+        if is_native_segwit_mode():
+            return TYPE_P2WPKH
         return TYPE_P2SH_P2WPKH
     return TYPE_P2PKH
 
@@ -937,6 +939,10 @@ def open_test_wallet_maybe(path, seed, max_mixdepth,
     returns:
         wallet object
     """
+    # If the native flag is set in the config, it overrides the argument
+    # test_wallet_cls
+    if jm_single().config.get("POLICY", "native") == "true":
+        test_wallet_cls = SegwitWallet
     if len(seed) == test_wallet_cls.ENTROPY_BYTES * 2:
         try:
             seed = binascii.unhexlify(seed)

jmclient/test/test_payjoin.py

@@ -0,0 +1,163 @@
+from __future__ import (absolute_import, division,
+                        print_function, unicode_literals)
+from builtins import * # noqa: F401
+
+"""
+Test doing payjoin joins (with message channel layer mocked)
+"""
+
+import os
+import sys
+import pytest
+from twisted.internet import reactor
+from jmbase import get_log
+from jmclient import cryptoengine
+from jmclient import (load_program_config, jm_single, sync_wallet,
+                      P2EPMaker, P2EPTaker,
+                      LegacyWallet, SegwitLegacyWallet, SegwitWallet)
+from commontest import make_wallets
+from test_coinjoin import make_wallets_to_list, sync_wallets, create_orderbook
+
+testdir = os.path.dirname(os.path.realpath(__file__))
+log = get_log()
+
+def create_taker(wallet, schedule, monkeypatch):
+    taker = P2EPTaker("fakemaker", wallet, schedule,
+                      callbacks=(None, None, None))
+    return taker
+
+def dummy_user_check(message):
+    # No user interaction in test; just print message
+    # and assume acceptance.
+    log.info(message)
+    return True
+
+def getbals(wallet, mixdepth):
+    """ Retrieves balances for a mixdepth and the 'next'
+    """
+    bbm = wallet.get_balance_by_mixdepth()
+    return (bbm[mixdepth], bbm[(mixdepth + 1) % (wallet.mixdepth + 1)])
+
+def final_checks(wallets, amount, txfee, tsb, msb, source_mixdepth=0):
+    """We use this to check that the wallet contents are
+    as we've expected according to the test case.
+    amount is the payment amount going from taker to maker.
+    txfee is the bitcoin network transaction fee, paid by the
+    taker.
+    tsb, msb are taker and maker starting balances, each a tuple
+    of two entries, source and destination mixdepth respectively.
+    """
+    jm_single().bc_interface.tickchain()
+    for wallet in wallets:
+        sync_wallet(wallet)
+    takerbals = getbals(wallets[1], source_mixdepth)
+    makerbals = getbals(wallets[0], source_mixdepth)
+    # is the payment received?
+    maker_newcoin_amt = makerbals[1] - msb[1]
+    if not maker_newcoin_amt >= amount:
+        print("Maker expected to receive at least: ", amount,
+              " but got: ", maker_newcoin_amt)
+        return False
+    # assert that the maker received net exactly the right amount
+    maker_spentcoin_amt = msb[0] - makerbals[0]
+    if not maker_spentcoin_amt >= 0:
+        # for now allow the non-cj fallback case
+        print("maker's spent coin should have been positive, was: ", maker_spentcoin_amt)
+        return False
+    if not maker_newcoin_amt == amount + maker_spentcoin_amt:
+        print("maker's new coins should have been: ", amount + maker_spentcoin_amt,
+              " but was: ", maker_newcoin_amt)
+        return False
+
+    # Taker-side check
+    # assert that the taker's total ending minus total starting
+    # balance is the amount plus the txfee given.
+    if not (sum(takerbals) - sum(tsb) + txfee + amount) == 0:
+        print("Taker should have spent: ", txfee + amount,
+              " but spent: ", sum(tsb) - sum(takerbals))
+        return False
+    return True
+
+@pytest.mark.parametrize('wallet_cls, wallet_structures, mean_amt',
+        [([LegacyWallet, LegacyWallet],
+          [[4, 0, 0, 0, 0]] * 2, 1.0),
+         ([SegwitLegacyWallet, SegwitLegacyWallet],
+          [[1, 3, 0, 0, 0]] * 2, 2.0),
+         ([SegwitWallet, SegwitWallet],
+          [[1, 0, 0, 0, 0]] * 2, 4.0),
+         ([LegacyWallet, SegwitWallet],
+          [[4, 0, 0, 0, 0]] * 2, 1.0),
+         ([SegwitLegacyWallet, SegwitWallet],
+          [[1, 3, 0, 0, 0]] * 2, 2.0),
+         ([SegwitWallet, SegwitLegacyWallet],
+          [[1, 0, 0, 0, 0]] * 2, 4.0),
+         ])
+def test_simple_payjoin(monkeypatch, tmpdir, setup_cj, wallet_cls,
+                        wallet_structures, mean_amt):
+    def raise_exit(i):
+        raise Exception("sys.exit called")
+    monkeypatch.setattr(sys, 'exit', raise_exit)
+    wallets = []
+    wallets.append(make_wallets_to_list(make_wallets(
+        1, wallet_structures=[wallet_structures[0]],
+        mean_amt=mean_amt, wallet_cls=wallet_cls[0]))[0])
+    wallets.append(make_wallets_to_list(make_wallets(
+            1, wallet_structures=[wallet_structures[1]],
+            mean_amt=mean_amt, wallet_cls=wallet_cls[1]))[0])
+    jm_single().bc_interface.tickchain()
+    sync_wallets(wallets)
+
+    # For accounting purposes, record the balances
+    # at the start.
+    msb = getbals(wallets[0], 0)
+    tsb = getbals(wallets[1], 0)
+
+    cj_amount = int(1.1 * 10**8)
+    maker = P2EPMaker(wallets[0], 0, cj_amount)
+    destaddr = maker.destination_addr
+    monkeypatch.setattr(maker, 'user_check', dummy_user_check)
+    # TODO use this to sanity check behaviour
+    # in presence of the rest of the joinmarket orderbook.
+    orderbook = create_orderbook([maker])
+    assert len(orderbook) == 1
+    # mixdepth, amount, counterparties, dest_addr, waittime;
+    # in payjoin we only pay attention to the first two entries.
+    schedule = [(0, cj_amount, 1, destaddr, 0)]
+    taker = create_taker(wallets[-1], schedule, monkeypatch)
+    monkeypatch.setattr(taker, 'user_check', dummy_user_check)
+    init_data = taker.initialize(orderbook)
+    # the P2EPTaker.initialize() returns:
+    # (True, self.cjamount, "p2ep", "p2ep", {self.p2ep_receiver_nick:{}})
+    assert init_data[0], "taker.initialize error"
+    active_orders = init_data[4]
+    assert len(active_orders.keys()) == 1
+    response = taker.receive_utxos(list(active_orders.keys()))
+    assert response[0], "taker receive_utxos error"
+    # test for validity of signed fallback transaction; requires 0.17;
+    # note that we count this as an implicit test of fallback mode.
+    res = jm_single().bc_interface.rpc('testmempoolaccept', [[response[2]]])
+    assert res[0]["allowed"], "Proposed transaction was rejected from mempool."
+    maker_response = maker.on_tx_received("faketaker", response[2])
+    if not maker_response[0]:
+        print("maker on_tx_received failed, reason: ", maker_response[1])
+        assert False
+    taker_response = taker.on_tx_received("fakemaker", maker_response[2])
+    if not taker_response[1] == "OK":
+        print("Failure in taker on_tx_received, reason: ", taker_response[1])
+        assert False
+    # Although the above OK is proof that a transaction went through,
+    # it doesn't prove it was a good transaction! Here do balance checks:
+    assert final_checks(wallets, cj_amount, taker.total_txfee, tsb, msb)
+
+@pytest.fixture(scope='module')
+def setup_cj():
+    load_program_config()
+    jm_single().config.set('POLICY', 'tx_broadcast', 'self')
+    jm_single().bc_interface.tick_forward_chain_interval = 5
+    jm_single().bc_interface.simulate_blocks()
+    #see note in cryptoengine.py:
+    cryptoengine.BTC_P2WPKH.VBYTE = 100
+    yield None
+    # teardown
+    for dc in reactor.getDelayedCalls():
+        dc.cancel()

jmdaemon/jmdaemon/__init__.py

@@ -12,7 +12,7 @@ from .message_channel import MessageChannel, MessageChannelCollection
 from .orderbookwatch import OrderbookWatch
 from jmbase import commands
 from .daemon_protocol import (JMDaemonServerProtocolFactory, JMDaemonServerProtocol,
-                              start_daemon)
+                              start_daemon, P2EPDaemonServerProtocolFactory)
 from .protocol import (COMMAND_PREFIX, ORDER_KEYS, NICK_HASH_LENGTH,
                        NICK_MAX_ENCODED, JM_VERSION, JOINMARKET_NICK_HEADER)
 from .message_channel import MessageChannelCollection

jmdaemon/jmdaemon/daemon_protocol.py

@@ -647,6 +647,153 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
         if self.mcc:
             self.mcc.shutdown()
 
+class P2EPDaemonServerProtocol(JMDaemonServerProtocol):
+    @JMFill.responder
+    def on_JM_FILL(self, amount, commitment, revelation, filled_offers):
+        """ Takes the setup information for the transaction
+        from the Taker (p2ep sender) and starts the key negotiation,
+        which is handled entirely in the daemon.
+        """
+        if not isinstance(amount, Integral) and amount >= 0:
+            return {'accepted': False}
+        self.amount = amount
+        # The counterparty was passed within a offers dict struct;
+        # we retain the variable `active_orders` for compatibility
+        # with inherited code, even though it's not really an
+        # "order" or "offer", and there's only one.
+        self.active_orders = json.loads(filled_offers)
+        if len(self.active_orders.keys()) != 1:
+            return {'accepted': False}
+        # Here the p2ep communication starts with sending an ephemeral
+        # pubkey. The version data is appended.
+        pubkeymsg = self.versioned_pubkey_msg(self.kp.hex_pk().decode('ascii'))
+        self.mcc.prepare_privmsg(list(self.active_orders.keys())[0],
+                                 "pubkey", pubkeymsg)
+        return {'accepted': True}
+
+    def on_pubkey(self, nick, tmaker_pk):
+        """This is handled locally in the daemon; set up e2e
+        encrypted messaging with this counterparty.
+        Note that this message is used in both directions in p2ep
+        (as distinct from coinjoins where it's sent Maker->Taker
+        only).
+        The pubkey passed in hex and appended with two bytes;
+        for the proposer (sender) this is the minimum and maximum
+        of the p2ep version supported. For the receiver these two
+        bytes must be equal and are equal to the version chosen.
+        """
+        if len(tmaker_pk) < 6:
+            self.mcc.send_error(nick, "Invalid pubkey message: " + tmaker_pk)
+            return
+        try:
+            version_low, version_high = [int(x, 16) for x in [tmaker_pk[-4:-2],
+                                                          tmaker_pk[-2:]]]
+        except ValueError:
+            self.mcc.send_error(nick, "Invalid pubkey message: " + tmaker_pk)
+            return
+        # For both sender and receiver, the received version must be low 0
+        if version_low != 0:
+            self.mcc.send_error(nick,
+                            "Unsupported p2ep version: " + str(version_low))
+            return
+        tmaker_pk = tmaker_pk[:-4]
+        if self.role == "TAKER":
+            # For sender, when he receives pubkey message from receiver, it
+            # must be a choice of version 00 i.e. low, high = 0,0
+            if version_high != 0:
+                self.mcc.send_error(nick,
+                "Invalid p2ep version string, high: " + str(version_high))
+                return
+            maker_pk = tmaker_pk
+            if nick not in self.active_orders.keys():
+                log.msg("Counterparty not part of this transaction. Ignoring")
+                return
+            try:
+                self.crypto_boxes[nick] = [maker_pk, as_init_encryption(
+                    self.kp, init_pubkey(maker_pk))]
+            except NaclError as e:
+                print("Unable to setup crypto box with " + nick + ": " + repr(e))
+                self.mcc.send_error(nick, "invalid nacl pubkey: " + maker_pk)
+                return
+            d = self.callRemote(JMFillResponse, success=True,
+                    ioauth_data= json.dumps(list(self.active_orders.keys())))
+            self.defaultCallbacks(d)
+        elif self.role == "MAKER":
+            # Receiver only cares that version 0 (low) is accepted/in range,
+            # doesn't check high version value.
+            taker_pk = tmaker_pk
+            # Note that the Maker may receive this message from anyone,
+            # not only the sender he communicated with out of band;
+            # for this reason, the protocol only provides an ephemeral
+            # pubkey at this stage; the maker only sends private data,
+            # encrypted, when the Taker, in the next message (!tx), sends
+            # data demonstrating knowledge of the out-of-band info -
+            # specifically, the receiving address.
+
+            #prepare a pubkey for this valid transaction
+            kp = init_keypair()
+            try:
+                crypto_box = as_init_encryption(kp, init_pubkey(taker_pk))
+            except NaclError as e:
+                log.msg("Unable to set up cryptobox with counterparty: " + repr(e))
+                self.mcc.send_error(nick, "Invalid nacl pubkey: " + taker_pk)
+                return
+            #Note this sets the *whole* dict, old entries (e.g. changeaddr)
+            #are removed, so we can't have a conflict between old and new
+            #versions of active_orders[nick]
+            self.active_orders[nick] = {"crypto_box": crypto_box,
+                                        "kp": kp,
+                                        "offer": None,
+                                        "amount": None,
+                                        "commit": None}
+            self.mcc.prepare_privmsg(nick, "pubkey", self.versioned_pubkey_msg(
+                kp.hex_pk().decode('ascii')))
+        else:
+            raise JMProtocolError("Invalid role: " + self.role)
+
+        # This (on_pubkey) method can be called repeatedly without
+        # issue (see note above about active_orders); the following
+        # state update means that after this is called at least once,
+        # the JMMakeTx client call can be answered successfully.
+        self.jm_state = 4
+
+    def on_seen_tx(self, nick, txhex):
+        """Passes the txhex to the Taker and Maker (two way sending);
+        the Maker receives an unsigned tx from the Taker, checks it,
+        modifies it and sends a partially signed updated tx back.
+        This functions as a passthrough with a sanity check on the
+        counterparty identity.
+        """
+        if nick not in self.active_orders:
+            return
+        d = self.callRemote(JMTXReceived,
+                            nick=nick,
+                            txhex=txhex,
+                            offer="none")
+        self.defaultCallbacks(d)
+
+    # Remaining callbacks are mostly to be no-ops (answering
+    # !orderbook as maker, though, makes sense; on-error is also
+    # fine of course, and on-push-tx because it does nothing.)
+    def on_order_fill(self, nick, oid, amount, taker_pk, commit):
+        pass
+    def on_seen_auth(self, nick, commitment_revelation):
+        pass
+    def on_commitment_seen(self, nick, commitment):
+        pass
+    def on_commitment_transferred(self, nick, commitment):
+        pass
+    def on_ioauth(self, nick, utxo_list, auth_pub, cj_addr, change_addr,
+                  btc_sig):
+        pass
+    def on_sig(self, nick, sig):
+        pass
+
+    def versioned_pubkey_msg(self, pubkey_hex):
+        # TODO genericize when updating p2ep version
+        # Note that this is deliberately an instance method
+        # as in future will depend on self.role
+        return pubkey_hex + "0000"
 
 class JMDaemonServerProtocolFactory(ServerFactory):
     protocol = JMDaemonServerProtocol
@@ -654,6 +801,11 @@ class JMDaemonServerProtocolFactory(ServerFactory):
     def buildProtocol(self, addr):
         return JMDaemonServerProtocol(self)
 
+class P2EPDaemonServerProtocolFactory(ServerFactory):
+    protocol = P2EPDaemonServerProtocol
+
+    def buildProtocol(self, addr):
+        return P2EPDaemonServerProtocol(self)
 
 def start_daemon(host, port, factory, usessl=False, sslkey=None, sslcert=None):
     if usessl:

scripts/cli_options.py

@@ -7,6 +7,7 @@ from optparse import OptionParser, OptionValueError
 from configparser import NoOptionError
 
 import jmclient.support
+from jmclient import jm_single, RegtestBitcoinCoreInterface, cryptoengine
 
 """This exists as a separate module for two reasons:
 to reduce clutter in main scripts, and refactor out
@@ -199,6 +200,20 @@ max_cj_fee_rel = {rel_val}\n""".format(rel_val=rel_val, abs_val=abs_val))
     return rel_val, abs_val
 
 
+def check_regtest(blockchain_start=True):
+    """ Applies any regtest-specific configuration
+    """
+    if not isinstance(jm_single().bc_interface,
+                      RegtestBitcoinCoreInterface):
+        return
+    if blockchain_start:
+        #to allow testing of confirm/unconfirm callback for multiple txs
+        jm_single().bc_interface.tick_forward_chain_interval = 10
+        jm_single().bc_interface.simulating = True
+        jm_single().maker_timeout_sec = 5
+    # handles the custom regtest hrp for bech32
+    cryptoengine.BTC_P2WPKH.VBYTE = 100
+
 def get_tumbler_parser():
     parser = OptionParser(
             usage='usage: %prog [options] [wallet file] [destaddr(s)...]',
@@ -412,5 +427,15 @@ def get_sendpayment_parser():
                       dest='answeryes',
                       default=False,
                       help='answer yes to everything')
+    parser.add_option('--payjoin',
+                      '-T',
+                      type='str',
+                      action='store',
+                      dest='p2ep',
+                      default='',
+                      help='specify recipient IRC nick for a '
+                      'p2ep style payment, for example:\n'
+                      'J5Ehn3EieVZFtm4q ')
+
     add_common_options(parser)
     return parser

scripts/receive-payjoin.py

@@ -0,0 +1,88 @@
+#! /usr/bin/env python
+from __future__ import (absolute_import, division,
+                        print_function, unicode_literals)
+from builtins import * # noqa: F401
+
+from optparse import OptionParser
+
+import sys
+from twisted.python.log import startLogging
+from jmbase import get_log, set_logging_level
+from jmclient import P2EPMaker, jm_single, load_program_config, \
+    sync_wallet, JMClientProtocolFactory, start_reactor, \
+    open_test_wallet_maybe, get_wallet_path
+from cli_options import check_regtest
+
+jlog = get_log()
+
+def receive_payjoin_main(makerclass):
+    parser = OptionParser(usage='usage: %prog [options] [wallet file] [amount-to-receive]')
+    parser.add_option('-g', '--gap-limit', action='store', type="int",
+                      dest='gaplimit', default=6,
+                      help='gap limit for wallet, default=6')
+    parser.add_option('--fast',
+                      action='store_true',
+                      dest='fastsync',
+                      default=False,
+                      help=('choose to do fast wallet sync, only for Core and '
+                            'only for previously synced wallet'))
+    parser.add_option('-m', '--mixdepth', action='store', type='int',
+                      dest='mixdepth', default=0,
+                      help="mixdepth to source coins from")
+    parser.add_option('-a',
+                      '--amtmixdepths',
+                      action='store',
+                      type='int',
+                      dest='amtmixdepths',
+                      help='number of mixdepths in wallet, default 5',
+                      default=5)    
+    (options, args) = parser.parse_args()
+    if len(args) < 2:
+        parser.error('Needs a wallet, and a receiving amount in satoshis')
+        sys.exit(0)
+    wallet_name = args[0]
+    try:
+        receiving_amount = int(args[1])
+    except:
+        parser.error("Invalid receiving amount passed: " + receiving_amount)
+        sys.exit(0)
+    if receiving_amount < 0:
+        parser.error("Receiving amount must be a positive integer in satoshis")
+        sys.exit(0)
+    load_program_config()
+
+    check_regtest()
+
+    # This workflow requires command line reading; we force info level logging
+    # to remove noise, and mostly communicate to the user with the fn
+    # log.info (via P2EPMaker.user_info).
+    set_logging_level("INFO")
+
+    wallet_path = get_wallet_path(wallet_name, 'wallets')
+    max_mix_depth = max([options.mixdepth, options.amtmixdepths - 1])
+    wallet = open_test_wallet_maybe(
+        wallet_path, wallet_name, max_mix_depth,
+        gap_limit=options.gaplimit)
+
+    if jm_single().config.get("BLOCKCHAIN", "blockchain_source") == "electrum-server":
+        jm_single().bc_interface.synctype = "with-script"
+
+    while not jm_single().bc_interface.wallet_synced:
+        sync_wallet(wallet, fast=options.fastsync)
+
+    maker = makerclass(wallet, options.mixdepth, receiving_amount)
+    
+    jlog.info('starting receive-payjoin')
+    clientfactory = JMClientProtocolFactory(maker, proto_type="MAKER")
+
+    nodaemon = jm_single().config.getint("DAEMON", "no_daemon")
+    daemon = True if nodaemon == 1 else False
+    if jm_single().config.get("BLOCKCHAIN", "network") in ["regtest", "testnet"]:
+        startLogging(sys.stdout)
+    start_reactor(jm_single().config.get("DAEMON", "daemon_host"),
+                  jm_single().config.getint("DAEMON", "daemon_port"),
+                  clientfactory, daemon=daemon, p2ep=True)
+
+if __name__ == "__main__":
+    receive_payjoin_main(P2EPMaker)
+    print('done')

scripts/sendpayment.py

@@ -14,13 +14,14 @@ import sys
 from twisted.internet import reactor
 import pprint
 
-from jmclient import Taker, load_program_config, get_schedule,\
+from jmclient import Taker, P2EPTaker, load_program_config, get_schedule,\
     JMClientProtocolFactory, start_reactor, validate_address, jm_single,\
-    sync_wallet, RegtestBitcoinCoreInterface, estimate_tx_fee, direct_send,\
+    sync_wallet, estimate_tx_fee, direct_send,\
     open_test_wallet_maybe, get_wallet_path
 from twisted.python.log import startLogging
-from jmbase.support import get_log, jmprint
+from jmbase.support import get_log, set_logging_level, jmprint
-from cli_options import get_sendpayment_parser, get_max_cj_fee_values
+from cli_options import get_sendpayment_parser, get_max_cj_fee_values, \
+     check_regtest
 
 log = get_log()
 
@@ -86,11 +87,7 @@ def main():
 
     wallet_name = args[0]
 
-    #to allow testing of confirm/unconfirm callback for multiple txs
+    check_regtest()
-    if isinstance(jm_single().bc_interface, RegtestBitcoinCoreInterface):
-        jm_single().bc_interface.tick_forward_chain_interval = 10
-        jm_single().bc_interface.simulating = True
-        jm_single().maker_timeout_sec = 15
 
     if options.pickorders:
         chooseOrdersFunc = pick_order
@@ -105,12 +102,12 @@ def main():
     # we guess conservatively with 2 inputs and 2 outputs each.
     if options.txfee == -1:
         options.txfee = max(options.txfee, estimate_tx_fee(2, 2,
-                                                           txtype="p2sh-p2wpkh"))
+                                        txtype="p2sh-p2wpkh"))
         log.debug("Estimated miner/tx fee for each cj participant: " + str(
             options.txfee))
     assert (options.txfee >= 0)
 
-    if options.makercount != 0:
+    if not options.p2ep and options.makercount != 0:
         maxcjfee = get_max_cj_fee_values(jm_single().config, options)
         log.info("Using maximum coinjoin fee limits per maker of {:.4%}, {} "
                  "sat".format(*maxcjfee))
@@ -222,19 +219,28 @@ def main():
                 log.info("All transactions completed correctly")
             reactor.stop()
 
-    taker = Taker(wallet,
+    if options.p2ep:
-                  schedule,
+        # This workflow requires command line reading; we force info level logging
-                  order_chooser=chooseOrdersFunc,
+        # to remove noise, and mostly communicate to the user with the fn
-                  max_cj_fee=maxcjfee,
+        # log.info (directly or via default taker_info_callback).
-                  callbacks=(filter_orders_callback, None, taker_finished))
+        set_logging_level("INFO")
+        taker = P2EPTaker(options.p2ep, wallet, schedule,
+                          callbacks=(None, None, None))
+    else:
+        taker = Taker(wallet,
+                      schedule,
+                      order_chooser=chooseOrdersFunc,
+                      max_cj_fee=maxcjfee,
+                      callbacks=(filter_orders_callback, None, taker_finished))
     clientfactory = JMClientProtocolFactory(taker)
     nodaemon = jm_single().config.getint("DAEMON", "no_daemon")
     daemon = True if nodaemon == 1 else False
+    p2ep = True if options.p2ep != "" else False
     if jm_single().config.get("BLOCKCHAIN", "network") in ["regtest", "testnet"]:
         startLogging(sys.stdout)
     start_reactor(jm_single().config.get("DAEMON", "daemon_host"),
                   jm_single().config.getint("DAEMON", "daemon_port"),
-                  clientfactory, daemon=daemon)
+                  clientfactory, daemon=daemon, p2ep=p2ep)
 
 if __name__ == "__main__":
     main()

scripts/sendtomany.py

@@ -32,6 +32,7 @@ def sign(utxo, priv, destaddrs, segwit=True):
     assert results[0][0] == utxo
     amt = results[0][1]
     ins = [utxo]
+    # TODO extend to other utxo types
     txtype = 'p2sh-p2wpkh' if segwit else 'p2pkh'
     estfee = estimate_tx_fee(1, len(destaddrs), txtype=txtype)
     outs = []

scripts/tumbler.py

@@ -11,11 +11,13 @@ from twisted.python.log import startLogging
 from jmclient import Taker, load_program_config, get_schedule,\
     JMClientProtocolFactory, start_reactor, jm_single, get_wallet_path,\
     open_test_wallet_maybe, sync_wallet, get_tumble_schedule,\
-    RegtestBitcoinCoreInterface, schedule_to_text, restart_waiter,\
+    schedule_to_text, restart_waiter,\
     get_tumble_log, tumbler_taker_finished_update,\
     tumbler_filter_orders_callback
 from jmbase.support import get_log, jmprint
-from cli_options import get_tumbler_parser, get_max_cj_fee_values
+from cli_options import get_tumbler_parser, get_max_cj_fee_values, \
+     check_regtest
+
 log = get_log()
 logsdir = os.path.join(os.path.dirname(
     jm_single().config_location), "logs")
@@ -31,6 +33,8 @@ def main():
         sys.exit(0)
     load_program_config()
 
+    check_regtest()
+
     #Load the wallet
     wallet_name = args[0]
     max_mix_depth = options['mixdepthsrc'] + options['mixdepthcount']
@@ -109,12 +113,6 @@ def main():
         elif fromtx != "unconfirmed":
             reactor.callLater(waittime*60, clientfactory.getClient().clientStart)
 
-    #to allow testing of confirm/unconfirm callback for multiple txs
-    if isinstance(jm_single().bc_interface, RegtestBitcoinCoreInterface):
-        jm_single().bc_interface.tick_forward_chain_interval = 10
-        jm_single().bc_interface.simulating = True
-        jm_single().maker_timeout_sec = 15
-
     #instantiate Taker with given schedule and run
     taker = Taker(wallet,
                   schedule,

scripts/wallet-tool.py

@@ -4,8 +4,10 @@ from builtins import * # noqa: F401
 
 from jmbase import jmprint
 from jmclient import load_program_config, wallet_tool_main
+from cli_options import check_regtest
 
 if __name__ == "__main__":
     load_program_config()
+    check_regtest(blockchain_start=False)
     #JMCS follows same convention as JM original; wallet is in "wallets" localdir
     jmprint(wallet_tool_main("wallets"), "success")

test/ygrunner.py

@@ -19,7 +19,8 @@ import pytest
 import random
 from jmbase import jmprint
 from jmclient import YieldGeneratorBasic, load_program_config, jm_single,\
-    sync_wallet, JMClientProtocolFactory, start_reactor
+    sync_wallet, JMClientProtocolFactory, start_reactor, SegwitWallet,\
+    SegwitLegacyWallet, cryptoengine
 
 
 class MaliciousYieldGenerator(YieldGeneratorBasic):
@@ -102,9 +103,16 @@ def test_start_ygs(setup_ygrunner, num_ygs, wallet_structures, mean_amt,
     Then start the ygs in background and publish
     the seed of the sp wallet for easy import into -qt
     """
+    if jm_single().config.get("POLICY", "native") == "true":
+        walletclass = SegwitWallet
+    else:
+        # TODO add Legacy
+        walletclass = SegwitLegacyWallet
+
     wallets = make_wallets(num_ygs + 1,
                            wallet_structures=wallet_structures,
-                           mean_amt=mean_amt)
+                           mean_amt=mean_amt,
+                           walletclass=walletclass)
     #the sendpayment bot uses the last wallet in the list
     wallet = wallets[num_ygs]['wallet']
     jmprint("\n\nTaker wallet seed : " + wallets[num_ygs]['seed'])
@@ -147,3 +155,5 @@ def setup_ygrunner():
     load_program_config()
     jm_single().bc_interface.tick_forward_chain_interval = 10
     jm_single().bc_interface.simulate_blocks()
+    # handles the custom regtest hrp for bech32
+    cryptoengine.BTC_P2WPKH.VBYTE = 100