diff --git a/README.md b/README.md index a61f5b7..191085a 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ Joinmarket is coinjoin software, and includes a wallet, which requires Bitcoin C * Can specify exact amount of coinjoin (figures from 0.01 to 30.0 btc and higher are practical), can choose time and number of counterparties * Can run passively to receive small payouts for taking part in coinjoins (see "Maker" and "yield-generator" in docs) * GUI to support Taker role, including tumbler/automated coinjoin sequence. +* PayJoin - more economical and private payments between Joinmarket wallets. ### Quickstart @@ -45,6 +46,10 @@ If you are running Joinmarket-Qt, you can instead use the [walkthrough](docs/JOI If you used the old version of Joinmarket, the notes in the [scripts readme](scripts/README.md) help to understand what has and hasn't changed about the scripts. +### PayJoin + +If you want to use the PayJoin feature to pay/receive money to/from another Joinmarket wallet user, read [this guide](docs/PAYJOIN.md). + ### Joinmarket-Qt Provides single join and multi-join/tumbler functionality (i.e. "Taker") only, in a GUI. diff --git a/docs/PAYJOIN.md b/docs/PAYJOIN.md new file mode 100644 index 0000000..3905502 --- /dev/null +++ b/docs/PAYJOIN.md @@ -0,0 +1,384 @@ +### PayJoin (aka P2EP) user guide. + +(You've installed using the `install.sh` as per instructions in the README before +reading this). + +This document does **not** discuss why PayJoin is interesting or the general concept. +For that, see [this](TODO-BLOGPOST) post. + +Some instructions here will be redundant with the introductory [usage guide](USAGE.md); +this guide is aimed at users who have not/ will not use Joinmarket for ordinary coinjoins. +So just skip redundant info if you already know it. + +### Preparatory step: configuring for Bitcoin Core. + +Joinmarket currently requires a Bitcoin Core full node, although it can be pruned. + +First thing to do: in `scripts/`, run: + + python wallet-tool.py generate + +This *should* quit with an error, because the rpc is not configured. Open the newly created file `joinmarket.cfg`, +and edit: + + [BLOCKCHAIN] + rpc_user = yourusername-as-in-bitcoin.conf + rpc_password = yourpassword-as-in-bitcoin.conf + rpc_host = localhost #default usually correct + rpc_port = 8332 # default for mainnet + +Note, you can also use a cookie file by setting, in this section, a variable `rpc_cookie_file` to the location of the file, +as an alternative to using user/password. + +If you use Bitcoin Core's multiwallet feature, you can edit the value of `rpc_wallet_file` to your chosen wallet file. + +Then retry the same `generate` command; it should now not error - continue the generate process as per next step. + +If you still get rpc connection errors, make sure you can connect to your Core node using the command line first. + +### Make and fund the wallet + +Continue/complete the wallet generation with the above (`generate`) method. + +(Before you finish: want a bech32 wallet? you probably don't, +but read [this](#what-if-i-wanted-bech32-native-segwit-addresses) in case.) + +The wallet you create is BIP49 by default, using BIP39 12 word seed, +mnemonic extension optional (simplest to leave it out if you're not sure). + +Once the `generate` method run has completed, successfully, you need to look at the wallet contents. Use +the `display` method which is the default: + +``` +python wallet-tool.py wallet-name-you-chose.dat [display] +``` + +("display" is optional because default; use `python wallet-tool.py -h` to see all possible methods). + +Below is [an example](#sample-testnet-wallet-display-output) of what the wallet looks like (although +yours will be mainnet, so the addresses will start with '3' not '2'). + +Joinmarket by default uses *5* accounts, not only 1 as some other wallets (e.g. Electrum), this is to help +with coin isolation. Try to move coins from one account to another *only* via coinjoins; or, you can just +use one or two accounts (called "mixdepths" in Joinmarket parlance) as if they were just one, understanding +that if you don't bother to do anything special, the coins in those two mixdepths are linked. + +Fund the wallet by choosing one or more addresses in the *external* section of the first account (called +"Mixdepth 0" here). When you fund, fund to the external addresses. The internals will be used for change. + +(The new standard (BIP49) *should* be compatible with TREZOR, Ledger, Electrum, Samourai and some others, +including the 12 word seed, although consider privacy concerns when sending addresses to remote servers!). + +### Doing a PayJoin payment. + +(At the end of this file are full terminal outputs from a regtest run of the process, +you can read it after this to see that it makes sense; there's also a video +[here](https://joinmarket.me/blog/blog/payjoin-basic-demo) of the process running live with mainnet coins). + +* Receiver needs to start: run (still in scripts/ directory): + +``` +python receive-payjoin.py -m 1 receiver-wallet-name.jmdat amount-in-satoshis +``` + +Note : `-m 1` is choosing the *mixdepth* (see above) to *spend* coins from: in a payjoin, +the receiver also spends some coins, he just gets that amount back, as well as his payment. + +If you funded into mixdepth 0 at the start, and you only have coins there, you must choose 0 +here (which is the default). How much do you need? The code is fairly lenient and it doesn't +matter too much. But it needs to be more than zero! + +The receiver will be prompted to read/note the receiving address, amount and "ephemeral nick", and then +the script will just wait (indefinitely). It'll look something like this: + +``` +2019-01-16 16:11:40,018 [INFO] Your receiving address is: 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS +2019-01-16 16:11:40,018 [INFO] You will receive amount: 27000000 satoshis. +2019-01-16 16:11:40,018 [INFO] The sender also needs to know your ephemeral nickname: J5AFezpsuV95CBCH +2019-01-16 16:11:40,018 [INFO] This information has been stored in a file payjoin.txt; send it to your counterparty when you are ready. +``` + +The "ephemeral nick" starts with J, version (5 currently), then a short base58 string. It's how the sender will find you in the +"joinmarket trading pit" (multiple IRC servers are used for this currently). + +* Receiver sends data to sender (amount, address, ephemeral nick). + +This data is stored in the file payjoin.txt but not currently using any encoding (that's a TODO). + +* Sender starts up the sendpayment script: + +``` +python sendpayment.py -m 1 receiver-wallet.jmdat 27000000 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS -T J5AFezpsuV95CBCH +``` + +Notice that the user has specified the three pieces of data given; using the `-T` flags this as a PayJoin; if you don't do this you will be +doing a Joinmarket coinjoin accidentally! (which wouldn't be the end of the world, the receiver would still get the money!). + +As before -m 1 tells the wallet which mixdepth to source coins to spend from; it obviously needs to have at least the given amount +(in this case, 27 million satoshis or 0.27 btc). + +* The two sides communicate + +This takes generally only a couple of seconds, not including a relatively slow startup (30s-60s or so depending on configuration), +during which time the sender acts similarly to any other Joinmarket participant (and so does the receiver, likewise), to have +a slightly better "blend in with the crowd" effect. + +Once the transaction is broadcast, you'll see a message to that effect and the script will shutdown. You can check the txid +on your favourite block explorer. + +* What if something goes wrong and the payment fails to go through? + +First, if the sender succeeds in doing the first step, the receiver will have a non-coinjoin ordinary payment transaction +to fall back on. It'll look like this: + +``` +2019-01-16 16:34:55,168 [INFO] Network transaction fee is: 1672 satoshis. +2019-01-16 16:34:55,175 [INFO] We'll use this serialized transaction to broadcast if your counterparty fails to broadcast the payjoin version: +2019-01-16 16:34:55,177 [INFO] 0200000000010152da645b5a2ec3a166ad8a933b1442fa38fc119faf7659d72033ba863fdd8d470000000017160014297b55001daa905a3552137ef19755cf4eae7babfeffffff02b8be4f0a0000000017a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87c0fc9b010000000017a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc8702483045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a65012102f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f42030000 +2019-01-16 16:34:55,186 [INFO] The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing.. +.... +``` + +If the final step fails and we don't get a PayJoin, the receiver can just use his favorite broadcast method (e.g. +`sendrawtransaction` in Bitcoin Core) to send the signed transaction above (02000...). + +* Privacy and security controls + +Just like Joinmarket, do note, that here the private information is communicated with **end-to-end encryption** - it's not like +the people/operators on the IRC servers are going to learn any information about your transaction. + +Second, it's very recommended to use Tor connections to the messaging servers via their hidden services. See the +`[MESSAGING:serverN]` sections in the `joinmarket.cfg`. This keeps anyone from seeing your IP address origin. + +This altogether means it's possible to have anonymity from your sender/receiver counterparty, if that's an issue. +You *do* still have to send them the payment information out of band, though. We could perhaps fold this in, although +it seems a bit tricky to do so without introducing a security issue. + +An additional minor feature is that the receiver "fakes" being a Joinmarket maker, so his bot looks the same as the other +ones in the Joinmarket trading pit (at least, to a crude extent). And the sender fakes being a Joinmarket taker, too. + +Also we try to make the PayJoin look as much as possible as an ordinary payment. For example: + - we make the transaction version, locktime and sequence numbers look similar to those created by Core, Electrum. + - we try to avoid "UIH2", meaning we avoid a situation where one input is larger than any output, since that would + mean no other inputs are needed; wallet coin selection algorithms *usually* don't do that. + +Security - since the receiver passively waits, what happens if a bad actor tries to connect to him? Nothing; an attacker +would fail to even start the process without knowing the payment amount and address, which the receiver is not broadcasting +around everywhere (especially not the amount and ephemeral nickname), and even if they knew that, the worst +they can do is learn at least 1 utxo of the receiver. The receiver won't pay attention to non-PayJoin messages, either. + +#### What if I wanted bech32 native segwit addresses? + +You can do this, but bear in mind: PayJoin only gives its full effect if you and your receiver are using +the same kind of addresses; so do this only if you and your receiver(s)/sender(s) agree on it. + +Also note: you *cannot* do Joinmarket coinjoins if you choose a bech32 wallet (this may change in future). + +In the configuration file `joinmarket.cfg` (which was created in the preparatory step above), go to the +POLICY section and set: + +``` +[POLICY] +native = true +``` + +Note that this must be done *before* generating the wallet, as +the wallet internally, of course, stores which type of addresses it manages, and it can only be of two +types currently (ignoring legacy upgrades): bech32 or p2sh-segwit (starting with '3'), the latter being +the default (and the one used in Joinmarket itself). + +Note that the bech32 style wallet is written to conform to [BIP84](https://github.com/bitcoin/bips/blob/master/bip-0084.mediawiki), +analogous to the BIP49 case for p2sh. + +#### Sample testnet wallet display output + +``` +JM wallet +mixdepth 0 tpubDC4qk8DsyiFYY85uktoKaiR1srWLSNRxZ2A4MXYg5LHy9XHKSTcF2tNtFpGz4LzXPcDH1kNkiww7MwipNqNSps6HSyjPTqTB18J7C4jrFMi +external addresses m/49'/1'/0'/0 tpubDFFCJfi4y6xPKquA6H6aP5EBZPupb6A9tJz8sErW8GN6D7D3s9MABPt1BczpQ3n8rBv7VLSVpu3ddvb7xEKMfNX2sMZ7XxiBD4J6kpfF7Ag +m/49'/1'/0'/0/0 2NBT6npWKxBEG8fkDjSFLDZJ7fNda4kYnAx 2.00000000 deposit +m/49'/1'/0'/0/1 2Mt7dBFikYwCQTDU129VTF9ahKWxeJjEUuF 0.00000000 new +m/49'/1'/0'/0/2 2N94bte8xWojSaX6yq2th3R3mUhvKzZqDJ9 0.00000000 new +m/49'/1'/0'/0/3 2MvSBTkCKwPHLdNPRXbovPXUM8oAfjUFPYc 0.00000000 new +m/49'/1'/0'/0/4 2MzVn3Nc3RRyN7shiVajA225xCTcaGn1PRw 0.00000000 new +m/49'/1'/0'/0/5 2MxJwv2dmkMupDuBLsEMa4HgG6GAieWHiXr 0.00000000 new +m/49'/1'/0'/0/6 2MwhdkeAcnCkam1LdVBQJ7un8syyoYB1HVH 0.00000000 new +Balance: 2.00000000 +internal addresses m/49'/1'/0'/1 +Balance: 0.00000000 +Balance for mixdepth 0: 2.00000000 +mixdepth 1 tpubDC4qk8DsyiFYba4t8cSpadjoLYUdPwV5dAtBpzpPzgaDKPfSP42xNJq48QUKEVGHQRfFej6DCUjQqCKD8TtcqN932f27jmyjXaxVMpksos4 +external addresses m/49'/1'/1'/0 tpubDEnijtftQiJpVgezdRNyKVVWGr9xKV9RgPQWHYDHpQ5utdHF7Sqh7xMUyNHcpdeKcKdQ753hFbyccRZNEHTUkHLnDaVqoXRo9XkATPtHhCp +m/49'/1'/1'/0/0 2NEsN45waxdkqjP5EnsP3K8YjMeZJh2RLx6 2.00000000 deposit +m/49'/1'/1'/0/1 2Mzhewfg6fr5jR122txdTShmLi7rH9ZscTm 2.00000000 deposit +m/49'/1'/1'/0/2 2N1vn44cv5m6PhRJMbZ1mR8dAJgHybnsT5u 2.00000000 deposit +m/49'/1'/1'/0/3 2NFwBsHkQ8mxCJWRJhkxAY28Tj8YuUJio4t 0.00000000 new +m/49'/1'/1'/0/4 2NA1YAa2VqHMSH9b2GYGBA8waUMxZTZwW5Z 0.00000000 new +m/49'/1'/1'/0/5 2MvLLp4cnP8ZVuWWDZzRCLFM2YcTfo9ALec 0.00000000 new +m/49'/1'/1'/0/6 2Mux3ZUHGmaMBiMsPDbQS56gRGGN26jMGzd 0.00000000 new +m/49'/1'/1'/0/7 2N3MWFiSHyRY3QLZgb63Vfcp4BGSFs6Y3bV 0.00000000 new +m/49'/1'/1'/0/8 2N3z5zKJPTfPc41esRRvMvjdKaSv6D6jqvY 0.00000000 new +Balance: 6.00000000 +internal addresses m/49'/1'/1'/1 +Balance: 0.00000000 +Balance for mixdepth 1: 6.00000000 +mixdepth 2 tpubDC4qk8DsyiFYdQBNNzo3vHq63Gag4eUJT29UaTVNHM89hJk6CshZ9WGemQNDh2LGDXCud8anAQ4UR7n7tSWiJtviR8WJuTB78ZbEHpFNcLH +external addresses m/49'/1'/2'/0 tpubDEY4sVvs1TX82DftUUB51Agg4Ln7BksoRGESNWtrWTDntV2fCs5wNrqiPENcXBAEHtnaY9ZaK48PRFEw1GLhcTxdDNHUyuqDd2YyNYKoVAo +m/49'/1'/2'/0/0 2NBSnSB3nVN4TA7EcNkhcRsmrdyXSQepDFE 0.00000000 new +m/49'/1'/2'/0/1 2NFtrtpdvmCRXMy1fV8w1eLWpF8MC3nre7n 0.00000000 new +m/49'/1'/2'/0/2 2NFHJvGLWU7KuNkbo8rzPwfGtCS5RtNDw7c 0.00000000 new +m/49'/1'/2'/0/3 2NGJcmeScFnTSzZzSRNHiLL8zjYeA9ngx5A 0.00000000 new +m/49'/1'/2'/0/4 2NBUAxKrNQtp49xYYHh9f6YHfo2FvYBP1NL 0.00000000 new +m/49'/1'/2'/0/5 2MyQfzKyPTfYT4vTe2d3fyGvQMoGX6GAhcr 0.00000000 new +Balance: 0.00000000 +internal addresses m/49'/1'/2'/1 +Balance: 0.00000000 +Balance for mixdepth 2: 0.00000000 +mixdepth 3 tpubDC4qk8DsyiFYgXoNb3UmiaG2veSTdrLCxEBUsTEQnDgQLCaC8Yg1z1Bcwn4ZQivNhgBxHEWH7j8hbRx7rab2kYLy4r4PXxor4Ho3A5AJvVH +external addresses m/49'/1'/3'/0 tpubDEnwog2atqaSLe9xVTTdqxY5ynysqeQPsXuaKrZ6HaYCJcFPY7LmhepmwFTJYTkqf1w5jgLQmoZvREyk9qiq4P2A2fSGTyj62WUE4VjXK76 +m/49'/1'/3'/0/0 2MtckYQLD6bJZiPXffW7m5rryMao7u4ktng 0.00000000 new +m/49'/1'/3'/0/1 2N9KrH5Bi35ZH3DVUwD67eh8MtQD1LLhgCa 0.00000000 new +m/49'/1'/3'/0/2 2N6zLabMeXTXfyzQttr6PBV2JdbZPrVd9i8 0.00000000 new +m/49'/1'/3'/0/3 2MzYTWMjXcv4NBce5PKcDAceT7gMuYrGovc 0.00000000 new +m/49'/1'/3'/0/4 2N6tYykyCZLtvP1RJZvgZ9a7c6xbQaqpE66 0.00000000 new +m/49'/1'/3'/0/5 2N2o3eJ9h2BC5q3TzPVqhn9gmgWBjaL67Hu 0.00000000 new +Balance: 0.00000000 +internal addresses m/49'/1'/3'/1 +Balance: 0.00000000 +Balance for mixdepth 3: 0.00000000 +mixdepth 4 tpubDC4qk8DsyiFYj7GF4LV97c9f7Yff1mrwnxkpi5twGSLmesmPyM4xXBWsxMw9ZLFycVVC4TeeX1ESjNP4rVVrJEDVCm7C3UMvZH9vs6srsAi +external addresses m/49'/1'/4'/0 tpubDFA3XKgf2ZiusZHr4we5utkrQ9toN5s7QGKndNMrdQFQfjQU6yiiMT65tmXFCPduSc7muLFegAi36pv4LCdRnhpRYp2QUpm1izyrboWSjzV +m/49'/1'/4'/0/0 2N18HGRJyaaUwLQFTyfjqZoXGCV8Yv5rbwD 0.00000000 new +m/49'/1'/4'/0/1 2Mt94Y4iguYLkBAhsXT5a1P8VMTQ4kxdZJD 0.00000000 new +m/49'/1'/4'/0/2 2N5vYyG1gx8ht2JCKkuVqow3Qn51cHrwaxh 0.00000000 new +m/49'/1'/4'/0/3 2N44ZtKYu21p1qN4DBoTCYZL2sms6YBeWVW 0.00000000 new +m/49'/1'/4'/0/4 2NF9p2b2PfHvXiPmPP5JPq4CRxKn47LPZrW 0.00000000 new +m/49'/1'/4'/0/5 2MwgfAvbPc4ASD84WdBYo2FM5bXBVG1rRG9 0.00000000 new +Balance: 0.00000000 +internal addresses m/49'/1'/4'/1 +Balance: 0.00000000 +Balance for mixdepth 4: 0.00000000 +Total balance: 8.00000000 +``` + +#### Full sender-side log of a regtest run of PayJoin + +(Note that the "Received offers from joinmarket pit" message is a privacy feature, we don't actually respond to any offers). + +``` +(jmvenv) me@here:~/jm/scripts$ python sendpayment.py -m1 b80d142a466dbf56f518a3a8c017ab85 27000000 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS -T J5AFezpsuV95CBCH +2019-01-16 16:34:44,877 [WARNING] Cannot listen on port 27183, trying next port +2019-01-16 16:34:44,877 [WARNING] Cannot listen on port 27184, trying next port +2019-01-16 16:34:44,877 [WARNING] Cannot listen on port 27185, trying next port +2019-01-16 16:34:44,878 [WARNING] Cannot listen on port 27186, trying next port +2019-01-16 16:34:44,878 [INFO] Listening on port 27187 +2019-01-16 16:34:47,495 [INFO] Could not connect to *ALL* servers yet, waiting up to 60 more seconds. +2019-01-16 16:34:47,496 [INFO] All IRC servers connected, starting execution. +2019-01-16 16:34:47,501 [INFO] JM daemon setup complete +2019-01-16 16:34:52,508 [INFO] INFO:Received offers from joinmarket pit +2019-01-16 16:34:52,531 [INFO] total estimated amount spent = 27020000 +2019-01-16 16:34:52,594 [INFO] Makers responded with: ["J5AFezpsuV95CBCH"] +2019-01-16 16:34:52,600 [INFO] Obtained proposed payjoin tx +{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52', + 'index': 0}, + 'script': '', + 'sequence': 4294967294}], + 'locktime': 834, + 'outs': [{'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87', + 'value': 172998328}, + {'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87', + 'value': 27000000}], + 'version': 2} +2019-01-16 16:34:52,602 [INFO] INFO:Built tx proposal, sending to receiver. +2019-01-16 16:34:56,622 [INFO] Obtained tx from receiver: +{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52', + 'index': 0}, + 'script': '', + 'sequence': 4294967294, + 'txinwitness': []}, + {'outpoint': {'hash': '5e19edacc10298e7761be7231db5fa44ad73903c3690c4b71b443355267d346a', + 'index': 0}, + 'script': '160014a5b07a64bff72b442c71761599e0b627c687661f', + 'sequence': 4294967294, + 'txinwitness': ['3044022077a811939910a935934f68ac8a70439bab665ce9ad4cd02e42289c3046d606e1022054ef241980733caa7c7c5b6bb1babf96caba83a9fb0a8e091d8358b1cb35fcdb01', + '02c7ceecf0783ecc7530e018397db434af63eda63765c79816f023699a22663926']}], + 'locktime': 834, + 'outs': [{'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87', + 'value': 227000000}, + {'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87', + 'value': 172997415}], + 'version': 2} +2019-01-16 16:34:56,630 [INFO] INFO:Network transaction fee is: 2585 satoshis. +2019-01-16 16:34:56,641 [INFO] txid = 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae +2019-01-16 16:34:56,661 [INFO] Transaction broadcast OK. +2019-01-16 16:35:01,651 [INFO] Transaction seen on network, shutting down. +2019-01-16 16:35:01,652 [INFO] Txid was: 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae +``` + +#### Full receiver-side log of a regtest run of PayJoin + +``` +(jmvenv) me@here:~/jm/scripts$ python receive-payjoin.py -m1 84735f364c2cf4c8ddaa614315aeae14 27000000 +2019-01-16 16:11:39,952 [INFO] offerlist=[{'maxsize': 112412265, 'ordertype': 'swreloffer', 'cjfee': '0.00035792', 'oid': 0, 'minsize': 1095839, 'txfee': 493}] +2019-01-16 16:11:39,952 [INFO] starting receive-payjoin +2019-01-16 16:11:40,011 [WARNING] Cannot listen on port 27183, trying next port +2019-01-16 16:11:40,011 [WARNING] Cannot listen on port 27184, trying next port +2019-01-16 16:11:40,011 [WARNING] Cannot listen on port 27185, trying next port +2019-01-16 16:11:40,012 [INFO] Listening on port 27186 +2019-01-16 16:11:40,018 [INFO] Your receiving address is: 2NA65YN6eXf3LiciBb1dEdS6ovaZ8HVBcHS +2019-01-16 16:11:40,018 [INFO] You will receive amount: 27000000 satoshis. +2019-01-16 16:11:40,018 [INFO] The sender also needs to know your ephemeral nickname: J5AFezpsuV95CBCH +2019-01-16 16:11:40,018 [INFO] This information has been stored in a file payjoin.txt; send it to your counterparty when you are ready. +2019-01-16 16:35:00+0100 [-] Enter 'y' to wait for the payment:y +2019-01-16 16:34:32,449 [INFO] Could not connect to *ALL* servers yet, waiting up to 60 more seconds. +2019-01-16 16:34:32,450 [INFO] All IRC servers connected, starting execution. +2019-01-16 16:34:32,453 [INFO] JM daemon setup complete + +(note time gap here; just waiting) + +2019-01-16 16:34:55,163 [INFO] obtained tx proposal from sender: +{'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52', + 'index': 0}, + 'script': '160014297b55001daa905a3552137ef19755cf4eae7bab', + 'sequence': 4294967294, + 'txinwitness': ['3045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a6501', + '02f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f']}], + 'locktime': 834, + 'outs': [{'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87', + 'value': 172998328}, + {'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87', + 'value': 27000000}], + 'version': 2} +2019-01-16 16:34:55,165 [WARNING] Connection had broken pipe, attempting reconnect. +2019-01-16 16:34:55,168 [INFO] Network transaction fee is: 1672 satoshis. +2019-01-16 16:34:55,175 [INFO] We'll use this serialized transaction to broadcast if your counterparty fails to broadcast the payjoin version: +2019-01-16 16:34:55,177 [INFO] 0200000000010152da645b5a2ec3a166ad8a933b1442fa38fc119faf7659d72033ba863fdd8d470000000017160014297b55001daa905a3552137ef19755cf4eae7babfeffffff02b8be4f0a0000000017a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87c0fc9b010000000017a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc8702483045022100e3605548f5e07ebd14a0700dc3e54e7c8ae90ce0057a3fdc80b5fd37636b44a002202fd7942104fb344b80b8fd91faf0394a0f274f7a68fa462084b8ecf2fa245a65012102f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f42030000 +2019-01-16 16:34:55,186 [INFO] The proposed tx does not trigger UIH2, which means it is indistinguishable from a normal payment. This is the ideal case. Continuing.. +2019-01-16 16:34:55,187 [INFO] We selected inputs worth: 200000000 +2019-01-16 16:34:55,195 [INFO] We estimated a fee of: 2585 +2019-01-16 16:34:55,196 [INFO] We calculated a new change amount of: 172997415 +2019-01-16 16:34:55,197 [INFO] We calculated a new destination amount of: 227000000 +2019-01-16 16:35:00,232 [INFO] The transaction has been broadcast. +2019-01-16 16:35:00,232 [INFO] Txid is: 1031780b31bd3b1cfdec44296fdb82e467284e93f871eb48d7d3e72df059f0ae +2019-01-16 16:35:00,233 [INFO] Transaction in detail: {'ins': [{'outpoint': {'hash': '478ddd3f86ba3320d75976af9f11fc38fa42143b938aad66a1c32e5a5b64da52', + 'index': 0}, + 'script': '160014297b55001daa905a3552137ef19755cf4eae7bab', + 'sequence': 4294967294, + 'txinwitness': ['3045022100c1d579fb19f5b8710b407e3995657313d0b1c9fe1180b852834b2210ba19b5e30220464f034eef8d8975b4fe10db7edbf14fcfdc6d71ada3d2063d34fa5c04a38cb801', + '02f823d62891d8bc8544d4369bb98c6fb8235a372d2c36196d40c448690b42754f']}, + {'outpoint': {'hash': '5e19edacc10298e7761be7231db5fa44ad73903c3690c4b71b443355267d346a', + 'index': 0}, + 'script': '160014a5b07a64bff72b442c71761599e0b627c687661f', + 'sequence': 4294967294, + 'txinwitness': ['3044022077a811939910a935934f68ac8a70439bab665ce9ad4cd02e42289c3046d606e1022054ef241980733caa7c7c5b6bb1babf96caba83a9fb0a8e091d8358b1cb35fcdb01', + '02c7ceecf0783ecc7530e018397db434af63eda63765c79816f023699a22663926']}], + 'locktime': 834, + 'outs': [{'script': 'a914b8bf57f3bae00d23f9a60c9a6e4d4c7182ec87cc87', + 'value': 227000000}, + {'script': 'a914e3cb2bf72ccd4412035f9668e07e17b6c9ebd58d87', + 'value': 172997415}], + 'version': 2} +2019-01-16 16:35:00,234 [INFO] shutting down. +``` + + diff --git a/jmbitcoin/jmbitcoin/secp256k1_main.py b/jmbitcoin/jmbitcoin/secp256k1_main.py index 67f3c50..ff839fe 100644 --- a/jmbitcoin/jmbitcoin/secp256k1_main.py +++ b/jmbitcoin/jmbitcoin/secp256k1_main.py @@ -584,6 +584,13 @@ def ecdsa_raw_verify(msg, pub, sig, usehex, rawmsg=False): def estimate_tx_size(ins, outs, txtype='p2pkh'): '''Estimate transaction size. + The txtype field as detailed below is used to distinguish + the type, but there is at least one source of meaningful roughness: + we assume the output types are the same as the input (to be fair, + outputs only contribute a little to the overall total). This combined + with a few bytes variation in signature sizes means we will expect, + say, 10% inaccuracy here. + Assuming p2pkh: out: 8+1+3+2+20=34, in: 1+32+4+1+1+~73+1+1+33=147, ver:4,seq:4, +2 (len in,out) @@ -592,6 +599,16 @@ def estimate_tx_size(ins, outs, txtype='p2pkh'): "ins" must contain M, N so ins= (numins, M, N) (crude assuming all same) 74*M + 34*N + 45 per input, so total ins ~ len_ins * (45+74M+34N) so total ~ 34*len_out + (45+74M+34N)*len_in + 10 + Assuming p2sh-p2wpkh: + witness are roughly 3+~73+33 for each input + (txid, vin, 4+20 for witness program encoded as scriptsig, 4 for sequence) + non-witness input fields are roughly 32+4+4+20+4=64, so total becomes + n_in * 64 + 4(ver) + 4(locktime) + n_out*34 + Assuming p2wpkh native: + witness as previous case + non-witness loses the 24 witnessprogram, replaced with 1 zero, + in the scriptSig, so becomes: + n_in * 41 + 4(ver) + 4(locktime) +2 (len in, out) + n_out*34 ''' if txtype == 'p2pkh': return 10 + ins * 147 + 34 * outs @@ -599,12 +616,15 @@ def estimate_tx_size(ins, outs, txtype='p2pkh'): #return the estimate for the witness and non-witness #portions of the transaction, assuming that all the inputs #are of segwit type p2sh-p2wpkh - #witness are roughly 3+~73+33 for each input - #non-witness input fields are roughly 32+4+4+20+4=64, so total becomes - #n_in * 64 + 4(ver) + 4(locktime) + n_out*34 + n_in * 109 + # Note as of Jan19: this misses 2 bytes (trivial) for len in, out + # and also overestimates output size by 2 bytes. witness_estimate = ins*109 non_witness_estimate = 4 + 4 + outs*34 + ins*64 return (witness_estimate, non_witness_estimate) + elif txtype == 'p2wpkh': + witness_estimate = ins*109 + non_witness_estimate = 4 + 4 + 2 + outs*31 + ins*41 + return (witness_estimate, non_witness_estimate) elif txtype == 'p2shMofN': ins, M, N = ins return 10 + (45 + 74*M + 34*N) * ins + 34 * outs diff --git a/jmbitcoin/jmbitcoin/secp256k1_transaction.py b/jmbitcoin/jmbitcoin/secp256k1_transaction.py index d1e6088..031bcc4 100644 --- a/jmbitcoin/jmbitcoin/secp256k1_transaction.py +++ b/jmbitcoin/jmbitcoin/secp256k1_transaction.py @@ -438,7 +438,7 @@ def is_segwit_native_script(script): return True return False -def script_to_address(script, vbyte=0, witver=0): +def script_to_address(script, vbyte=b'\x00', witver=0): """ Given a hex or bytes script, and optionally a version byte (for P2SH) and/or a witness version (for native segwit witness programs), convert to a valid address (either bech32 or Base58CE). @@ -452,7 +452,7 @@ def script_to_address(script, vbyte=0, witver=0): script = binascii.unhexlify(script) if is_segwit_native_script(script): #hrp interpreted from the vbyte entry, TODO: better way? - if vbyte in [0, 5]: + if vbyte in [b'\x00', b'\x05']: hrp = 'bc' elif vbyte == 100: hrp = 'bcrt' @@ -465,7 +465,7 @@ def script_to_address(script, vbyte=0, witver=0): return bin_to_b58check(script[3:-2], vbyte) else: # BIP0016 scripthash addresses: requires explicit vbyte set - if vbyte == 0: raise Exception("Invalid version byte for P2SH") + if vbyte == b'\x00': raise Exception("Invalid version byte for P2SH") return bin_to_b58check(script[2:-1], vbyte) def pubkey_to_script(pubkey, script_pre, script_post=b'', @@ -820,8 +820,23 @@ def apply_multisignatures(*args): txobj["ins"][i]["script"] = serialize_script([None] + sigs + [script]) return serialize(txobj) -def mktx(ins, outs, version=1): - txobj = {"locktime": 0, "version": version, "ins": [], "outs": []} +def mktx(ins, outs, version=1, locktime=0): + """ Given a list of input dicts with key "output" + which are txid:n strings in hex, and a list of outputs + which are dicts with keys "address", "value", outputs + a hex serialized tranasction encoding this data. + Tx version and locktime are optionally set, for non-default + locktimes, inputs are given nSequence as per below comment. + """ + txobj = {"locktime": locktime, "version": version, "ins": [], "outs": []} + # This does NOT trigger rbf and mimics Core's standard behaviour as of + # Jan 2019. + # Tx creators wishing to use rbf will need to set it explicitly outside + # of this function. + if locktime != 0: + sequence = 0xffffffff - 1 + else: + sequence = 0xffffffff for i in ins: if isinstance(i, dict) and "outpoint" in i: txobj["ins"].append(i) @@ -832,7 +847,7 @@ def mktx(ins, outs, version=1): "outpoint": {"hash": i[:64], "index": int(i[65:])}, "script": "", - "sequence": 4294967295 + "sequence": sequence }) for o in outs: if isinstance(o, str): diff --git a/jmclient/jmclient/__init__.py b/jmclient/jmclient/__init__.py index aaa3bad..8d7fcf6 100644 --- a/jmclient/jmclient/__init__.py +++ b/jmclient/jmclient/__init__.py @@ -8,21 +8,22 @@ from .support import (calc_cj_fee, choose_sweep_orders, choose_orders, cheapest_order_choose, weighted_order_choose, rand_norm_array, rand_pow_array, rand_exp_array, select, select_gradual, select_greedy, select_greediest, - get_random_bytes, random_under_max_order_choose) + get_random_bytes, random_under_max_order_choose, + select_one_utxo) from .jsonrpc import JsonRpcError, JsonRpcConnectionError, JsonRpc from .old_mnemonic import mn_decode, mn_encode -from .taker import Taker +from .taker import Taker, P2EPTaker from .wallet import (Mnemonic, estimate_tx_fee, WalletError, BaseWallet, ImportWalletMixin, BIP39WalletMixin, BIP32Wallet, BIP49Wallet, LegacyWallet, SegwitWallet, SegwitLegacyWallet, UTXOManager, - WALLET_IMPLEMENTATIONS) + WALLET_IMPLEMENTATIONS, make_shuffled_tx) from .storage import (Argon2Hash, Storage, StorageError, StoragePasswordError, VolatileStorage) from .cryptoengine import BTCEngine, BTC_P2PKH, BTC_P2SH_P2WPKH, EngineError from .configure import ( load_program_config, get_p2pk_vbyte, jm_single, get_network, validate_address, get_irc_mchannels, get_blockchain_interface_instance, - get_p2sh_vbyte, set_config, is_segwit_mode) + get_p2sh_vbyte, set_config, is_segwit_mode, is_native_segwit_mode) from .blockchaininterface import (BlockchainInterface, sync_wallet, RegtestBitcoinCoreInterface, BitcoinCoreInterface) from .electruminterface import ElectrumInterface @@ -45,7 +46,7 @@ from .wallet_utils import ( wallet_tool_main, wallet_generate_recover_bip39, open_wallet, open_test_wallet_maybe, create_wallet, get_wallet_cls, get_wallet_path, wallet_display) -from .maker import Maker +from .maker import Maker, P2EPMaker from .yieldgenerator import YieldGenerator, YieldGeneratorBasic, ygmain # Set default logging handler to avoid "No handler found" warnings. diff --git a/jmclient/jmclient/client_protocol.py b/jmclient/jmclient/client_protocol.py index 047e721..414434f 100644 --- a/jmclient/jmclient/client_protocol.py +++ b/jmclient/jmclient/client_protocol.py @@ -72,6 +72,9 @@ class JMClientProtocol(amp.AMP): #The constructed length will be 1 + 1 + NICK_MAX_ENCODED self.nick = self.nick_header + str(self.jm_version) + self.nick_pkh jm_single().nickname = self.nick + informuser = getattr(self.client, "inform_user_details", None) + if callable(informuser): + informuser() @commands.JMInitProto.responder def on_JM_INIT_PROTO(self, nick_hash_length, nick_max_encoded, @@ -126,6 +129,41 @@ class JMClientProtocol(amp.AMP): self.defaultCallbacks(d) return {'accepted': True} + def make_tx(self, nick_list, txhex): + d = self.callRemote(commands.JMMakeTx, + nick_list= json.dumps(nick_list), + txhex=txhex) + self.defaultCallbacks(d) + + def on_p2ep_tx_received(self, nick, txhex): + """ This is called by both "maker" and "taker" + in the p2ep protocol; taker sends signed fallback + non-coinjoin transaction to maker, then maker sends + partially signed payjoin to taker. + Processing at client protocol level is the + same, business logic deferred to the P2EPMaker,Taker + classes. + """ + retval = self.client.on_tx_received(nick, txhex) + if not retval[0]: + if retval[1] == "OK": + jlog.info("Transaction broadcast OK.") + else: + jlog.info("We refused to continue on receiving tx") + jlog.info("Reason: " + retval[1]) + else: + # This will only be called on the "maker"/receiver + # side, who will pass back the new partially signed + # version; on Taker side, we stop above, since we + # just sign and push within the P2EPTaker. + nick, txhex = retval[1:] + # make_tx is slightly misnamed; it just transfers + # the transaction to the given counterparties (here + # only one) over the message channel, via the + # AMP command JMMakeTx. + self.make_tx([nick], txhex) + return {"accepted": True} + class JMMakerClientProtocol(JMClientProtocol): def __init__(self, factory, maker, nick_priv=None): self.factory = factory @@ -209,6 +247,11 @@ class JMMakerClientProtocol(JMClientProtocol): @commands.JMTXReceived.responder def on_JM_TX_RECEIVED(self, nick, txhex, offer): + # "none" flags p2ep protocol; pass through to the generic + # on_tx handler for that: + if offer == "none": + return self.on_p2ep_tx_received(nick, txhex) + offer = json.loads(offer) retval = self.client.on_tx_received(nick, txhex, offer) if not retval[0]: @@ -437,16 +480,20 @@ class JMTakerClientProtocol(JMClientProtocol): self.push_tx(nick_to_use, txhex) return {'accepted': True} + @commands.JMTXReceived.responder + def on_JM_TX_RECEIVED(self, nick, txhex, offer): + """ Only used in the P2EP protocol. + """ + if not offer == "none": + # Protocol error; this message should only ever + # be received, Taker side, in the P2EP protocol. + raise JMProtocolError("Taker received unexpected JMTXReceived") + return self.on_p2ep_tx_received(nick, txhex) + def get_offers(self): d = self.callRemote(commands.JMRequestOffers) self.defaultCallbacks(d) - def make_tx(self, nick_list, txhex): - d = self.callRemote(commands.JMMakeTx, - nick_list= json.dumps(nick_list), - txhex=txhex) - self.defaultCallbacks(d) - def push_tx(self, nick_to_push, txhex_to_push): d = self.callRemote(commands.JMPushTx, nick=str(nick_to_push), txhex=str(txhex_to_push)) @@ -470,21 +517,26 @@ class JMClientProtocolFactory(protocol.ClientFactory): def buildProtocol(self, addr): return self.protocol(self, self.client) -def start_reactor(host, port, factory, ish=True, daemon=False, rs=True, gui=False): #pragma: no cover +def start_reactor(host, port, factory, ish=True, daemon=False, rs=True, + gui=False, p2ep=False): #pragma: no cover #(Cannot start the reactor in tests) #Not used in prod (twisted logging): #startLogging(stdout) usessl = True if jm_single().config.get("DAEMON", "use_ssl") != 'false' else False if daemon: try: - from jmdaemon import JMDaemonServerProtocolFactory, start_daemon + from jmdaemon import JMDaemonServerProtocolFactory, start_daemon,\ + P2EPDaemonServerProtocolFactory except ImportError: jlog.error("Cannot start daemon without jmdaemon package; " "either install it, and restart, or, if you want " "to run the daemon separately, edit the DAEMON " "section of the config. Quitting.") return - dfactory = JMDaemonServerProtocolFactory() + if not p2ep: + dfactory = JMDaemonServerProtocolFactory() + else: + dfactory = P2EPDaemonServerProtocolFactory() orgport = port while True: try: diff --git a/jmclient/jmclient/configure.py b/jmclient/jmclient/configure.py index 367c0d6..35a01dc 100644 --- a/jmclient/jmclient/configure.py +++ b/jmclient/jmclient/configure.py @@ -175,9 +175,15 @@ unconfirm_timeout_sec = 180 confirm_timeout_hours = 6 [POLICY] -#Use segwit style wallets and transactions +# Use segwit style wallets and transactions +# Only set to false for old wallets, Joinmarket is now segwit only. segwit = true +# Use native segwit (bech32) wallet. This is NOT +# currently supported in Joinmarket coinjoins. Only set to "true" +# if specifically advised to do so. +native = false + # for dust sweeping, try merge_algorithm = gradual # for more rapid dust sweeping, try merge_algorithm = greedy # for most rapid dust sweeping, try merge_algorithm = greediest @@ -352,7 +358,12 @@ def validate_address(addr): try: assert len(addr) > 2 if addr[:2].lower() in ['bc', 'tb']: - #Enforce testnet/mainnet per config + # Regtest special case + if addr[:4] == 'bcrt': + if btc.bech32addr_decode('bcrt', addr)[1]: + return True, 'address validated' + return False, 'Invalid bech32 regtest address' + #Else, enforce testnet/mainnet per config if get_network() == "testnet": hrpreq = 'tb' else: @@ -526,3 +537,8 @@ def get_blockchain_interface_instance(_config): def is_segwit_mode(): return jm_single().config.get('POLICY', 'segwit') != 'false' + +def is_native_segwit_mode(): + if not is_segwit_mode(): + return False + return jm_single().config.get('POLICY', 'native') != 'false' \ No newline at end of file diff --git a/jmclient/jmclient/cryptoengine.py b/jmclient/jmclient/cryptoengine.py index e51cde5..3a434f3 100644 --- a/jmclient/jmclient/cryptoengine.py +++ b/jmclient/jmclient/cryptoengine.py @@ -11,7 +11,7 @@ import jmbitcoin as btc from .configure import get_network -TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH = range(3) +TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, TYPE_P2SH_M_N = range(4) NET_MAINNET, NET_TESTNET = range(2) NET_MAP = {'mainnet': NET_MAINNET, 'testnet': NET_TESTNET} WIF_PREFIX_MAP = {'mainnet': b'\x80', 'testnet': b'\xef'} diff --git a/jmclient/jmclient/maker.py b/jmclient/jmclient/maker.py index e8f9c1a..67f444b 100644 --- a/jmclient/jmclient/maker.py +++ b/jmclient/jmclient/maker.py @@ -2,9 +2,10 @@ from __future__ import (absolute_import, division, print_function, unicode_literals) from builtins import * # noqa: F401 - +from future.utils import iteritems import base64 import pprint +import random import sys import abc from binascii import unhexlify @@ -12,11 +13,12 @@ from binascii import unhexlify from jmbitcoin import SerializationError, SerializationTruncationError import jmbitcoin as btc +from jmclient.wallet import estimate_tx_fee, make_shuffled_tx from jmclient.configure import jm_single from jmbase.support import get_log -from jmclient.support import (calc_cj_fee) +from jmclient.support import calc_cj_fee, select_one_utxo from jmclient.podle import verify_podle, PoDLE, PoDLEError -from twisted.internet import task +from twisted.internet import task, reactor from .cryptoengine import EngineError jlog = get_log() @@ -284,3 +286,374 @@ class Maker(object): """Performs actions on receipt of 1st confirmation of a transaction into a block (e.g. announce orders) """ + +class P2EPMaker(Maker): + """ The P2EP Maker object is instantiated for a specific payment, + with a specific address and expected payment amount. It inherits + normal Maker behaviour on startup and makes fake offers, which + it does not follow up in direct peer interaction (to be specific: + `!fill` requests in privmsg are simply ignored). Under the hood, + the daemon protocol will allow pubkey exchange with any counterparty, + but only after the Taker makes a !tx proposal matching our intended + address and payment amount, which were agreed out of band with the + sender(Taker) counterparty, do we pass over our intended inputs + and partially signed transaction, thus information leak to snoopers + is not possible. + """ + def __init__(self, wallet, mixdepth, amount): + self.receiving_amount = amount + self.mixdepth = mixdepth + # destination mixdepth must be different from that + # which we source coins from; use the standard "next" + dest_mixdepth = (self.mixdepth + 1) % wallet.max_mixdepth + # Select an unused destination in the external branch + self.destination_addr = wallet.get_new_addr(dest_mixdepth, 0) + super(P2EPMaker, self).__init__(wallet) + # Callback to request user permission (for e.g. GUI) + # args: (1) message, as string + # returns: True or False + self.user_check = self.default_user_check + self.user_info = self.default_user_info_callback + + def default_user_check(self, message): + if input(message) == 'y': + return True + return False + + def default_user_info_callback(self, message): + """ TODO this is basically the same function + as taker_info_callback (currently used for GUI); + fold this and some other convenience functions together + and use a root CJPeer class in jmbase to avoid code + duplication. + """ + jlog.info(message) + + def inform_user_details(self): + self.user_info("Your receiving address is: " + self.destination_addr) + self.user_info("You will receive amount: " + str( + self.receiving_amount) + " satoshis.") + self.user_info("The sender also needs to know your ephemeral " + "nickname: " + jm_single().nickname) + self.user_info("This information has been stored in a file payjoin.txt;" + " send it to your counterparty when you are ready.") + with open("payjoin.txt", "w") as f: + f.write("Payjoin transfer details:\n\n") + f.write("Address: " + self.destination_addr + "\n") + f.write("Amount (in sats): " + str(self.receiving_amount) + "\n") + f.write("Receiver nick: " + jm_single().nickname + "\n") + if not self.user_check("Enter 'y' to wait for the payment:"): + sys.exit(0) + + def create_my_orders(self): + """ Fake offer for public consumption. + Requests to fill will be ignored. + """ + ordertype = random.choice(("swreloffer", "swabsoffer")) + minsize = random.randint(100000, 10000000) + maxsize = random.randint(100000, 1000000000) + minsize + txfee = random.randint(0, 1000) + if ordertype == "swreloffer": + cjfee = str(random.randint(0, 100000)/100000000.0) + else: + cjfee = random.randint(0, 10000) + order = {'oid': 0, + 'ordertype': ordertype, + 'minsize': minsize, + 'maxsize': maxsize, + 'txfee': txfee, + 'cjfee': cjfee} + + # sanity check + assert order['minsize'] >= 0 + assert order['maxsize'] > 0 + if order['minsize'] > order['maxsize']: + jlog.info('minsize (' + str(order['minsize']) + ') > maxsize (' + str( + order['maxsize']) + ')') + return [] + + return [order] + + def oid_to_order(self, offer, amount): + # unreachable; only here to satisy abc. + pass + + def on_tx_unconfirmed(self, txd, txid): + self.user_info("The transaction has been broadcast.") + self.user_info("Txid is: " + txid) + self.user_info("Transaction in detail: " + pprint.pformat(txd)) + self.user_info("shutting down.") + reactor.stop() + + def on_tx_confirmed(self, offer, confirmations, txid): + # will not be reached except in testing + self.on_tx_unconfirmed(offer, confirmations) + + def on_tx_received(self, nick, txhex): + """ Called when the sender-counterparty has sent a transaction proposal. + 1. First we check for the expected destination and amount (this is + sufficient to identify our cp, as this info was presumably passed + out of band, as for any normal payment). + 2. Then we verify the validity of the proposed non-coinjoin + transaction; if not, reject, otherwise store this as a + fallback transaction in case the protocol doesn't complete. + 3. Next, we select utxos from our wallet, to add into the + payment transaction as input. Try to select so as to not + trigger the UIH2 condition, but continue (and inform user) + even if we can't (if we can't select any coins, broadcast the + non-coinjoin payment, if the user agrees). + Proceeding with payjoin: + 4. We update the output amount at the destination address. + 5. We modify the change amount in the original proposal (which + will be the only other output other than the destination), + reducing it to account for the increased transaction fee + caused by our additional proposed input(s). + 6. Finally we sign our own input utxo(s) and re-serialize the + tx, allowing it to be sent back to the counterparty. + 7. If the transaction is not fully signed and broadcast within + the time unconfirm_timeout_sec as specified in the joinmarket.cfg, + we broadcast the non-coinjoin fallback tx instead. + """ + try: + tx = btc.deserialize(txhex) + except (IndexError, SerializationError, SerializationTruncationError) as e: + return (False, 'malformed txhex. ' + repr(e)) + self.user_info('obtained tx proposal from sender:\n' + pprint.pformat(tx)) + + if len(tx["outs"]) != 2: + return (False, "Transaction has more than 2 outputs; not supported.") + dest_found = False + destination_index = -1 + change_index = -1 + proposed_change_value = 0 + for index, out in enumerate(tx["outs"]): + if out["script"] == btc.address_to_script(self.destination_addr): + # we found the expected destination; is the amount correct? + if not out["value"] == self.receiving_amount: + return (False, "Wrong payout value in proposal from sender.") + dest_found = True + destination_index = index + else: + change_found = True + proposed_change_out = out["script"] + proposed_change_value = out["value"] + change_index = index + + if not dest_found: + return (False, "Our expected destination address was not found.") + + # Verify valid input utxos provided and check their value. + # batch retrieval of utxo data + utxo = {} + ctr = 0 + for index, ins in enumerate(tx['ins']): + utxo_for_checking = ins['outpoint']['hash'] + ':' + str(ins[ + 'outpoint']['index']) + utxo[ctr] = [index, utxo_for_checking] + ctr += 1 + + utxo_data = jm_single().bc_interface.query_utxo_set( + [x[1] for x in utxo.values()]) + + total_sender_input = 0 + for i, u in iteritems(utxo): + if utxo_data[i] is None: + return (False, "Proposed transaction contains invalid utxos") + total_sender_input += utxo_data[i]["value"] + + # Check that the transaction *as proposed* balances; check that the + # included fee is within 0.3-3x our own current estimates, if not user + # must decide. + btc_fee = total_sender_input - self.receiving_amount - proposed_change_value + self.user_info("Network transaction fee is: " + str(btc_fee) + " satoshis.") + fee_est = estimate_tx_fee(len(tx['ins']), len(tx['outs']), + txtype=self.wallet.get_txtype()) + fee_ok = False + if btc_fee > 0.3 * fee_est and btc_fee < 3 * fee_est: + fee_ok = True + else: + if self.user_check("Is this transaction fee acceptable? (y/n):"): + fee_ok = True + if not fee_ok: + return (False, + "Proposed transaction fee not accepted due to tx fee: " + str( + btc_fee)) + + # This direct rpc call currently assumes Core 0.17, so not using now. + # It has the advantage of (a) being simpler and (b) allowing for any + # non standard coins. + # + #res = jm_single().bc_interface.rpc('testmempoolaccept', [txhex]) + #print("Got this result from rpc call: ", res) + #if not res["accepted"]: + # return (False, "Proposed transaction was rejected from mempool.") + + # Manual verification of the transaction signatures. Passing this + # test does imply that the transaction is valid (unless there is + # a double spend during the process), but is restricted to standard + # types: p2pkh, p2wpkh, p2sh-p2wpkh only. Double spend is not counted + # as a risk as this is a payment. + for i, u in iteritems(utxo): + if "txinwitness" in tx["ins"][u[0]]: + ver_amt = utxo_data[i]["value"] + try: + ver_sig, ver_pub = tx["ins"][u[0]]["txinwitness"] + except Exception as e: + self.user_info("Segwit error: " + repr(e)) + return (False, "Segwit input not of expected type, " + "either p2sh-p2wpkh or p2wpkh") + # note that the scriptCode is the same whether nested or not + # also note that the scriptCode has to be inferred if we are + # only given a transaction serialization. + scriptCode = "76a914" + btc.hash160(unhexlify(ver_pub)) + "88ac" + else: + scriptCode = None + ver_amt = None + scriptSig = btc.deserialize_script(tx["ins"][u[0]]["script"]) + if len(scriptSig) != 2: + return (False, + "Proposed transaction contains unsupported input type") + ver_sig, ver_pub = scriptSig + if not btc.verify_tx_input(txhex, u[0], + utxo_data[i]['script'], + ver_sig, ver_pub, + scriptCode=scriptCode, + amount=ver_amt): + return (False, "Proposed transaction is not correctly signed.") + + # At this point we are satisfied with the proposal. Record the fallback + # in case the sender disappears and the payjoin tx doesn't happen: + self.user_info("We'll use this serialized transaction to broadcast if your" + " counterparty fails to broadcast the payjoin version:") + self.user_info(txhex) + # Keep a local copy for broadcast fallback: + self.fallback_tx = txhex + + # Now we add our own inputs: + # See the gist comment here: + # https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e#gistcomment-2799709 + # which sets out the decision Bob must make. + # In cases where Bob can add any amount, he selects one utxo + # to keep it simple. + # In cases where he must choose at least X, he selects one utxo + # which provides X if possible, otherwise defaults to a normal + # selection algorithm. + # In those cases where he must choose X but X is unavailable, + # he selects all coins, and proceeds anyway with payjoin, since + # it has other advantages (CIOH and utxo defrag). + my_utxos = {} + largest_out = max(self.receiving_amount, proposed_change_value) + max_sender_amt = max([u['value'] for u in utxo_data]) + not_uih2 = False + if max_sender_amt < largest_out: + # just select one coin. + # have some reasonable lower limit but otherwise choose + # randomly; note that this is actually a great way of + # sweeping dust ... + self.user_info("Choosing one coin at random") + try: + my_utxos = self.wallet.select_utxos( + self.mixdepth, jm_single().DUST_THRESHOLD, + select_fn=select_one_utxo) + except: + return self.no_coins_fallback() + not_uih2 = True + else: + # get an approximate required amount assuming 4 inputs, which is + # fairly conservative (but guess by necessity). + fee_for_select = estimate_tx_fee(len(tx['ins']) + 4, 2, + txtype=self.wallet.get_txtype()) + approx_sum = max_sender_amt - largest_out + fee_for_select + try: + my_utxos = self.wallet.select_utxos(self.mixdepth, approx_sum) + not_uih2 = True + except Exception: + # TODO probably not logical to always sweep here. + self.user_info("Sweeping all coins in this mixdepth.") + try: + my_utxos = self.wallet.get_utxos_by_mixdepth()[self.mixdepth] + except: + return self.no_coins_fallback() + if not_uih2: + self.user_info("The proposed tx does not trigger UIH2, which " + "means it is indistinguishable from a normal " + "payment. This is the ideal case. Continuing..") + else: + self.user_info("The proposed tx does trigger UIH2, which it makes " + "it somewhat distinguishable from a normal payment," + " but proceeding with payjoin..") + + my_total_in = sum([va['value'] for va in my_utxos.values()]) + self.user_info("We selected inputs worth: " + str(my_total_in)) + # adjust the output amount at the destination based on our contribution + new_destination_amount = self.receiving_amount + my_total_in + # estimate the required fee for the new version of the transaction + total_ins = len(tx["ins"]) + len(my_utxos.keys()) + est_fee = estimate_tx_fee(total_ins, 2, txtype=self.wallet.get_txtype()) + self.user_info("We estimated a fee of: " + str(est_fee)) + new_change_amount = total_sender_input + my_total_in - \ + new_destination_amount - est_fee + self.user_info("We calculated a new change amount of: " + str(new_change_amount)) + self.user_info("We calculated a new destination amount of: " + str(new_destination_amount)) + # now reconstruct the transaction with the new inputs and the + # amount-changed outputs + new_outs = [{"address": self.destination_addr, + "value": new_destination_amount}] + if new_change_amount >= jm_single().BITCOIN_DUST_THRESHOLD: + new_outs.append({"script": proposed_change_out, + "value": new_change_amount}) + new_ins = [x[1] for x in utxo.values()] + new_ins.extend(my_utxos.keys()) + # set locktime for best anonset (Core, Electrum) - most recent block. + # this call should never fail so no catch here. + currentblock = jm_single().bc_interface.rpc( + "getblockchaininfo", [])["blocks"] + new_tx = make_shuffled_tx(new_ins, new_outs, False, 2, currentblock) + new_tx_deser = btc.deserialize(new_tx) + + # sign our inputs before transfer + our_inputs = {} + for index, ins in enumerate(new_tx_deser['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo not in my_utxos: + continue + script = self.wallet.addr_to_script(my_utxos[utxo]['address']) + amount = my_utxos[utxo]['value'] + our_inputs[index] = (script, amount) + + txs = self.wallet.sign_tx(btc.deserialize(new_tx), our_inputs) + jm_single().bc_interface.add_tx_notify(txs, + self.on_tx_unconfirmed, self.on_tx_confirmed, + self.destination_addr, + wallet_name=jm_single().bc_interface.get_wallet_name(self.wallet), + txid_flag=False, vb=self.wallet._ENGINE.VBYTE) + # The blockchain interface just abandons monitoring if the transaction + # is not broadcast before the configured timeout; we want to take + # action in this case, so we add an additional callback to the reactor: + reactor.callLater(jm_single().config.getint("TIMEOUT", + "unconfirm_timeout_sec"), self.broadcast_fallback) + return (True, nick, btc.serialize(txs)) + + def no_coins_fallback(self): + """ Broadcast, optionally, the fallback non-coinjoin transaction + because we were not able to select coins to contribute. + """ + self.user_info("Unable to select any coins; this mixdepth is empty.") + if self.user_check("Would you like to broadcast the non-coinjoin payment?"): + self.broadcast_fallback() + return (False, "Coinjoin unsuccessful, fallback attempted.") + else: + self.user_info("You chose not to broadcast; the payment has NOT been made.") + return (False, "No transaction made.") + + def broadcast_fallback(self): + self.user_info("Broadcasting non-coinjoin fallback transaction.") + txid = btc.txhash(self.fallback_tx) + success = jm_single().bc_interface.pushtx(self.fallback_tx) + if not success: + self.user_info("ERROR: the fallback transaction did not broadcast. " + "The payment has NOT been made.") + else: + self.user_info("Payment received successfully, but it was NOT a coinjoin.") + self.user_info("Txid: " + txid) + reactor.stop() diff --git a/jmclient/jmclient/support.py b/jmclient/jmclient/support.py index 8507d1a..583e851 100644 --- a/jmclient/jmclient/support.py +++ b/jmclient/jmclient/support.py @@ -2,7 +2,6 @@ from __future__ import (absolute_import, division, print_function, unicode_literals) from builtins import * # noqa: F401 from functools import reduce - import random from jmbase.support import get_log from decimal import Decimal @@ -163,6 +162,10 @@ def select_greediest(unspent, value): end += 1 return low[0:end] +def select_one_utxo(unspent, value): + key = lambda u: u['value'] + return [random.choice([u for u in unspent if key(u) >= value])] + def calc_cj_fee(ordertype, cjfee, cj_amount): if ordertype in ['swabsoffer', 'absoffer']: diff --git a/jmclient/jmclient/taker.py b/jmclient/jmclient/taker.py index 8e34ea6..477cb9f 100644 --- a/jmclient/jmclient/taker.py +++ b/jmclient/jmclient/taker.py @@ -7,14 +7,16 @@ from future.utils import iteritems import base64 import pprint import random +from twisted.internet import reactor from binascii import hexlify, unhexlify +from jmbitcoin import SerializationError, SerializationTruncationError import jmbitcoin as btc from jmclient.configure import get_p2sh_vbyte, jm_single, validate_address from jmbase.support import get_log from jmclient.support import (calc_cj_fee, weighted_order_choose, choose_orders, choose_sweep_orders) -from jmclient.wallet import estimate_tx_fee +from jmclient.wallet import estimate_tx_fee, make_shuffled_tx from jmclient.podle import generate_podle, get_podle_commitments, PoDLE from .output import generate_podle_error_string from .cryptoengine import EngineError @@ -491,9 +493,7 @@ class Taker(object): for u in sum(self.utxos.values(), [])] self.outputs.append({'address': self.coinjoin_address(), 'value': self.cjamount}) - random.shuffle(self.utxo_tx) - random.shuffle(self.outputs) - tx = btc.mktx(self.utxo_tx, self.outputs) + tx = make_shuffled_tx(self.utxo_tx, self.outputs, False) jlog.info('obtained tx\n' + pprint.pformat(btc.deserialize(tx))) self.latest_tx = btc.deserialize(tx) @@ -842,3 +842,344 @@ class Taker(object): return assert hasattr(bci, 'get_wallet_name') bci.import_addresses(addr_list, bci.get_wallet_name(self.wallet)) + +class P2EPTaker(Taker): + """ The P2EP Taker will initialize its protocol directly + with the prescribed counterparty (see -T argument to + sendpayment). It inherits the normal behaviour of requesting + an orderbook on startup, but does nothing with it; this + improves the privacy of the operation. + """ + + def __init__(self, counterparty, wallet, schedule, callbacks): + super(P2EPTaker, self).__init__(wallet, schedule, callbacks=callbacks) + self.p2ep_receiver_nick = counterparty + # Callback to request user permission (for e.g. GUI) + # args: (1) message, as string + # returns: True or False + self.user_check = self.default_user_check + + def default_user_check(self, message): + if input(message) == 'y': + return True + return False + + def register_user_check_callback(self, user_check): + self.user_check = user_check + + def unconfirm_callback(self, txd, txid): + jlog.info("Transaction seen on network, shutting down.") + jlog.info("Txid was: " + txid) + # In P2EP we stop the protocol here. + reactor.stop() + + def confirm_callback(self, txd, txid, confirmations): + # Will never trigger except in testing + self.unconfirm_callback(txd, txid) + + def initialize(self, orderbook): + """ Note that the orderbook parameter is ignored. + Here the schedule data (the standard format for coinjoin + request specification) passes in the amount, destination + and source mixdepth information. We then select coins + using the inherited Taker method to do so. + """ + if self.aborted: + return (False,) + self.taker_info_callback("INFO", "Received offers from joinmarket pit") + # only one schedule item has been allowed; parse from it. + self.schedule_index += 1 + si = self.schedule[0] + self.mixdepth = si[0] + self.cjamount = si[1] + # For the p2ep taker, the variable 'my_cj_addr' is the destination: + self.my_cj_addr = si[3] + if isinstance(self.cjamount, float): + raise JMTakerError("P2EP coinjoin must use amount in satoshis") + if self.cjamount == 0: + # Note that we don't allow sweep, currently, since the coin + # choosing algo would not apply in that case (we'd have to rewrite + # prepare_my_bitcoin_data for that case). + raise JMTakerError("P2EP coinjoin does not currently support sweep") + + # Next we prepare our coins with the inherited method + # for this purpose; for this we must set the + # number of counterparties and fee per cpy, for fee estimation; + # the estimates will be rather rough, but that's fine. Here we + # will end up selecting 20k sats more than the destination amount, + # which will make the already ultra-rare edge case of not selecting + # enough for fees, even more rare. "Stuck" coins due to edge cases + # are not an issue since the wallet has direct-send sweep. + self.n_counterparties = 1 + self.txfee_default = 10000 + self.total_cj_fee = 0 + # Preparing bitcoin data here includes choosing utxos/coins. + # We don't trust the user on the selection algo choice; we want it + # to be fairly greedy for technical reasons explained in the comment + # thread to this gist: + # https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e + jm_single().config.set("POLICY", "merge_algorithm", "greedy") + self.noncj_fee_est = 0 + if not self.prepare_my_bitcoin_data(): + return (False, ) + self.outputs = [] + self.cjfee_total = 0 + self.txfee_default = 5000 + self.latest_tx = None + self.txid = None + + # we return dummy values for commitment and revelation, + # and the offer dict only signals the nick of the counterparty. + return (True, self.cjamount, "p2ep", "p2ep", {self.p2ep_receiver_nick:{}}) + + def receive_utxos(self, ioauth_data): + """ TODO this function is misnamed for the + purpose of code reuse; fix it(e.g. 'make_tx_proposal'). + + The ioauth_data field will be a list containing the single + nick which we intend to send to (which fact we should check), + and then construct a transaction with the intended amount + to the intended destination. This transaction will not, in + normal operation, be broadcast; because the Maker (receiver) + will add his own inputs and change the total to be received at + that address. Thus we are functionally only sending our own input + utxos - the signed tx may be used as a fallback by the counterparty + in case we disappear - however this also serves the purpose of + signalling that we are the right counterparty. + """ + if not ioauth_data[0] == self.p2ep_receiver_nick: + return (False, "Wrong counterparty IRC nick: " + ioauth_data[0]) + # Transaction construction: use inputs as per `prepare_my_bitcoin_data`, + # use output destination self.my_cj_addr and use amount self.amount + self.outputs.append({'address': self.my_cj_addr, + 'value': self.cjamount}) + my_total_in = sum([va['value'] for u, va in iteritems(self.input_utxos)]) + # estimate the fee for the version of the transaction which is + # not coinjoined: + est_fee = estimate_tx_fee(len(self.input_utxos), 2, + txtype=self.wallet.get_txtype()) + my_change_value = my_total_in - self.cjamount - est_fee + if my_change_value <= 0: + # as discussed in initialize(), this should be an extreme edge case. + raise ValueError("Wallet utxo selection chose too few coins") + elif self.my_change_addr and my_change_value <= jm_single( + ).BITCOIN_DUST_THRESHOLD: + jlog.info("Dynamically calculated change lower than dust: " + str( + my_change_value) + "; dropping.") + self.my_change_addr = None + my_change_value = 0 + # Note that the sweep case (my_change_addr is None, but not due to dust) + # is not currently allowed here. + if self.my_change_addr is not None: + self.outputs.append({'address': self.my_change_addr, + 'value': my_change_value}) + # set locktime for best anonset (Core, Electrum) - most recent block. + # this call should never fail so no catch here. + currentblock = jm_single().bc_interface.rpc( + "getblockchaininfo", [])["blocks"] + # As for JM coinjoins, the `None` key is used for our own inputs + # to the transaction; this preparatory version contains only those. + tx = make_shuffled_tx(self.utxos[None], self.outputs, + False, 2, currentblock) + jlog.info('Obtained proposed payjoin tx\n' + pprint.pformat( + btc.deserialize(tx))) + # We now sign as a courtesy, because if we disappear the recipient + # can still claim his coins with this. + # sign our inputs before transfer + our_inputs = {} + dtx = btc.deserialize(tx) + for index, ins in enumerate(dtx['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + script = self.wallet.addr_to_script(self.input_utxos[utxo]['address']) + amount = self.input_utxos[utxo]['value'] + our_inputs[index] = (script, amount) + self.signed_noncj_tx = btc.serialize(self.wallet.sign_tx(dtx, our_inputs)) + self.taker_info_callback("INFO", "Built tx proposal, sending to receiver.") + return (True, [self.p2ep_receiver_nick], self.signed_noncj_tx) + + def on_tx_received(self, nick, txhex): + """ Here the taker (payer) retrieves a version of the + transaction from the maker (receiver) which should have + the following properties: + * Destination address as previously agreed. + * Our correct change output with amount corresponding to fee. + * Net of (destination amount) - (receiver input amount) + must be equal to the original amount self.cjamount. + * Our inputs must be unchanged from original proposal. + * Counterparty should not provide more than 5 utxos; this + is a crude avoidance of over-paying fees, but note that + the maker selection should mean this almost never happens. + * Counterparties' transaction signatures must be valid. + If all conditions are met, we sign each of our inputs + and then broadcast (TODO broadcast delay or don't broadcast). + """ + try: + tx = btc.deserialize(txhex) + except (IndexError, SerializationError, SerializationTruncationError) as e: + return (False, "malformed txhex. " + repr(e)) + jlog.info("Obtained tx from receiver:\n" + pprint.pformat(tx)) + cjaddr_script = btc.address_to_script(self.my_cj_addr) + changeaddr_script = btc.address_to_script(self.my_change_addr) + + # We ensure that the coinjoin address and our expected change + # address are still in the outputs, once (with the caveat that + # the change address is allowed to be absent in a special case + # of dust change, which we assess after). + times_seen_cj_addr = 0 + times_seen_change_addr = 0 + for outs in tx['outs']: + if outs['script'] == cjaddr_script: + times_seen_cj_addr += 1 + new_cj_amount = outs['value'] + if new_cj_amount < self.cjamount: + # This is a violation of protocol; + # receiver must be providing extra bitcoin + # as input, so his receiving amount should have increased. + return (False, + 'Wrong cj_amount. I expect at least' + str(self.cjamount)) + if outs['script'] == changeaddr_script: + times_seen_change_addr += 1 + new_change_amount = outs['value'] + if times_seen_cj_addr != 1: + fmt = ('cj addr not in tx outputs once, #cjaddr={}').format + return (False, (fmt(times_seen_cj_addr))) + if times_seen_change_addr != 1: + if times_seen_change_addr > 1: + return (False, "proposed tx has change address duplicated") + # Otherwise change has been ditched; will check this later. + new_change_amount = 0 + + # Check that our inputs are present. + tx_utxo_set = set(ins['outpoint']['hash'] + ':' + str( + ins['outpoint']['index']) for ins in tx['ins']) + if not tx_utxo_set.issuperset(set(self.utxos[None])): + return (False, "my utxos are not contained") + + # Before even starting fee calculations, reject > 5 + # inputs from counterparty as an abuse (accidental or + # not) of PayJoin to sweep utxos at no cost. + # (TODO This is very kludgy, more sophisticated approach + # should be used in future): + if len(tx["ins"]) - len (self.utxos[None]) > 5: + return (False, + "proposed tx has more than 5 inputs from " + "the recipient, which is too expensive.") + + # If we ignored fees, we would only need to check that + # the difference between our inputs and outputs was equal + # to the expected payment; but this difference will include + # the bitcoin transaction fee. + # Hence, we retrieve the counterparty's input amount, + # and find the overall bitcoin network fee, and decide + # from this whether the change value is as expected + # (our inputs - expected payment - network fee); + # and if that is dusty, agree to sign without change. + + # batch retrieval of utxo data; we collect the utxos + # in the inputs which do not belong to us, and put + # them into a dict (`retrieve_utxos`), keyed by their + # index in the inputs, so we can use the collected + # script and amount data when we do the next stages, + # checking input validity and transaction balance. + retrieve_utxos = {} + ctr = 0 + for index, ins in enumerate(tx['ins']): + utxo_for_checking = ins['outpoint']['hash'] + ':' + str( + ins['outpoint']['index']) + if utxo_for_checking in self.utxos[None]: + continue + retrieve_utxos[ctr] = [index, utxo_for_checking] + ctr += 1 + # we always accept unconf utxos from receiver; it's their payment: + utxo_data = jm_single().bc_interface.query_utxo_set( + [x[1] for x in retrieve_utxos.values()], includeunconf=True) + + # Next we'll verify each of the counterparty's inputs, + # while at the same time gathering the total they spent. + total_receiver_input = 0 + for i, u in iteritems(retrieve_utxos): + if utxo_data[i] is None: + return (False, "Proposed transaction contains invalid utxos") + total_receiver_input += utxo_data[i]["value"] + scriptCode = None + ver_amt = None + idx = retrieve_utxos[i][0] + if "txinwitness" in tx["ins"][idx]: + ver_amt = utxo_data[i]["value"] + try: + ver_sig, ver_pub = tx["ins"][idx]["txinwitness"] + except Exception as e: + print("Segwit error: ", repr(e)) + return (False, "Segwit input not of expected type, " + "either p2sh-p2wpkh or p2wpkh") + # note that the scriptCode is the same whether nested or not + # also note that the scriptCode has to be inferred if we are + # only given a transaction serialization. + scriptCode = "76a914" + btc.hash160(unhexlify(ver_pub)) + "88ac" + else: + scriptSig = btc.deserialize_script(tx["ins"][idx]["script"]) + if len(scriptSig) != 2: + return (False, + "Proposed transaction contains unsupported input type") + ver_sig, ver_pub = scriptSig + if not btc.verify_tx_input(txhex, idx, + utxo_data[i]['script'], + ver_sig, ver_pub, + scriptCode=scriptCode, + amount=ver_amt): + return (False, + "Proposed transaction is not correctly signed.") + payment = new_cj_amount - total_receiver_input + if payment != self.cjamount: + return (False, "Proposed transaction has wrong payment amount: " +\ + str(payment) + ", should be: " + str(self.cjamount)) + # reminder: the keys of the input_utxos dict == self.utxos[None] + total_sender_input = sum([va['value'] for va in self.input_utxos.values()]) + # check full transaction balance + btc_fee = total_receiver_input + total_sender_input - new_cj_amount - new_change_amount + self.taker_info_callback("INFO", + "Network transaction fee is: " + str(btc_fee) + " satoshis.") + if btc_fee <= 0: + return (False, "Proposed transaction has no bitcoin fee") + # To validate the fee, we need to check the size, but this can only be estimated + # until it's fully signed; we now know the number of inputs, so we can use + # our fee estimator. Its return value will be governed by our own fee settings + # in joinmarket.cfg; allow either (a) automatic agreement for any value within + # a range of 0.3 to 3x this figure, or (b) user to agree on prompt. + fee_est = estimate_tx_fee(len(tx['ins']), len(tx['outs']), + txtype=self.wallet.get_txtype()) + fee_ok = False + if btc_fee > 0.3 * fee_est and btc_fee < 3 * fee_est: + fee_ok = True + else: + if self.user_check("Is this transaction fee acceptable? (y/n):"): + fee_ok = True + if not fee_ok: + return (False, + "Proposed transaction fee not accepted due to tx fee: " + str( + btc_fee)) + + self.total_txfee = btc_fee + # now that the fee is known and accepted, we can check our change + if new_change_amount == 0: + # calculate what the change would be, after subtracting the agreed fee; + # if it's dusty, then we continue with no change; otherwise we prompt/ + # reject. + hyp_change_amount = total_receiver_input - self.cjamount - btc_fee + if hyp_change_amount <= jm_single().BITCOIN_DUST_THRESHOLD: + jlog.info("Counterparty correctly removed dusty change value:"\ + + str(hyp_change_amount)) + else: + jlog.info(('WARNING CHANGE NOT BEING ' + 'USED\nCHANGEVALUE = {}').format( + hyp_change_amount)) + if not self.user_check("OK to broadcast with this change spent " + "to miner fee? (y/n):"): + return (False, "Proposed transaction not accepted due to " + "absent change.") + + # All checks have passed; we sign and broadcast + self.latest_tx = tx + self.self_sign_and_push() + # returning False here is not an error condition, only stops processing. + return (False, "OK") diff --git a/jmclient/jmclient/taker_utils.py b/jmclient/jmclient/taker_utils.py index eb931d4..516d4e9 100644 --- a/jmclient/jmclient/taker_utils.py +++ b/jmclient/jmclient/taker_utils.py @@ -7,7 +7,6 @@ import pprint import os import time import numbers -from binascii import unhexlify from jmbase import get_log, jmprint from .configure import jm_single, validate_address from .schedule import human_readable_schedule_entry, tweak_tumble_schedule,\ @@ -113,7 +112,7 @@ def sign_tx(wallet, tx, utxos): script = wallet.addr_to_script(utxos[utxo]['address']) amount = utxos[utxo]['value'] our_inputs[index] = (script, amount) - return wallet.sign_tx(deserialize(unhexlify(serialize(stx))), our_inputs) + return wallet.sign_tx(stx, our_inputs) def import_new_addresses(wallet, addr_list): # FIXME: same code as in maker.py and taker.py diff --git a/jmclient/jmclient/wallet.py b/jmclient/jmclient/wallet.py index 6578266..0152f7d 100644 --- a/jmclient/jmclient/wallet.py +++ b/jmclient/jmclient/wallet.py @@ -4,6 +4,7 @@ from builtins import * # noqa: F401 from configparser import NoOptionError import warnings +import random import functools import collections import numbers @@ -66,6 +67,18 @@ class Mnemonic(MnemonicParent): def detect_language(cls, code): return "english" +def make_shuffled_tx(ins, outs, deser=True, version=1, locktime=0): + """ Simple utility to ensure transaction + inputs and outputs are randomly ordered. + Can possibly be replaced by BIP69 in future + """ + random.shuffle(ins) + random.shuffle(outs) + tx = btc.mktx(ins, outs, version=version, locktime=locktime) + if deser: + return btc.deserialize(tx) + else: + return tx def estimate_tx_fee(ins, outs, txtype='p2pkh'): '''Returns an estimate of the number of satoshis required @@ -82,9 +95,9 @@ def estimate_tx_fee(ins, outs, txtype='p2pkh'): if txtype in ['p2pkh', 'p2shMofN']: tx_estimated_bytes = btc.estimate_tx_size(ins, outs, txtype) return int((tx_estimated_bytes * fee_per_kb)/Decimal(1000.0)) - elif txtype=='p2sh-p2wpkh': + elif txtype in ['p2wpkh', 'p2sh-p2wpkh']: witness_estimate, non_witness_estimate = btc.estimate_tx_size( - ins, outs, 'p2sh-p2wpkh') + ins, outs, txtype) return int(int(( non_witness_estimate + 0.25*witness_estimate)*fee_per_kb)/Decimal(1000.0)) else: @@ -168,12 +181,13 @@ class UTXOManager(object): self._utxo[mixdepth][(txid, index)] = (path, value) - def select_utxos(self, mixdepth, amount, utxo_filter=()): + def select_utxos(self, mixdepth, amount, utxo_filter=(), select_fn=None): assert isinstance(mixdepth, numbers.Integral) utxos = self._utxo[mixdepth] available = [{'utxo': utxo, 'value': val} for utxo, (addr, val) in utxos.items() if utxo not in utxo_filter] - selected = self.selector(available, amount) + selector = select_fn or self.selector + selected = selector(available, amount) return {s['utxo']: {'path': utxos[s['utxo']][0], 'value': utxos[s['utxo']][1]} for s in selected} @@ -315,6 +329,8 @@ class BaseWallet(object): return 'p2pkh' elif self.TYPE == TYPE_P2SH_P2WPKH: return 'p2sh-p2wpkh' + elif self.TYPE == TYPE_P2WPKH: + return 'p2wpkh' assert False def sign_tx(self, tx, scripts, **kwargs): @@ -549,12 +565,12 @@ class BaseWallet(object): self._utxos.add_utxo(txid, index, path, value, mixdepth) @deprecated - def select_utxos(self, mixdepth, amount, utxo_filter=None): + def select_utxos(self, mixdepth, amount, utxo_filter=None, select_fn=None): utxo_filter_new = None if utxo_filter: utxo_filter_new = [(unhexlify(utxo[:64]), int(utxo[65:])) for utxo in utxo_filter] - ret = self.select_utxos_(mixdepth, amount, utxo_filter_new) + ret = self.select_utxos_(mixdepth, amount, utxo_filter_new, select_fn) ret_conv = {} for utxo, data in ret.items(): addr = self.get_addr_path(data['path']) @@ -562,7 +578,8 @@ class BaseWallet(object): ret_conv[utxo_txt] = {'address': addr, 'value': data['value']} return ret_conv - def select_utxos_(self, mixdepth, amount, utxo_filter=None): + def select_utxos_(self, mixdepth, amount, utxo_filter=None, + select_fn=None): """ Select a subset of available UTXOS for a given mixdepth whose value is greater or equal to amount. @@ -585,7 +602,8 @@ class BaseWallet(object): assert len(i) == 2 assert isinstance(i[0], bytes) assert isinstance(i[1], numbers.Integral) - ret = self._utxos.select_utxos(mixdepth, amount, utxo_filter) + ret = self._utxos.select_utxos( + mixdepth, amount, utxo_filter, select_fn) for data in ret.values(): data['script'] = self.get_script_path(data['path']) diff --git a/jmclient/jmclient/wallet_utils.py b/jmclient/jmclient/wallet_utils.py index bf9f81f..59682cd 100644 --- a/jmclient/jmclient/wallet_utils.py +++ b/jmclient/jmclient/wallet_utils.py @@ -14,10 +14,10 @@ from collections import Counter from itertools import islice from jmclient import (get_network, WALLET_IMPLEMENTATIONS, Storage, podle, jm_single, BitcoinCoreInterface, JsonRpcError, sync_wallet, WalletError, - VolatileStorage, StoragePasswordError, - is_segwit_mode, SegwitLegacyWallet, LegacyWallet) + VolatileStorage, StoragePasswordError, is_segwit_mode, SegwitLegacyWallet, + LegacyWallet, SegwitWallet, is_native_segwit_mode) from jmbase.support import get_password, jmprint -from .cryptoengine import TYPE_P2PKH, TYPE_P2SH_P2WPKH +from .cryptoengine import TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH import jmbitcoin as btc @@ -897,6 +897,8 @@ def wallet_signmessage(wallet, hdpath, message): def get_wallet_type(): if is_segwit_mode(): + if is_native_segwit_mode(): + return TYPE_P2WPKH return TYPE_P2SH_P2WPKH return TYPE_P2PKH @@ -937,6 +939,10 @@ def open_test_wallet_maybe(path, seed, max_mixdepth, returns: wallet object """ + # If the native flag is set in the config, it overrides the argument + # test_wallet_cls + if jm_single().config.get("POLICY", "native") == "true": + test_wallet_cls = SegwitWallet if len(seed) == test_wallet_cls.ENTROPY_BYTES * 2: try: seed = binascii.unhexlify(seed) diff --git a/jmclient/test/test_payjoin.py b/jmclient/test/test_payjoin.py new file mode 100644 index 0000000..e068666 --- /dev/null +++ b/jmclient/test/test_payjoin.py @@ -0,0 +1,163 @@ +from __future__ import (absolute_import, division, + print_function, unicode_literals) +from builtins import * # noqa: F401 + +""" +Test doing payjoin joins (with message channel layer mocked) +""" + +import os +import sys +import pytest +from twisted.internet import reactor +from jmbase import get_log +from jmclient import cryptoengine +from jmclient import (load_program_config, jm_single, sync_wallet, + P2EPMaker, P2EPTaker, + LegacyWallet, SegwitLegacyWallet, SegwitWallet) +from commontest import make_wallets +from test_coinjoin import make_wallets_to_list, sync_wallets, create_orderbook + +testdir = os.path.dirname(os.path.realpath(__file__)) +log = get_log() + +def create_taker(wallet, schedule, monkeypatch): + taker = P2EPTaker("fakemaker", wallet, schedule, + callbacks=(None, None, None)) + return taker + +def dummy_user_check(message): + # No user interaction in test; just print message + # and assume acceptance. + log.info(message) + return True + +def getbals(wallet, mixdepth): + """ Retrieves balances for a mixdepth and the 'next' + """ + bbm = wallet.get_balance_by_mixdepth() + return (bbm[mixdepth], bbm[(mixdepth + 1) % (wallet.mixdepth + 1)]) + +def final_checks(wallets, amount, txfee, tsb, msb, source_mixdepth=0): + """We use this to check that the wallet contents are + as we've expected according to the test case. + amount is the payment amount going from taker to maker. + txfee is the bitcoin network transaction fee, paid by the + taker. + tsb, msb are taker and maker starting balances, each a tuple + of two entries, source and destination mixdepth respectively. + """ + jm_single().bc_interface.tickchain() + for wallet in wallets: + sync_wallet(wallet) + takerbals = getbals(wallets[1], source_mixdepth) + makerbals = getbals(wallets[0], source_mixdepth) + # is the payment received? + maker_newcoin_amt = makerbals[1] - msb[1] + if not maker_newcoin_amt >= amount: + print("Maker expected to receive at least: ", amount, + " but got: ", maker_newcoin_amt) + return False + # assert that the maker received net exactly the right amount + maker_spentcoin_amt = msb[0] - makerbals[0] + if not maker_spentcoin_amt >= 0: + # for now allow the non-cj fallback case + print("maker's spent coin should have been positive, was: ", maker_spentcoin_amt) + return False + if not maker_newcoin_amt == amount + maker_spentcoin_amt: + print("maker's new coins should have been: ", amount + maker_spentcoin_amt, + " but was: ", maker_newcoin_amt) + return False + + # Taker-side check + # assert that the taker's total ending minus total starting + # balance is the amount plus the txfee given. + if not (sum(takerbals) - sum(tsb) + txfee + amount) == 0: + print("Taker should have spent: ", txfee + amount, + " but spent: ", sum(tsb) - sum(takerbals)) + return False + return True + +@pytest.mark.parametrize('wallet_cls, wallet_structures, mean_amt', + [([LegacyWallet, LegacyWallet], + [[4, 0, 0, 0, 0]] * 2, 1.0), + ([SegwitLegacyWallet, SegwitLegacyWallet], + [[1, 3, 0, 0, 0]] * 2, 2.0), + ([SegwitWallet, SegwitWallet], + [[1, 0, 0, 0, 0]] * 2, 4.0), + ([LegacyWallet, SegwitWallet], + [[4, 0, 0, 0, 0]] * 2, 1.0), + ([SegwitLegacyWallet, SegwitWallet], + [[1, 3, 0, 0, 0]] * 2, 2.0), + ([SegwitWallet, SegwitLegacyWallet], + [[1, 0, 0, 0, 0]] * 2, 4.0), + ]) +def test_simple_payjoin(monkeypatch, tmpdir, setup_cj, wallet_cls, + wallet_structures, mean_amt): + def raise_exit(i): + raise Exception("sys.exit called") + monkeypatch.setattr(sys, 'exit', raise_exit) + wallets = [] + wallets.append(make_wallets_to_list(make_wallets( + 1, wallet_structures=[wallet_structures[0]], + mean_amt=mean_amt, wallet_cls=wallet_cls[0]))[0]) + wallets.append(make_wallets_to_list(make_wallets( + 1, wallet_structures=[wallet_structures[1]], + mean_amt=mean_amt, wallet_cls=wallet_cls[1]))[0]) + jm_single().bc_interface.tickchain() + sync_wallets(wallets) + + # For accounting purposes, record the balances + # at the start. + msb = getbals(wallets[0], 0) + tsb = getbals(wallets[1], 0) + + cj_amount = int(1.1 * 10**8) + maker = P2EPMaker(wallets[0], 0, cj_amount) + destaddr = maker.destination_addr + monkeypatch.setattr(maker, 'user_check', dummy_user_check) + # TODO use this to sanity check behaviour + # in presence of the rest of the joinmarket orderbook. + orderbook = create_orderbook([maker]) + assert len(orderbook) == 1 + # mixdepth, amount, counterparties, dest_addr, waittime; + # in payjoin we only pay attention to the first two entries. + schedule = [(0, cj_amount, 1, destaddr, 0)] + taker = create_taker(wallets[-1], schedule, monkeypatch) + monkeypatch.setattr(taker, 'user_check', dummy_user_check) + init_data = taker.initialize(orderbook) + # the P2EPTaker.initialize() returns: + # (True, self.cjamount, "p2ep", "p2ep", {self.p2ep_receiver_nick:{}}) + assert init_data[0], "taker.initialize error" + active_orders = init_data[4] + assert len(active_orders.keys()) == 1 + response = taker.receive_utxos(list(active_orders.keys())) + assert response[0], "taker receive_utxos error" + # test for validity of signed fallback transaction; requires 0.17; + # note that we count this as an implicit test of fallback mode. + res = jm_single().bc_interface.rpc('testmempoolaccept', [[response[2]]]) + assert res[0]["allowed"], "Proposed transaction was rejected from mempool." + maker_response = maker.on_tx_received("faketaker", response[2]) + if not maker_response[0]: + print("maker on_tx_received failed, reason: ", maker_response[1]) + assert False + taker_response = taker.on_tx_received("fakemaker", maker_response[2]) + if not taker_response[1] == "OK": + print("Failure in taker on_tx_received, reason: ", taker_response[1]) + assert False + # Although the above OK is proof that a transaction went through, + # it doesn't prove it was a good transaction! Here do balance checks: + assert final_checks(wallets, cj_amount, taker.total_txfee, tsb, msb) + +@pytest.fixture(scope='module') +def setup_cj(): + load_program_config() + jm_single().config.set('POLICY', 'tx_broadcast', 'self') + jm_single().bc_interface.tick_forward_chain_interval = 5 + jm_single().bc_interface.simulate_blocks() + #see note in cryptoengine.py: + cryptoengine.BTC_P2WPKH.VBYTE = 100 + yield None + # teardown + for dc in reactor.getDelayedCalls(): + dc.cancel() diff --git a/jmdaemon/jmdaemon/__init__.py b/jmdaemon/jmdaemon/__init__.py index 996fcbc..d4b3c3f 100644 --- a/jmdaemon/jmdaemon/__init__.py +++ b/jmdaemon/jmdaemon/__init__.py @@ -12,7 +12,7 @@ from .message_channel import MessageChannel, MessageChannelCollection from .orderbookwatch import OrderbookWatch from jmbase import commands from .daemon_protocol import (JMDaemonServerProtocolFactory, JMDaemonServerProtocol, - start_daemon) + start_daemon, P2EPDaemonServerProtocolFactory) from .protocol import (COMMAND_PREFIX, ORDER_KEYS, NICK_HASH_LENGTH, NICK_MAX_ENCODED, JM_VERSION, JOINMARKET_NICK_HEADER) from .message_channel import MessageChannelCollection diff --git a/jmdaemon/jmdaemon/daemon_protocol.py b/jmdaemon/jmdaemon/daemon_protocol.py index 510a4a8..93deee1 100644 --- a/jmdaemon/jmdaemon/daemon_protocol.py +++ b/jmdaemon/jmdaemon/daemon_protocol.py @@ -647,6 +647,153 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch): if self.mcc: self.mcc.shutdown() +class P2EPDaemonServerProtocol(JMDaemonServerProtocol): + @JMFill.responder + def on_JM_FILL(self, amount, commitment, revelation, filled_offers): + """ Takes the setup information for the transaction + from the Taker (p2ep sender) and starts the key negotiation, + which is handled entirely in the daemon. + """ + if not isinstance(amount, Integral) and amount >= 0: + return {'accepted': False} + self.amount = amount + # The counterparty was passed within a offers dict struct; + # we retain the variable `active_orders` for compatibility + # with inherited code, even though it's not really an + # "order" or "offer", and there's only one. + self.active_orders = json.loads(filled_offers) + if len(self.active_orders.keys()) != 1: + return {'accepted': False} + # Here the p2ep communication starts with sending an ephemeral + # pubkey. The version data is appended. + pubkeymsg = self.versioned_pubkey_msg(self.kp.hex_pk().decode('ascii')) + self.mcc.prepare_privmsg(list(self.active_orders.keys())[0], + "pubkey", pubkeymsg) + return {'accepted': True} + + def on_pubkey(self, nick, tmaker_pk): + """This is handled locally in the daemon; set up e2e + encrypted messaging with this counterparty. + Note that this message is used in both directions in p2ep + (as distinct from coinjoins where it's sent Maker->Taker + only). + The pubkey passed in hex and appended with two bytes; + for the proposer (sender) this is the minimum and maximum + of the p2ep version supported. For the receiver these two + bytes must be equal and are equal to the version chosen. + """ + if len(tmaker_pk) < 6: + self.mcc.send_error(nick, "Invalid pubkey message: " + tmaker_pk) + return + try: + version_low, version_high = [int(x, 16) for x in [tmaker_pk[-4:-2], + tmaker_pk[-2:]]] + except ValueError: + self.mcc.send_error(nick, "Invalid pubkey message: " + tmaker_pk) + return + # For both sender and receiver, the received version must be low 0 + if version_low != 0: + self.mcc.send_error(nick, + "Unsupported p2ep version: " + str(version_low)) + return + tmaker_pk = tmaker_pk[:-4] + if self.role == "TAKER": + # For sender, when he receives pubkey message from receiver, it + # must be a choice of version 00 i.e. low, high = 0,0 + if version_high != 0: + self.mcc.send_error(nick, + "Invalid p2ep version string, high: " + str(version_high)) + return + maker_pk = tmaker_pk + if nick not in self.active_orders.keys(): + log.msg("Counterparty not part of this transaction. Ignoring") + return + try: + self.crypto_boxes[nick] = [maker_pk, as_init_encryption( + self.kp, init_pubkey(maker_pk))] + except NaclError as e: + print("Unable to setup crypto box with " + nick + ": " + repr(e)) + self.mcc.send_error(nick, "invalid nacl pubkey: " + maker_pk) + return + d = self.callRemote(JMFillResponse, success=True, + ioauth_data= json.dumps(list(self.active_orders.keys()))) + self.defaultCallbacks(d) + elif self.role == "MAKER": + # Receiver only cares that version 0 (low) is accepted/in range, + # doesn't check high version value. + taker_pk = tmaker_pk + # Note that the Maker may receive this message from anyone, + # not only the sender he communicated with out of band; + # for this reason, the protocol only provides an ephemeral + # pubkey at this stage; the maker only sends private data, + # encrypted, when the Taker, in the next message (!tx), sends + # data demonstrating knowledge of the out-of-band info - + # specifically, the receiving address. + + #prepare a pubkey for this valid transaction + kp = init_keypair() + try: + crypto_box = as_init_encryption(kp, init_pubkey(taker_pk)) + except NaclError as e: + log.msg("Unable to set up cryptobox with counterparty: " + repr(e)) + self.mcc.send_error(nick, "Invalid nacl pubkey: " + taker_pk) + return + #Note this sets the *whole* dict, old entries (e.g. changeaddr) + #are removed, so we can't have a conflict between old and new + #versions of active_orders[nick] + self.active_orders[nick] = {"crypto_box": crypto_box, + "kp": kp, + "offer": None, + "amount": None, + "commit": None} + self.mcc.prepare_privmsg(nick, "pubkey", self.versioned_pubkey_msg( + kp.hex_pk().decode('ascii'))) + else: + raise JMProtocolError("Invalid role: " + self.role) + + # This (on_pubkey) method can be called repeatedly without + # issue (see note above about active_orders); the following + # state update means that after this is called at least once, + # the JMMakeTx client call can be answered successfully. + self.jm_state = 4 + + def on_seen_tx(self, nick, txhex): + """Passes the txhex to the Taker and Maker (two way sending); + the Maker receives an unsigned tx from the Taker, checks it, + modifies it and sends a partially signed updated tx back. + This functions as a passthrough with a sanity check on the + counterparty identity. + """ + if nick not in self.active_orders: + return + d = self.callRemote(JMTXReceived, + nick=nick, + txhex=txhex, + offer="none") + self.defaultCallbacks(d) + + # Remaining callbacks are mostly to be no-ops (answering + # !orderbook as maker, though, makes sense; on-error is also + # fine of course, and on-push-tx because it does nothing.) + def on_order_fill(self, nick, oid, amount, taker_pk, commit): + pass + def on_seen_auth(self, nick, commitment_revelation): + pass + def on_commitment_seen(self, nick, commitment): + pass + def on_commitment_transferred(self, nick, commitment): + pass + def on_ioauth(self, nick, utxo_list, auth_pub, cj_addr, change_addr, + btc_sig): + pass + def on_sig(self, nick, sig): + pass + + def versioned_pubkey_msg(self, pubkey_hex): + # TODO genericize when updating p2ep version + # Note that this is deliberately an instance method + # as in future will depend on self.role + return pubkey_hex + "0000" class JMDaemonServerProtocolFactory(ServerFactory): protocol = JMDaemonServerProtocol @@ -654,6 +801,11 @@ class JMDaemonServerProtocolFactory(ServerFactory): def buildProtocol(self, addr): return JMDaemonServerProtocol(self) +class P2EPDaemonServerProtocolFactory(ServerFactory): + protocol = P2EPDaemonServerProtocol + + def buildProtocol(self, addr): + return P2EPDaemonServerProtocol(self) def start_daemon(host, port, factory, usessl=False, sslkey=None, sslcert=None): if usessl: diff --git a/scripts/cli_options.py b/scripts/cli_options.py index 29f121f..f5639cc 100644 --- a/scripts/cli_options.py +++ b/scripts/cli_options.py @@ -7,6 +7,7 @@ from optparse import OptionParser, OptionValueError from configparser import NoOptionError import jmclient.support +from jmclient import jm_single, RegtestBitcoinCoreInterface, cryptoengine """This exists as a separate module for two reasons: to reduce clutter in main scripts, and refactor out @@ -199,6 +200,20 @@ max_cj_fee_rel = {rel_val}\n""".format(rel_val=rel_val, abs_val=abs_val)) return rel_val, abs_val +def check_regtest(blockchain_start=True): + """ Applies any regtest-specific configuration + """ + if not isinstance(jm_single().bc_interface, + RegtestBitcoinCoreInterface): + return + if blockchain_start: + #to allow testing of confirm/unconfirm callback for multiple txs + jm_single().bc_interface.tick_forward_chain_interval = 10 + jm_single().bc_interface.simulating = True + jm_single().maker_timeout_sec = 5 + # handles the custom regtest hrp for bech32 + cryptoengine.BTC_P2WPKH.VBYTE = 100 + def get_tumbler_parser(): parser = OptionParser( usage='usage: %prog [options] [wallet file] [destaddr(s)...]', @@ -412,5 +427,15 @@ def get_sendpayment_parser(): dest='answeryes', default=False, help='answer yes to everything') + parser.add_option('--payjoin', + '-T', + type='str', + action='store', + dest='p2ep', + default='', + help='specify recipient IRC nick for a ' + 'p2ep style payment, for example:\n' + 'J5Ehn3EieVZFtm4q ') + add_common_options(parser) return parser diff --git a/scripts/receive-payjoin.py b/scripts/receive-payjoin.py new file mode 100644 index 0000000..9e90792 --- /dev/null +++ b/scripts/receive-payjoin.py @@ -0,0 +1,88 @@ +#! /usr/bin/env python +from __future__ import (absolute_import, division, + print_function, unicode_literals) +from builtins import * # noqa: F401 + +from optparse import OptionParser + +import sys +from twisted.python.log import startLogging +from jmbase import get_log, set_logging_level +from jmclient import P2EPMaker, jm_single, load_program_config, \ + sync_wallet, JMClientProtocolFactory, start_reactor, \ + open_test_wallet_maybe, get_wallet_path +from cli_options import check_regtest + +jlog = get_log() + +def receive_payjoin_main(makerclass): + parser = OptionParser(usage='usage: %prog [options] [wallet file] [amount-to-receive]') + parser.add_option('-g', '--gap-limit', action='store', type="int", + dest='gaplimit', default=6, + help='gap limit for wallet, default=6') + parser.add_option('--fast', + action='store_true', + dest='fastsync', + default=False, + help=('choose to do fast wallet sync, only for Core and ' + 'only for previously synced wallet')) + parser.add_option('-m', '--mixdepth', action='store', type='int', + dest='mixdepth', default=0, + help="mixdepth to source coins from") + parser.add_option('-a', + '--amtmixdepths', + action='store', + type='int', + dest='amtmixdepths', + help='number of mixdepths in wallet, default 5', + default=5) + (options, args) = parser.parse_args() + if len(args) < 2: + parser.error('Needs a wallet, and a receiving amount in satoshis') + sys.exit(0) + wallet_name = args[0] + try: + receiving_amount = int(args[1]) + except: + parser.error("Invalid receiving amount passed: " + receiving_amount) + sys.exit(0) + if receiving_amount < 0: + parser.error("Receiving amount must be a positive integer in satoshis") + sys.exit(0) + load_program_config() + + check_regtest() + + # This workflow requires command line reading; we force info level logging + # to remove noise, and mostly communicate to the user with the fn + # log.info (via P2EPMaker.user_info). + set_logging_level("INFO") + + wallet_path = get_wallet_path(wallet_name, 'wallets') + max_mix_depth = max([options.mixdepth, options.amtmixdepths - 1]) + wallet = open_test_wallet_maybe( + wallet_path, wallet_name, max_mix_depth, + gap_limit=options.gaplimit) + + if jm_single().config.get("BLOCKCHAIN", "blockchain_source") == "electrum-server": + jm_single().bc_interface.synctype = "with-script" + + while not jm_single().bc_interface.wallet_synced: + sync_wallet(wallet, fast=options.fastsync) + + maker = makerclass(wallet, options.mixdepth, receiving_amount) + + jlog.info('starting receive-payjoin') + clientfactory = JMClientProtocolFactory(maker, proto_type="MAKER") + + nodaemon = jm_single().config.getint("DAEMON", "no_daemon") + daemon = True if nodaemon == 1 else False + if jm_single().config.get("BLOCKCHAIN", "network") in ["regtest", "testnet"]: + startLogging(sys.stdout) + start_reactor(jm_single().config.get("DAEMON", "daemon_host"), + jm_single().config.getint("DAEMON", "daemon_port"), + clientfactory, daemon=daemon, p2ep=True) + +if __name__ == "__main__": + receive_payjoin_main(P2EPMaker) + print('done') diff --git a/scripts/sendpayment.py b/scripts/sendpayment.py index 30e0008..533c577 100644 --- a/scripts/sendpayment.py +++ b/scripts/sendpayment.py @@ -14,13 +14,14 @@ import sys from twisted.internet import reactor import pprint -from jmclient import Taker, load_program_config, get_schedule,\ +from jmclient import Taker, P2EPTaker, load_program_config, get_schedule,\ JMClientProtocolFactory, start_reactor, validate_address, jm_single,\ - sync_wallet, RegtestBitcoinCoreInterface, estimate_tx_fee, direct_send,\ + sync_wallet, estimate_tx_fee, direct_send,\ open_test_wallet_maybe, get_wallet_path from twisted.python.log import startLogging -from jmbase.support import get_log, jmprint -from cli_options import get_sendpayment_parser, get_max_cj_fee_values +from jmbase.support import get_log, set_logging_level, jmprint +from cli_options import get_sendpayment_parser, get_max_cj_fee_values, \ + check_regtest log = get_log() @@ -86,11 +87,7 @@ def main(): wallet_name = args[0] - #to allow testing of confirm/unconfirm callback for multiple txs - if isinstance(jm_single().bc_interface, RegtestBitcoinCoreInterface): - jm_single().bc_interface.tick_forward_chain_interval = 10 - jm_single().bc_interface.simulating = True - jm_single().maker_timeout_sec = 15 + check_regtest() if options.pickorders: chooseOrdersFunc = pick_order @@ -105,12 +102,12 @@ def main(): # we guess conservatively with 2 inputs and 2 outputs each. if options.txfee == -1: options.txfee = max(options.txfee, estimate_tx_fee(2, 2, - txtype="p2sh-p2wpkh")) + txtype="p2sh-p2wpkh")) log.debug("Estimated miner/tx fee for each cj participant: " + str( options.txfee)) assert (options.txfee >= 0) - if options.makercount != 0: + if not options.p2ep and options.makercount != 0: maxcjfee = get_max_cj_fee_values(jm_single().config, options) log.info("Using maximum coinjoin fee limits per maker of {:.4%}, {} " "sat".format(*maxcjfee)) @@ -222,19 +219,28 @@ def main(): log.info("All transactions completed correctly") reactor.stop() - taker = Taker(wallet, - schedule, - order_chooser=chooseOrdersFunc, - max_cj_fee=maxcjfee, - callbacks=(filter_orders_callback, None, taker_finished)) + if options.p2ep: + # This workflow requires command line reading; we force info level logging + # to remove noise, and mostly communicate to the user with the fn + # log.info (directly or via default taker_info_callback). + set_logging_level("INFO") + taker = P2EPTaker(options.p2ep, wallet, schedule, + callbacks=(None, None, None)) + else: + taker = Taker(wallet, + schedule, + order_chooser=chooseOrdersFunc, + max_cj_fee=maxcjfee, + callbacks=(filter_orders_callback, None, taker_finished)) clientfactory = JMClientProtocolFactory(taker) nodaemon = jm_single().config.getint("DAEMON", "no_daemon") daemon = True if nodaemon == 1 else False + p2ep = True if options.p2ep != "" else False if jm_single().config.get("BLOCKCHAIN", "network") in ["regtest", "testnet"]: startLogging(sys.stdout) start_reactor(jm_single().config.get("DAEMON", "daemon_host"), jm_single().config.getint("DAEMON", "daemon_port"), - clientfactory, daemon=daemon) + clientfactory, daemon=daemon, p2ep=p2ep) if __name__ == "__main__": main() diff --git a/scripts/sendtomany.py b/scripts/sendtomany.py index a596451..f3b88e1 100644 --- a/scripts/sendtomany.py +++ b/scripts/sendtomany.py @@ -32,6 +32,7 @@ def sign(utxo, priv, destaddrs, segwit=True): assert results[0][0] == utxo amt = results[0][1] ins = [utxo] + # TODO extend to other utxo types txtype = 'p2sh-p2wpkh' if segwit else 'p2pkh' estfee = estimate_tx_fee(1, len(destaddrs), txtype=txtype) outs = [] diff --git a/scripts/tumbler.py b/scripts/tumbler.py index d043837..04d3d3b 100644 --- a/scripts/tumbler.py +++ b/scripts/tumbler.py @@ -11,11 +11,13 @@ from twisted.python.log import startLogging from jmclient import Taker, load_program_config, get_schedule,\ JMClientProtocolFactory, start_reactor, jm_single, get_wallet_path,\ open_test_wallet_maybe, sync_wallet, get_tumble_schedule,\ - RegtestBitcoinCoreInterface, schedule_to_text, restart_waiter,\ + schedule_to_text, restart_waiter,\ get_tumble_log, tumbler_taker_finished_update,\ tumbler_filter_orders_callback from jmbase.support import get_log, jmprint -from cli_options import get_tumbler_parser, get_max_cj_fee_values +from cli_options import get_tumbler_parser, get_max_cj_fee_values, \ + check_regtest + log = get_log() logsdir = os.path.join(os.path.dirname( jm_single().config_location), "logs") @@ -31,6 +33,8 @@ def main(): sys.exit(0) load_program_config() + check_regtest() + #Load the wallet wallet_name = args[0] max_mix_depth = options['mixdepthsrc'] + options['mixdepthcount'] @@ -109,12 +113,6 @@ def main(): elif fromtx != "unconfirmed": reactor.callLater(waittime*60, clientfactory.getClient().clientStart) - #to allow testing of confirm/unconfirm callback for multiple txs - if isinstance(jm_single().bc_interface, RegtestBitcoinCoreInterface): - jm_single().bc_interface.tick_forward_chain_interval = 10 - jm_single().bc_interface.simulating = True - jm_single().maker_timeout_sec = 15 - #instantiate Taker with given schedule and run taker = Taker(wallet, schedule, diff --git a/scripts/wallet-tool.py b/scripts/wallet-tool.py index c2080e6..23e4ed4 100644 --- a/scripts/wallet-tool.py +++ b/scripts/wallet-tool.py @@ -4,8 +4,10 @@ from builtins import * # noqa: F401 from jmbase import jmprint from jmclient import load_program_config, wallet_tool_main +from cli_options import check_regtest if __name__ == "__main__": load_program_config() + check_regtest(blockchain_start=False) #JMCS follows same convention as JM original; wallet is in "wallets" localdir jmprint(wallet_tool_main("wallets"), "success") \ No newline at end of file diff --git a/test/ygrunner.py b/test/ygrunner.py index 96d71d1..76587b2 100644 --- a/test/ygrunner.py +++ b/test/ygrunner.py @@ -19,7 +19,8 @@ import pytest import random from jmbase import jmprint from jmclient import YieldGeneratorBasic, load_program_config, jm_single,\ - sync_wallet, JMClientProtocolFactory, start_reactor + sync_wallet, JMClientProtocolFactory, start_reactor, SegwitWallet,\ + SegwitLegacyWallet, cryptoengine class MaliciousYieldGenerator(YieldGeneratorBasic): @@ -102,9 +103,16 @@ def test_start_ygs(setup_ygrunner, num_ygs, wallet_structures, mean_amt, Then start the ygs in background and publish the seed of the sp wallet for easy import into -qt """ + if jm_single().config.get("POLICY", "native") == "true": + walletclass = SegwitWallet + else: + # TODO add Legacy + walletclass = SegwitLegacyWallet + wallets = make_wallets(num_ygs + 1, wallet_structures=wallet_structures, - mean_amt=mean_amt) + mean_amt=mean_amt, + walletclass=walletclass) #the sendpayment bot uses the last wallet in the list wallet = wallets[num_ygs]['wallet'] jmprint("\n\nTaker wallet seed : " + wallets[num_ygs]['seed']) @@ -146,4 +154,6 @@ def test_start_ygs(setup_ygrunner, num_ygs, wallet_structures, mean_amt, def setup_ygrunner(): load_program_config() jm_single().bc_interface.tick_forward_chain_interval = 10 - jm_single().bc_interface.simulate_blocks() \ No newline at end of file + jm_single().bc_interface.simulate_blocks() + # handles the custom regtest hrp for bech32 + cryptoengine.BTC_P2WPKH.VBYTE = 100