Add support to sign externally prepared PSBT

... with wallet-tool method `signpsbt`. Specifically,
supports co-signing of PSBTs prepared elsewhere (so,
basic Updater/Signer and Finalizer roles, but not
Creator).
Provides detailed user feedback, command line only, and
supports broadcast of finalized transactions.
Supports native and p2sh segwit (and mixed, of course).
Also adds tests using externally prepared PSBTs.
Documentation of PSBT function added to USAGE.md.

docs/USAGE.md

@@ -35,6 +35,8 @@ followed a manual installation as per [here](INSTALL.md)).
 
    j. [What is the Gap Limit](#gaplimit)
 
+   k. [Co-signing a PSBT](#psbt)
+
 4. [Try out a coinjoin; using sendpayment.py](#try-coinjoin)
 
 5. [Running a "Maker" or "yield generator"](#run-maker)
@@ -402,6 +404,19 @@ You can create as many addresses as you like, but not all of them will appear th
 
 When you are starting JoinMarket it does not know which is the last address used. So you start at the beginning and see what is on the blockchain. Then you look for the next one in the sequence. The gap limit is how many *misses* you accept before you give up and stop looking. The same concept is used in other deterministic wallets like Electrum.
 
+<a name="psbt" />
+
+### Co-signing a PSBT
+
+Joinmarket now (version 0.8.1+) has (limited) PSBT support. You can take a PSBT from another source, which has created a transaction spending some of your utxos, as well as others, and co-sign it (for example, a custom coinjoin). If your wallet recognizes the coins as belonging to you, it will sign them and present the transaction ready for broadcast, if the signing is complete. **BE CAREFUL USING THIS FEATURE**. This is, for now, intended to be a manual process; before broadcasting a transaction, you **MUST** carefully read the presented information to ensure you are not spending coins you don't intend to. This feature should work with both native and p2sh segwit wallets (but not tested with non-segwit). For creating such PSBTs in other wallets, before co-signing them here, you will probably need to use the xpub of the account (mixdepth) as shown in the output of `wallet-tool.py` default method `display`; for general analysis of what's going on, don't forget you can use `wallet-tool.py` method `showutxos`.
+
+The syntax is:
+
+`python wallet-tool.py mywalletname.jmdat signpsbt <base64-psbt>`.
+
+Note that you only need the PSBT itself, you don't need e.g. mixdepth or other metadata, the tool will figure out for itself whether this wallet is able to co-sign. If it does not end with a "finalized" (fully signed) PSBT, it will in any case output the latest "updated" PSBT and tell you how many signings it did.
+
+
 <a name="try-coinjoin" />
 
 ## Try out a coinjoin; using `sendpayment.py`

jmclient/jmclient/wallet_utils.py

@@ -1,8 +1,9 @@
+import base64
+import binascii
 import json
 import os
-import sys
 import sqlite3
-import binascii
+import sys
 from datetime import datetime
 from calendar import timegm
 from optparse import OptionParser, IndentedHelpFormatter
@@ -16,7 +17,7 @@ from jmclient import (get_network, WALLET_IMPLEMENTATIONS, Storage, podle,
     is_native_segwit_mode, load_program_config, add_base_options, check_regtest)
 from jmclient.wallet_service import WalletService
 from jmbase.support import (get_password, jmprint, EXIT_FAILURE,
-                            EXIT_ARGERROR, utxo_to_utxostr, hextobin)
+                            EXIT_ARGERROR, utxo_to_utxostr, hextobin, bintohex)
 
 from .cryptoengine import TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, \
     TYPE_SEGWIT_LEGACY_WALLET_FIDELITY_BONDS
@@ -51,6 +52,7 @@ The method is one of the following:
 (dumpprivkey) Export a single private key, specify an hd wallet path.
 (signmessage) Sign a message with the private key from an address in
     the wallet. Use with -H and specify an HD wallet path for the address.
+(signpsbt) Sign PSBT with JoinMarket wallet.
 (freeze) Freeze or un-freeze a specific utxo. Specify mixdepth with -m.
 (gettimelockaddress) Obtain a timelocked address. Argument is locktime value as yyyy-mm. For example `2021-03`.
 (addtxoutproof) Add a tx out proof as metadata to a burner transaction. Specify path with
@@ -1064,6 +1066,37 @@ def wallet_signmessage(wallet, hdpath, message):
     retval = "Signature: {}\nTo verify this in Bitcoin Core".format(sig)
     return retval + " use the RPC command 'verifymessage'"
 
+def wallet_signpsbt(wallet_service, psbt):
+    if not psbt:
+        return "Error: no PSBT specified"
+
+    signed_psbt_and_signresult, err = wallet_service.sign_psbt(
+        base64.b64decode(psbt.encode('ascii')), with_sign_result=True)
+    if err:
+        return "Failed to sign PSBT, quitting. Error message: {}".format(err)
+    signresult, signedpsbt = signed_psbt_and_signresult
+    jmprint(wallet_service.human_readable_psbt(signedpsbt))
+    jmprint("Base64 of the above PSBT:")
+    jmprint(signedpsbt.to_base64())
+    if signresult.is_final:
+        if input("Above PSBT is fully signed. Do you want to broadcast?"
+                 "(y/n):") != "y":
+            jmprint("Not broadcasting.")
+        else:
+            jmprint("Broadcasting...")
+            tx = signedpsbt.extract_transaction()
+            if jm_single().bc_interface.pushtx(tx.serialize()):
+                jmprint("Transaction sent: " + bintohex(
+                    tx.GetTxid()[::-1]))
+            else:
+                jmprint("Transaction broadcast failed!", "error")
+    else:
+        # if the signing action did not result in a finalized PSBT,
+        # inform the user of the current status:
+        jmprint("The PSBT is not yet fully signed, we signed: {} "
+                "inputs.".format(signresult.num_inputs_signed))
+    return ""
+
 def display_utxos_for_disable_choice_default(wallet_service, utxos_enabled,
         utxos_disabled):
     """ CLI implementation of the callback required as described in
@@ -1503,6 +1536,11 @@ def wallet_tool_main(wallet_root_path):
             jmprint('Must provide message to sign', "error")
             sys.exit(EXIT_ARGERROR)
         return wallet_signmessage(wallet_service, options.hd_path, args[2])
+    elif method == "signpsbt":
+        if len(args) < 3:
+            jmprint("Must provide PSBT to sign", "error")
+            sys.exit(EXIT_ARGERROR)
+        return wallet_signpsbt(wallet_service, args[2])
     elif method == "freeze":
         return wallet_freezeutxo(wallet_service, options.mixdepth)
     elif method == "gettimelockaddress":

jmclient/test/test_psbt_wallet.py

@@ -7,16 +7,78 @@
    '''
 
 import copy
+import base64
 from commontest import make_wallets, dummy_accept_callback, dummy_info_callback
 
 import jmbitcoin as bitcoin
 import pytest
-from jmbase import get_log, bintohex, hextobin
+from jmbase import get_log, bintohex, hextobin, utxostr_to_utxo
 from jmclient import (load_test_config, jm_single, direct_send,
-                      SegwitLegacyWallet, SegwitWallet, LegacyWallet)
+                      SegwitLegacyWallet, SegwitWallet, LegacyWallet,
+                      VolatileStorage, get_network)
 from jmclient.wallet import PSBTWalletMixin
 log = get_log()
 
+def create_volatile_wallet(seedphrase, wallet_cls=SegwitWallet):
+    storage = VolatileStorage()
+    wallet_cls.initialize(storage, get_network(), max_mixdepth=4,
+                          entropy=wallet_cls.entropy_from_mnemonic(seedphrase))
+    storage.save()
+    return wallet_cls(storage)
+
+@pytest.mark.parametrize('walletseed, xpub, spktype_wallet, spktype_destn, partial, psbt', [
+    ("prosper diamond marriage spy across start shift elevator job lunar edge gallery",
+     "tpubDChjiEhsafnW2LcmK1C77XiEAgZddi6xZyxjMujBzUqZPTMRwsv3e5vSBYsdiPtCyc6TtoHTCjkxBjtF22tf8Z5ABRdeBUNwHCsqEyzR5wT",
+     "p2wpkh", "p2sh-p2wpkh", False,
+     "cHNidP8BAMQCAAAAA7uEliZeXLPfjeUiRBw6e5oZV1DtBrDmLthfDC4oaHQLAAAAAAD/////+X1Exketc4o5b9BPxsj70O+VlGvgiZz0KP1OMRtVLUQAAAAAAP////9r5ylMhQyxbJvCbU8aNE3NOPoXJwUaUZm4H3iT4RnaSwAAAAAA/////wKMz/AIAAAAABepFJwmRAefvZS7VQStD4k52Rn0k71Gh4zP8AgAAAAAFgAUA2shnTVftDXq+ssPwzml2UKdu1QAAAAAAAEBHwDh9QUAAAAAFgAUqw1Ifto4LztwcsxV6q+sQThIdloiBgMDZ5u3RN6Xum+OLkgAzwLFXGWFLwBUraMi7Oin4fYfrwzvYoLxAAAAAAAAAAAAAQEfAOH1BQAAAAAWABQpSCwoeMSghUoVflvtTPiqBPi+5yIGA/tAH4kVpqd3wzidaTNFxtwdpHTydkmB825us2w/3cAVDO9igvEAAAAAAQAAAAABAR8A4fUFAAAAABYAFEqM0KJ5FJ7ak2NL8PDqOPI0I1PaIgYD84aDwOqXKfGvEbre+bpNpuT0uZv6syESzz5PMu4RyLkM72KC8QAAAAACAAAAAAEAF6kUnCZEB5+9lLtVBK0PiTnZGfSTvUaHACICAw8k2gGGcF5sR8yKO5JeAkrkH15rmtCq8sCoDYbywTNzEO9igvEMAAAAIgAAAJQCAAAA"),
+    ("prosper diamond marriage spy across start shift elevator job lunar edge gallery",
+     "tpubDChjiEhsafnW2LcmK1C77XiEAgZddi6xZyxjMujBzUqZPTMRwsv3e5vSBYsdiPtCyc6TtoHTCjkxBjtF22tf8Z5ABRdeBUNwHCsqEyzR5wT",
+     "p2wpkh", "p2wpkh", False,
+     "cHNidP8BAMMCAAAAA7uEliZeXLPfjeUiRBw6e5oZV1DtBrDmLthfDC4oaHQLAAAAAAD/////+X1Exketc4o5b9BPxsj70O+VlGvgiZz0KP1OMRtVLUQAAAAAAP////9r5ylMhQyxbJvCbU8aNE3NOPoXJwUaUZm4H3iT4RnaSwAAAAAA/////wKMz/AIAAAAABYAFBaOTObQIdtCaryiPxaDV5rsGYWUjM/wCAAAAAAWABR9TJm5rcSoIMW7bE1bnj7REL/eygAAAAAAAQEfAOH1BQAAAAAWABSrDUh+2jgvO3ByzFXqr6xBOEh2WiIGAwNnm7dE3pe6b44uSADPAsVcZYUvAFStoyLs6Kfh9h+vDO9igvEAAAAAAAAAAAABAR8A4fUFAAAAABYAFClILCh4xKCFShV+W+1M+KoE+L7nIgYD+0AfiRWmp3fDOJ1pM0XG3B2kdPJ2SYHzbm6zbD/dwBUM72KC8QAAAAABAAAAAAEBHwDh9QUAAAAAFgAUSozQonkUntqTY0vw8Oo48jQjU9oiBgPzhoPA6pcp8a8Rut75uk2m5PS5m/qzIRLPPk8y7hHIuQzvYoLxAAAAAAIAAAAAACICAuqCicVUfcM5IiVSiB/0ZemodybG5Im9Fu8MLorQSE4UEO9igvEMAAAAIgAAAOkAAAAA"),
+    ("prosper diamond marriage spy across start shift elevator job lunar edge gallery",
+     "tpubDChjiEhsafnW2LcmK1C77XiEAgZddi6xZyxjMujBzUqZPTMRwsv3e5vSBYsdiPtCyc6TtoHTCjkxBjtF22tf8Z5ABRdeBUNwHCsqEyzR5wT",
+     "p2wpkh", "p2wpkh", True,
+     "cHNidP8BAMMCAAAAA7uEliZeXLPfjeUiRBw6e5oZV1DtBrDmLthfDC4oaHQLAAAAAAD/////+X1Exketc4o5b9BPxsj70O+VlGvgiZz0KP1OMRtVLUQAAAAAAP////9r5ylMhQyxbJvCbU8aNE3NOPoXJwUaUZm4H3iT4RnaSwAAAAAA/////wKMz/AIAAAAABYAFLH/IL11rTJ3wX1NcmUIsJ/T4j4jjM/wCAAAAAAWABR8GPNb1HUpCz8PKOc8aQXLD1wjcAAAAAAAAQEfAOH1BQAAAAAWABSrDUh+2jgvO3ByzFXqr6xBOEh2WiIGAwNnm7dE3pe6b44uSADPAsVcZYUvAFStoyLs6Kfh9h+vDE5vcGUAAAAAAAAAAAABAR8A4fUFAAAAABYAFClILCh4xKCFShV+W+1M+KoE+L7nIgYD+0AfiRWmp3fDOJ1pM0XG3B2kdPJ2SYHzbm6zbD/dwBUM72KC8QAAAAABAAAAAAEBHwDh9QUAAAAAFgAUSozQonkUntqTY0vw8Oo48jQjU9oiBgPzhoPA6pcp8a8Rut75uk2m5PS5m/qzIRLPPk8y7hHIuQzvYoLxAAAAAAIAAAAAACICAsQ7ZvU9tsbBoSje5rIJQBStlUkQaRCssKylEixre3AYEO9igvEMAAAAIgAAABcCAAAA"),
+])
+def test_sign_external_psbt(setup_psbt_wallet, walletseed, xpub,
+                            spktype_wallet, spktype_destn, partial, psbt):
+    bitcoin.select_chain_params("bitcoin")
+    wallet_cls = SegwitWallet if spktype_wallet == "p2wpkh" else SegwitLegacyWallet
+    wallet = create_volatile_wallet(walletseed, wallet_cls=wallet_cls)
+    # if we want to actually sign, our wallet has to recognize the fake utxos
+    # as being in the wallet, so we inject them:
+    class DummyUtxoManager(object):
+        _utxo = {0:{}}
+        def add_utxo(self, utxo, path, value, height):
+            self._utxo[0][utxo] = (path, value, height)
+    wallet._index_cache[0][0] = 1000
+    wallet._utxos = DummyUtxoManager()
+    p0, p1, p2 = (wallet.get_path(0, 0, i) for i in range(3))
+    if not partial:
+        wallet._utxos.add_utxo(utxostr_to_utxo(
+            "0b7468282e0c5fd82ee6b006ed5057199a7b3a1c4422e58ddfb35c5e269684bb:0"),
+                               p0, 10000, 1)
+    wallet._utxos.add_utxo(utxostr_to_utxo(
+        "442d551b314efd28f49c89e06b9495efd0fbc8c64fd06f398a73ad47c6447df9:0"),
+                           p1, 10000, 1)
+    wallet._utxos.add_utxo(utxostr_to_utxo(
+        "4bda19e193781fb899511a052717fa38cd4d341a4f6dc29b6cb10c854c29e76b:0"),
+                           p2, 10000, 1)
+    signresult_and_signedpsbt, err = wallet.sign_psbt(base64.b64decode(
+        psbt.encode("ascii")),with_sign_result=True)
+    assert not err
+    signresult, signedpsbt = signresult_and_signedpsbt
+    if partial:
+        assert not signresult.is_final
+        assert signresult.num_inputs_signed == 2
+        assert signresult.num_inputs_final == 2
+    else:
+        assert signresult.is_final
+        assert signresult.num_inputs_signed == 3
+        assert signresult.num_inputs_final == 3
+    print(PSBTWalletMixin.human_readable_psbt(signedpsbt))
+    bitcoin.select_chain_params("bitcoin/regtest")
+
 def test_create_and_sign_psbt_with_legacy(setup_psbt_wallet):
     """ The purpose of this test is to check that we can create and
     then partially sign a PSBT where we own one input and the other input