In the previous commit, all peers served an onion.
After this commit, taker client instances will automatically
send a config var to the jmdaemon backend that instructs
the OnionMessageChannel instance to not start an onion service,
and the handshake messages sent by these peers replace the
onion location with a placeholder string NOT-SERVING-ONION.
Directories and maker peers will not therefore to connect outbound
to them, but privmsging still happens p2p with connections from
takers to makers after the directory has communicated their
reachable .onion addresses.
This change reduces the configuration requirement for takers and
is better for their privacy and security (without sacrificing
the gain we get from having p2p connections).
The above comments re: takers also apply to ob-watcher bots.
This commit also fixes a large number of minor bugs and errors in
documentation, as well as many Python cleanups after review from
@PulpCattel. A few concrete items are:
It fixes the ob-watcher functionality to work with the new subclass
of MessageChannel (OnionMessageChannel).
It corrects the on_nick_leave trigger to make dynamic nick switching
between MessageChannels (as implemented in MessageChannelCollection)
work correctly.
It corrects the order of events in the add_peer workflow to ensure that
a handshake can always be sent so that the activation of the connection
always works.
It sets a default messaging config with onion, 2 active IRC servers and
one inactive IRC server. The onion config has 2 signet directory nodes,
so this will change to mainnet after the PR is merged to master.
Joinmarket bots run their own onion services allowing inbound connections.
Both takers and makers connect to other makers at the mentioned
onion services, over Tor.
Directory nodes run persistent onion services allowing peers to
find other (maker) peers to connect to, and also forwarding
messages where necessary.
This is implemented as an alternative to IRC, i.e. a new
implementation of the abstract class MessageChannel, in onionmc.py.
Note that using both this *and* IRC servers is supported; Joinmarket
supports multiple, redundant different communication methods,
simultaneously.
Messaging is done with a derived class of twisted's LineReceiver,
and there is an additional layer of syntax, similar to but not the
same as the IRC syntax for ensuring that messages are passed with
the same J5.. nick as is used on IRC. This allows us to keep the
message signing logic the same as before. As well as Joinmarket line
messages, we use additional control messages to communicate peer lists,
and to manage connections.
Peers which send messages not conforming to the syntax are dropped.
See https://github.com/JoinMarket-Org/JoinMarket-Docs/pull/12 for
documentation of the syntax.
Connections to directory nodes are robust as for IRC servers, in
that we use a ReconnectingClientFactory to keep trying to re-establish
broken connections with exponential backoff. Connections to maker
peers do not require this feature, as they will often disconnect
in normal operation.
Multiple directory nodes can and should be configured by bots.
Prior to this commit, the file commitmentlist, which stores
commitments (podle) for makers to let them blacklist/prevent
reuse was stored in the local directory for the script, which
allowed remote running of jmdaemon but was very unhelpful for
any situation where multiple bots are running at once, e.g.
in testing or using multiple wallets against the same codebase.
This could result in incorrect rejection of coinjoins.
After this commit, by default, the commitmentlist file is stored
in datadir/cmtdata/commitmentlist, so it will be local to any
custom data directory as would be the case for running multiple
wallets on the same machine. A user can set the POLICY variable
commitment_list_location to "." to revert to the previous behaviour.
Fixes#1027. Previous to this commit, the onion
service used by the payjoin receiver was automatically
served on localhost (at an arbitrary port), after this
commit these values can be specified in the PAYJOIN
section of the config.
1. Moves the JMWalletDaemon service class into
the jmclient package (see the wallet_rpc.py module).
2. Adds dependencies "klein" and "autobahn" to the
jmclient package, as well as "pyjwt".
3. Adds another module websocketserver.py, using
autobahn, to allow the JMWalletDaemon service to
serve subscriptions over a websocket, for e.g.
transaction notifications.
4. Adds tests both for the websocket connection
and for the JSON-RPC HTTP connection.
JmwalletdWebSocketServerFactory.sendTxNotification
sends the json-ified transaction details using
jmbitcoin.human_readable_transaction (as is currently
used in our CLI), along with the txid.
Also adds a coinjoin state update event sent via
the websocket (switch from taker/maker/none).
Require authentication to connect to websocket.
5. Add OpenApi definition of API in yaml;
also auto-create human-readable API docs in markdown.
6. Add fidelity bond function to API
7. Add config read/write route to API
8. Remove snicker rpc calls temporarily
9. Updates to docoinjoin: corrects taker_finished
for this custom case, does not shut down at end.
10. Address detailed review comments of @PulpCattel.
The /utxos route is the equivalent of the showutxos
wallet tool method.
The heartbeat route /session allows a client to
make sure the backend is still running and in the
expected state (but see later commits for the
coinjoin state update via the websocket).
Fixes to sendpayment, the maker service,
the create wallet function.
sendpayment fix
Also substantially improved and made
functional the coinjoin route, a schedule
is now created and a complete taker-side
coinjoin is now possible.
Uses Klein to provide HTTP server support.
Adds cookie based auth to requests (made JWT token
based in later commits).
Basic routes are: /unlock, /lock, /display,
/create of wallet.
Encapsulates WalletDaemon as a Service
Add snicker receiver service start, stop
Adds yg/maker function as stoppable service.
Adds a JMShutdown command to
the AMP protocol, allowing a clean shutdown
of a long running bot (e.g. maker) by shutting
down its message channel connections, without
shutting down the entire process.
Adds payment(direct send) request, first draft
Fixes#899.
Before this commit, the dust threshold used to filter
orders in `jmdaemon.OrderbookWatch` was different than
that used in `jmclient`. This is corrected by setting
the threshold in client config and passing this as a
parameter in the `JMInit` command (since the daemon
does not know the client config).
If a yield generator is run with a fidelity bond wallet then the
most-valuable bond will be found and announced.
The announcement includes a proof of a UTXO and its locktime. Also a
proof that the maker's IRC nickname controls the UTXO.
There is also an intermediate signature called the certificate
signature which can later be used when holding fidelity bond UTXOs in
cold storage without the protocol needing to change. Right now this
feature is unused so certificates are generated dynamically on each
request. The certificates have an expiry block height, which is defined
as the number of 2016-block retargeting periods since the genesis
block, so to check if the expiry was passed the taker will check
`current_block_height > cert_expiry*2016`.
Parse incoming and announce outgoing fidelity bond messages
Fidelity bond proof messages will be checked and added to the internal
database just like offers. Such messages are not announced in public
but only directly to takers who ask for them, this is because the
signature proofs must commmit to the maker's and taker's IRC nicknames
in order to avoid replay attacks.
This commit fixes two minor bugs encountered in
running create-snicker-proposal.py. First, the
port for AMP connections was not correct (by
convention it is now the Joinmarket proto main port
minus 1000), second in the above mentioned script,
the reactor startup client side was not disabling
the Joinmarket protocol, as it should.
This completes the task of enabling
network isolation by running the receiver
side using a hidden service in the daemon,
and communicating over AMP, as is already
the case for the sender.
Updates test_payjoin for daemon receiver.
Qt BIP78 receiver update for daemon.
In case of all-in-one (client and daemon in same process,
the default), this commit ensures that if there was a port
conflict and the port number for the daemon was incremented,
the client connects to the newly created protocol instance.
And this applies to all three protocols (jm, snicker, bip78).
This PR creates a client-daemon protocol for
the BIP78 sender, using the base protocol
`HTTPPassThrough` which provides tor and non-tor
agents with POST and GET request functionality.
As for Joinmarket coinjoins, the use of an in-process
daemon is the default option, but it can be isolated
by changing the `[DAEMON]` section of the config.
The receiver side of BIP78 will be addressed in a
future PR.
Fixes#784. In `Taker.receive_utxos` we check, in the
sweep case, whether the fee violates the user config
setting `max_sweep_fee_change`; if so, the tx is aborted,
and we shut down for single-shot coinjoins, but wait for
stallMonitor to restart for multi-schedules. The value is
defaulted to 80% to give plenty of breathing room to avoid
using up too many commitments.
This commit implements a command line script and a GUI
dialog to receive a payment using the BIP78 protocol,
by setting up an ephemeral hidden service.
It also deprecates the pre-existing inter-Joinmarket
protocol for payjoin payments, since we now have
both sending and receiving support for BIP78. Thus,
much code in Maker, Taker and client-daemon protocol
is removed, as is some documentation in docs/PAYJOIN.md.
Also the script `sendpayment.py` is altered to support
only the BIP78 variant.
The test in jmclient/test/test_payjoin now implements
BIP78 over a TCP connection, while the custom tests in
test/payjoinserver.py can support hidden service based
tests, but the latter is not included in the test suite
and may not always work (it is only for manual
investigations).
The following features of BIP78 are supported:
minfeerate
additionalfeeoutputindex - but *only* for single
change output transactions
maxadditionalfeecontribution
The receiver does not have nor request payment
output substitution.
Utxo selection is no longer sophisticated, instead
we only choose a single utxo to keep the size
increase of the transaction minimal. Thus UIH is
not addressed at the moment.
Errors returned are in line with BIP78.
Sequence numbers are checked by receiver, and
kept identical if uniform, otherwise respected.
Receiver uses transaction monitor to shut down
when the payment is seen.
The workflow is almost entirely implemented in
jmclient/payjoin.py and the command line script
is in scripts/receive-payjoin.py. The setup, including
configuration changes for Tor, are documented in
docs/PAYJOIN.md, including a user guide video linked.
Prior to this commit, users setting the POLICY config
option `tx_broadcast` to anything other than `self` would
cause a crash after the merge of #536 due to a bin/hex
conversion failure (before this merge, the tx would simply
fail to broadcast).
This commit adds a `JMTXBroadcast` AMP command so that makers
can send arbitrary transactions from daemon to client, for
broadcast via the blockchain interface. This allows the
existing code in `taker.push()` to function correctly, after
fixing the bin/hex conversion bug. Hence users can now select
`random-peer` or `not-self` and the transaction will be
broadcast as expected according to the comments, and the
WalletService will react to the broadcast just as it does
currently for self-broadcast.
Note that this change will be ineffective if the counterparties
do not support it; the transaction will simply remain un-broadcast.
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
Introduces WalletService object which is in control of
blockchain and wallet access.
The service manages a single transaction monitoring loop,
instead of multiple, and allows updates to the wallet from
external sources to be handled in real time, so that both Qt
and other apps (yg) can respond to deposits or withdrawals
automatically.
The refactoring also controls access to both wallet and
blockchain so that client apps (Taker, Maker) will not need
to be changed for future new versions e.g. client-side filtering.
Also updates and improves Wallet Tab behaviour in Qt (memory
of expansion state).
Additionally, blockchain sync is now --fast by default, with
the former default of detailed sync being renamed --recoversync.
The hostid upper/lowercase seems can be different depending
on how someone connects to an IRC server (via TOR, clearnet,
etc.). This leads to message signature verification failures,
if the receiving counterparty has a differently capitalized id.
Prior to this commit, running joinmarketd as an isolated
process only supported serving the Joinmarket protocol;
attempting PayJoin would result in silent failure as the
receiver would reject the !pubkey message.
After this commit, the joinmarketd daemon serves both
protocols with the PayJoin protocol being served at port+1
- update mktx() to allow optional locktime setting (and sequence)
- add a mk_shuffled_tx method to the wallet module
- add a P2EPTaker and P2EPMaker class (inherit from Taker, Maker)
- add a -T option to sendpayment script for doing payjoins
- add a receive_payjoin script for receivers.
- add payjoin tests in jmclient/test/test_payjoin.py
- add a custom utxo selection method select_one_utxo to support.py
- support bech32 wallets (SegwitWallet, p2wpkh) with native=true
in config POLICY for PayJoin and direct send (not Joinmarket CJ)
- add a PayJoin.md usage guide in docs/
- include version bytes in pubkey message for forward compat
- taker pays fees but controls size (utxo number and fee/kB)
- add P2WPKH fee estimator
- Enforce INFO level logging in payjoin
- refactors regtest config settings into one place
- bugfix: script_to_address vbyte argument is bytes not integer
Previous to this commit, the jmbitcoin package was accessed via
a file in jmclient `btc.py` which was originally added as an
interface to allow the client to use a non-jmbitcoin package to
provide the implementation; while this idea is useful, the way
it was implemented was not, moreover it is not currently used
and contained duplicated code that was unmanaged. Also, the
original usage of this was only by the electrum plugin, which
has currently been abandoned. This simplifies the code and
avoids spurious error messages. Note that most of the changes
are a result of pulling the logging function directly from the
jmbase package instead of indirectly via this interface (which
was unnecessary and not connected with jmbitcoin).
If a user doing sendpayment decided to reject the offered fees,
before this commit this resulted in a crash because the call
to Taker.on_finished_callback was passed with fromtx=True for
a single line schedule, which attempts to restart. fromtx should
have been set to False when it is the last item in the schedule
(in this case, both first and last). This incorrect argument
existed before restart-with-subset but was then harmless, now
it attempts to recover with no Taker.nonrespondants and crashes.
Fix is simply to provide the right second argument (False) to
the function call.
Make RPC connection persistent to improve high throughput RPC
access where necessary; uses keep alive and recreates connection
when it drops.
Restrict listtransaction calls to the required account, plus only
looks back 100 txs (assuming concurrent txs less than this), thus
greatly reducing the number of gettransaction calls over RPC.
Fixes bug in choose_sweep_orders (was not filtering out non-sw
orders).
Removes tickchainthread from tests, so no longer any threads used
even in tests; replaces with reactor task loop.
Temporarily removes test_wallets and test_segwit from build tests,
since they used blocking which only worked in threaded tests; these
tests must be rebuilt.
Updates to joinmarket-qt to reflect; no signals, only callbacks
Includes code for watching txs both via notify addr and txs;
taker uses the latter (knows full txid), maker the former
Additional code for watching tx spends included for downstream.
In case Taker.initialize() fails for any reason such as no
liquidity, stallMonitor() will not wake up for 20*maker_timeout_sec;
so, in this change, IF the client schedule has length one, we
immediately figure the on_finished_callback with failure parameters
so that the reactor is stopped without waiting.
This change does not apply for the general multi-step schedule, as
that is principally for tumbler-style algos which are intended to
aggressively keep trying in case of any failure.
zebra-lucky
roshii
Adam Gibson
Adam Gibson
Kristaps Kaupe
abhishek0405
undeath
chris-belcher
James Hilliard
cindiv
Alex Cato
AlexCato