Browse Source

Merge #1182: Only onion message channels

5cc1695 Disconnection logic fixes, add btcnet to handshake (Adam Gibson)
830ac22 Allow taker peers to not serve onions + bugfixes. (Adam Gibson)
fd550ee Onion-based message channels with directory nodes (Adam Gibson)
master
Adam Gibson 4 years ago
parent
commit
52eeeb09b9
No known key found for this signature in database
GPG Key ID: 141001A1AF77F20B
  1. 209
      docs/onion-message-channels.md
  2. 4
      jmbase/jmbase/commands.py
  3. 77
      jmbase/jmbase/twisted_utils.py
  4. 6
      jmbase/test/test_commands.py
  5. 4
      jmclient/jmclient/__init__.py
  6. 19
      jmclient/jmclient/client_protocol.py
  7. 188
      jmclient/jmclient/configure.py
  8. 7
      jmclient/jmclient/wallet_rpc.py
  9. 4
      jmclient/test/test_client_protocol.py
  10. 1
      jmdaemon/jmdaemon/__init__.py
  11. 30
      jmdaemon/jmdaemon/daemon_protocol.py
  12. 8
      jmdaemon/jmdaemon/message_channel.py
  13. 1358
      jmdaemon/jmdaemon/onionmc.py
  14. 8
      jmdaemon/test/test_daemon_protocol.py
  15. 6
      jmdaemon/test/test_irc_messaging.py
  16. 4
      jmdaemon/test/test_orderbookwatch.py
  17. 4
      scripts/joinmarket-qt.py
  18. 68
      scripts/obwatch/ob-watcher.py
  19. 340
      test/e2e-coinjoin-test.py
  20. 42
      test/regtest_joinmarket.cfg
  21. 3
      test/ygrunner.py

209
docs/onion-message-channels.md

@ -0,0 +1,209 @@
# HOW TO SETUP ONION MESSAGE CHANNELS IN JOINMARKET
### Contents
1. [Overview](#overview)
2. [Testing, configuring for signet](#testing)
4. [Directory nodes](#directory)
<a name="overview" />
## Overview
This is a new way for Joinmarket bots to communicate, namely by serving and connecting to Tor onion services. This does not
introduce any new requirements to your Joinmarket installation, technically, because the use of Payjoin already required the need
to run such onion services, and connecting to IRC used a SOCKS5 proxy (used by almost all users) over Tor to
a remote onion service.
(Note however that taker bots will *not* be required to serve onions; they will only make outbound SOCKS connections, as they currently do on IRC).
The purpose of this new type of message channel is as follows:
* less reliance on any service external to Joinmarket
* most of the transaction negotiation will be happening directly peer to peer, not passed over a central server (
albeit it was and remains E2E encrypted data, in either case)
* the above can lead to better scalability at large numbers
* a substantial increase in the speed of transaction negotiation; this is mostly related to the throttling of high bursts of traffic on IRC
The configuration for a user is simple; in their `joinmarket.cfg` they will get a new `[MESSAGING]` section like this, if they start from scratch:
```
[MESSAGING:onion]
# onion based message channels must have the exact type 'onion'
# (while the section name above can be MESSAGING:whatever), and there must
# be only ONE such message channel configured (note the directory servers
# can be multiple, below):
type = onion
socks5_host = localhost
socks5_port = 9050
# the tor control configuration.
# for most people running the tor daemon
# on Linux, no changes are required here:
tor_control_host = localhost
# or, to use a UNIX socket
# tor_control_host = unix:/var/run/tor/control
tor_control_port = 9051
# the host/port actually serving the hidden service
# (note the *virtual port*, that the client uses,
# is hardcoded to 80):
onion_serving_host = 127.0.0.1
onion_serving_port = 8080
# directory node configuration
#
# This is mandatory for directory nodes (who must also set their
# own *.onion:port as the only directory in directory_nodes, below),
# but NOT TO BE USED by non-directory nodes (which is you, unless
# you know otherwise!), as it will greatly degrade your privacy.
# (note the default is no value, don't replace it with "").
hidden_service_dir =
#
# This is a comma separated list (comma can be omitted if only one item).
# Each item has format host:port ; both are required, though port will
# be 80 if created in this code.
# for MAINNET:
directory_nodes = 3kxw6lf5vf6y26emzwgibzhrzhmhqiw6ekrek3nqfjjmhwznb2moonad.onion,qqd22cwgygaxcy6vdw6mzwkyaxg5urb4ptbc5d74nrj25phspajxjbqd.onion
# for SIGNET (testing network):
# directory_nodes = rr6f6qtleiiwic45bby4zwmiwjrj3jsbmcvutwpqxjziaydjydkk5iad.onion:80,k74oyetjqgcamsyhlym2vgbjtvhcrbxr4iowd4nv4zk5sehw4v665jad.onion:80
# This setting is ONLY for developer regtest setups,
# running multiple bots at once. Don't alter it otherwise
regtest_count = 0,0
```
All of these can be left as default for most users - but most importantly, pay attention to:
* The list of `directory_nodes`, which will be comma separated if multiple directory nodes are configured (we expect there will be 2 or 3 as a normal situation). Make sure to choose the ones for your network (mainnet by default, or signet or otherwise); if it's wrong your bot will just get auto-disconnected.
* The `onion_serving_port` is the port on the local machine on which the onion service is served; you won't usually need to use it, but it mustn't conflict with some other usage (so if you have something running on port 8080, change it).
The `type` field must always be `onion` in this case, and distinguishes it from IRC message channels and others.
### Can/should I still run IRC message channels?
In short, yes, at least for now, though you are free to disable any message channel you like.
### Do I need to configure Tor, and if so, how?
To make outbound Tor connections to other onions in the network, you will need to configure the
SOCKS5 proxy settings (so, only directory nodes may *not* need this; everyone else does).
This is identical to what we already do for IRC, except that in this case, we disallow clearnet connections.
#### Running/testing as a maker
A maker will additionally allow *inbound* connections to an onion service.
This onion service will be ephemeral, that is, it will have a different onion address every time
you restart. This should work automatically, using your existing Tor daemon (here, we are using
the same code as we use when running the `receive-payjoin` script, essentially).
#### Running/testing as other bots (taker, ob-watcher)
A taker will not attempt to serve an onion; it will only use outbound connections, first to directory
nodes and then, as according to need, to individual makers, also.
As previously mentioned, both of these features - inbound and outbound, to onion, Tor connections - were already in use in Joinmarket. If you want to run/test as a maker bot, but never served an onion service before, it should work fine as long as you have the Tor service running in the background,
and the default control port 9051 (if not, change that value in the `joinmarket.cfg`, see above).
#### Why not use Lightning based onions?
(*Feel free to skip this section if you don't know what "Lightning based onions" refers to!*). The reason this architecture is
proposed as an alternative to the previously suggested Lightning-node-based network (see
[this PR](https://github.com/JoinMarket-Org/joinmarket-clientserver/pull/1000)), is mostly that:
* the latter has a bunch of extra installation and maintenance dependencies (just one example: pyln-client requires coincurve, which we just
removed)
* the latter requires establishing a new node "identity" which can be refreshed, but that creates more concern
* longer term ideas to integrate Lightning payments to the coinjoin workflow (and vice versa!) are not realizable yet
* using multi-hop onion messaging in the LN network itself is also a way off, and a bit problematic
So the short version is: the Lightning based alternative is certainly feasible, but has a lot more baggage that can't really be justified
unless we're actually using it for something.
<a name="testing" />
## Testing, and configuring for signet.
This testing section focuses on signet since that will be the less troublesome way of getting involved in tests for
the non-hardcore JM developer :)
(For the latter, please use the regtest setup by running `test/e2e-coinjoin-test.py` under `pytest`,
and pay attention to the settings in `regtest_joinmarket.cfg`.)
There is no separate/special configuration for signet other than the configuration that is already needed for running
Joinmarket against a signet backend (so e.g. RPC port of 38332).
You can just uncomment the `directory_nodes` entry listed as SIGNET, and comment out the one for MAINNET.
Then just make sure your bot has some signet coins and try running as maker or taker or both.
<a name="directory" />
## Directory nodes
**This last section is for people with a lot of technical knowledge in this area,
who would like to help by running a directory node. You can ignore it if that does not apply.**.
This requires a long running bot. It should be on a server you can keep running permanently, so perhaps a VPS,
but in any case, very high uptime. For reliability it also makes sense to configure to run as a systemd service.
The currently suggested way to run a directory node is to use the script found [here](https://github.com/JoinMarket-Org/custom-scripts/blob/0eda6154265e012b907c43e2ecdacb895aa9e3ab/start-dn.py); you can place it in your `joinmarket-clientserver/scripts` directory and run it *without* arguments, but with one option flag: `--datadir=/your/chosen/datadir` (as you'll see below).
This slightly unobvious approach is based on the following ideas: we run a Joinmarket script, with a Joinmarket python virtualenv, so that we are able to parse messages; this means that the directory node *can* be a bot, e.g. a maker bot, but need not be - and here it is basically a "crippled" maker bot that cannot do anything. This 'crippling' is actually very useful because (a) we use the `no-blockchain` argument (it is forced in-code; you don't need to set it) so we don't need a running Bitcoin node (of whatever flavour), and (b) we don't need a wallet either.
#### Joinmarket-specific configuration
Add a non-empty `hidden_service_dir` entry to your `[MESSAGING:onion]` with a directory accessible to your user. You may want to lock this down
a bit, but be careful changing permissions from what is created by the script, because Tor is very finicky about this.
The hostname for your onion service will not change and will be stored permanently in that directory.
The point to understand is: Joinmarket's `jmbase.JMHiddenService` will, if configured with a non-empty `hidden_service_dir`
field, actually start an *independent* instance of Tor specifically for serving this, under the current user.
(our Tor interface library `txtorcon` needs read access to the Tor HS dir, so it's troublesome to do this another way).
##### Question: How to configure the `directory-nodes` list in our `joinmarket.cfg` for this directory node bot?
Answer: **you must only enter your own node in this list!**. This way your bot will recognize that it is a directory node and it avoids weird edge case behaviour (so don't add *other* known directory nodes; you won't be talking to them).
A natural retort is: but I don't know my own node's onion service hostname before I start it the first time. Indeed. So, just run it once with the default `directory_nodes` entries, then note down the new onion service hostname you created, and insert that as the only entry in the list.
#### Suggested setup of a systemd service:
The most basic bare-bones service seems to work fine here:
```
[Unit]
Description=My JM signet directory node
Requires=network-online.target
After=network-online.target
[Service]
Type=simple
ExecStart=/bin/bash -c 'cd /path/to/joinmarket-clientserver && source jmvenv/bin/activate && cd scripts && python start-dn.py --datadir=/path/to/chosen/datadir'
User=user
[Install]
WantedBy=multi-user.target
```
... however, you need to kind of 'bootstrap' it the first time. For example:
* run once with systemctl start
* look at log with `journalctl`, service fails due to default `joinmarket.cfg` and quit.
* go to that cfg file. Remove the IRC settings, they serve no purpose here. Change the `hidden_service_dir` to `/yourlocation/hidserv` (the actual directory need not exist, it's better if it doesn't, this first time). Edit the `network` field in `BLOCKCHAIN` to whatever network (mainnet, signet) you intend to support - it can be only one for one directory node, for now.
* `systemctl start` again, now note the onion hostname created from the log or the directory
* set that hostname in `directory_nodes` in `joinmarket.cfg`
* now the service should start correctly
TODO: add some material on network hardening/firewalls here, I guess.

4
jmbase/jmbase/commands.py

@ -27,11 +27,11 @@ class JMInit(JMCommand):
"""Communicates the client's required setup
configuration.
Blockchain source is communicated only as a naming
tag for messagechannels (currently IRC 'realname' field).
tag for messagechannels (for IRC, 'realname' field).
"""
arguments = [(b'bcsource', Unicode()),
(b'network', Unicode()),
(b'irc_configs', JsonEncodable()),
(b'chan_configs', JsonEncodable()),
(b'minmakers', Integer()),
(b'maker_timeout_sec', Integer()),
(b'dust_threshold', Integer()),

77
jmbase/jmbase/twisted_utils.py

@ -128,16 +128,23 @@ def config_to_hs_ports(virtual_port, host, port):
class JMHiddenService(object):
""" Wrapper class around the actions needed to
create and serve on a hidden service; an object of
type Resource must be provided in the constructor,
which does the HTTP serving actions (GET, POST serving).
type either Resource or server.ProtocolFactory must
be provided in the constructor, which does the HTTP
(GET, POST) or other protocol serving actions.
"""
def __init__(self, resource, info_callback, error_callback,
onion_hostname_callback, tor_control_host,
def __init__(self, proto_factory_or_resource, info_callback,
error_callback, onion_hostname_callback, tor_control_host,
tor_control_port, serving_host, serving_port,
virtual_port = None,
shutdown_callback = None):
self.site = Site(resource)
self.site.displayTracebacks = False
virtual_port=None,
shutdown_callback=None,
hidden_service_dir=""):
if isinstance(proto_factory_or_resource, Resource):
# TODO bad naming, in this case it doesn't start
# out as a protocol factory; a Site is one, a Resource isn't.
self.proto_factory = Site(proto_factory_or_resource)
self.proto_factory.displayTracebacks = False
else:
self.proto_factory = proto_factory_or_resource
self.info_callback = info_callback
self.error_callback = error_callback
# this has a separate callback for convenience, it should
@ -155,6 +162,13 @@ class JMHiddenService(object):
# config object, so no default here:
self.serving_host = serving_host
self.serving_port = serving_port
# this is used to serve an onion from the filesystem,
# NB: Because of how txtorcon is set up, this option
# uses a *separate tor instance* owned by the owner of
# this script (because txtorcon needs to read the
# HS dir), whereas if this option is "", we set up
# an ephemeral HS on the global or pre-existing tor.
self.hidden_service_dir = hidden_service_dir
def start_tor(self):
""" This function executes the workflow
@ -162,19 +176,31 @@ class JMHiddenService(object):
"""
self.info_callback("Attempting to start onion service on port: {} "
"...".format(self.virtual_port))
if str(self.tor_control_host).startswith('unix:'):
control_endpoint = UNIXClientEndpoint(reactor,
self.tor_control_host[5:])
if self.hidden_service_dir == "":
if str(self.tor_control_host).startswith('unix:'):
control_endpoint = UNIXClientEndpoint(reactor,
self.tor_control_host[5:])
else:
control_endpoint = TCP4ClientEndpoint(reactor,
self.tor_control_host, self.tor_control_port)
d = txtorcon.connect(reactor, control_endpoint)
d.addCallback(self.create_onion_ep)
d.addErrback(self.setup_failed)
# TODO: add errbacks to the next two calls in
# the chain:
d.addCallback(self.onion_listen)
d.addCallback(self.print_host)
else:
control_endpoint = TCP4ClientEndpoint(reactor,
self.tor_control_host, self.tor_control_port)
d = txtorcon.connect(reactor, control_endpoint)
d.addCallback(self.create_onion_ep)
d.addErrback(self.setup_failed)
# TODO: add errbacks to the next two calls in
# the chain:
d.addCallback(self.onion_listen)
d.addCallback(self.print_host)
ep = "onion:" + str(self.virtual_port) + ":localPort="
ep += str(self.serving_port)
# endpoints.TCPHiddenServiceEndpoint creates version 2 by
# default for backwards compat (err, txtorcon needs to update that ...)
ep += ":version=3"
ep += ":hiddenServiceDir="+self.hidden_service_dir
onion_endpoint = serverFromString(reactor, ep)
d = onion_endpoint.listen(self.proto_factory)
d.addCallback(self.print_host_filesystem)
def setup_failed(self, arg):
# Note that actions based on this failure are deferred to callers:
@ -195,7 +221,8 @@ class JMHiddenService(object):
serverstring = "tcp:{}:interface={}".format(self.serving_port,
self.serving_host)
onion_endpoint = serverFromString(reactor, serverstring)
return onion_endpoint.listen(self.site)
print("created the onion endpoint, now calling listen")
return onion_endpoint.listen(self.proto_factory)
def print_host(self, ep):
""" Callback fired once the HS is available
@ -204,6 +231,14 @@ class JMHiddenService(object):
"""
self.onion_hostname_callback(self.onion.hostname)
def print_host_filesystem(self, port):
""" As above but needed to respect slightly different
callback chain for this case (where we start our own tor
instance for the filesystem-based onion).
"""
self.onion = port.onion_service
self.onion_hostname_callback(self.onion.hostname)
def shutdown(self):
self.tor_connection.protocol.transport.loseConnection()
self.info_callback("Hidden service shutdown complete")

6
jmbase/test/test_commands.py

@ -43,9 +43,9 @@ def end_test():
class JMTestServerProtocol(JMBaseProtocol):
@JMInit.responder
def on_JM_INIT(self, bcsource, network, irc_configs, minmakers,
def on_JM_INIT(self, bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location):
show_receipt("JMINIT", bcsource, network, irc_configs, minmakers,
show_receipt("JMINIT", bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location)
d = self.callRemote(JMInitProto,
nick_hash_length=1,
@ -137,7 +137,7 @@ class JMTestClientProtocol(JMBaseProtocol):
d = self.callRemote(JMInit,
bcsource="dummyblockchain",
network="dummynetwork",
irc_configs=['dummy', 'irc', 'config'],
chan_configs=['dummy', 'irc', 'config'],
minmakers=7,
maker_timeout_sec=8,
dust_threshold=1500,

4
jmclient/jmclient/__init__.py

@ -24,7 +24,7 @@ from .cryptoengine import (BTCEngine, BTC_P2PKH, BTC_P2SH_P2WPKH, BTC_P2WPKH, En
TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, detect_script_type)
from .configure import (load_test_config, process_shutdown,
load_program_config, jm_single, get_network, update_persist_config,
validate_address, is_burn_destination, get_irc_mchannels,
validate_address, is_burn_destination, get_mchannels,
get_blockchain_interface_instance, set_config, is_segwit_mode,
is_native_segwit_mode, JMPluginService, get_interest_rate,
get_bondless_makers_allowance, check_and_start_tor)
@ -33,7 +33,7 @@ from .blockchaininterface import (BlockchainInterface,
from .snicker_receiver import SNICKERError, SNICKERReceiver
from .client_protocol import (JMTakerClientProtocol, JMClientProtocolFactory,
start_reactor, SNICKERClientProtocolFactory,
BIP78ClientProtocolFactory,
BIP78ClientProtocolFactory, JMMakerClientProtocol,
get_daemon_serving_params)
from .podle import (set_commitment_file, get_commitment_file,
add_external_commitments,

19
jmclient/jmclient/client_protocol.py

@ -15,7 +15,7 @@ import os
import sys
from jmbase import (get_log, EXIT_FAILURE, hextobin, bintohex,
utxo_to_utxostr, bdict_sdict_convert)
from jmclient import (jm_single, get_irc_mchannels,
from jmclient import (jm_single, get_mchannels,
RegtestBitcoinCoreInterface,
SNICKERReceiver, process_shutdown)
import jmbitcoin as btc
@ -434,7 +434,7 @@ class JMMakerClientProtocol(JMClientProtocol):
"blockchain_source")
#needed only for channel naming convention
network = jm_single().config.get("BLOCKCHAIN", "network")
irc_configs = get_irc_mchannels()
chan_configs = self.factory.get_mchannels(mode="MAKER")
#only here because Init message uses this field; not used by makers TODO
minmakers = jm_single().config.getint("POLICY", "minimum_makers")
maker_timeout_sec = jm_single().maker_timeout_sec
@ -442,7 +442,7 @@ class JMMakerClientProtocol(JMClientProtocol):
d = self.callRemote(commands.JMInit,
bcsource=blockchain_source,
network=network,
irc_configs=irc_configs,
chan_configs=chan_configs,
minmakers=minmakers,
maker_timeout_sec=maker_timeout_sec,
dust_threshold=jm_single().DUST_THRESHOLD,
@ -601,7 +601,7 @@ class JMTakerClientProtocol(JMClientProtocol):
"blockchain_source")
#needed only for channel naming convention
network = jm_single().config.get("BLOCKCHAIN", "network")
irc_configs = get_irc_mchannels()
chan_configs = self.factory.get_mchannels(mode="TAKER")
minmakers = jm_single().config.getint("POLICY", "minimum_makers")
maker_timeout_sec = jm_single().maker_timeout_sec
@ -614,7 +614,7 @@ class JMTakerClientProtocol(JMClientProtocol):
d = self.callRemote(commands.JMInit,
bcsource=blockchain_source,
network=network,
irc_configs=irc_configs,
chan_configs=chan_configs,
minmakers=minmakers,
maker_timeout_sec=maker_timeout_sec,
dust_threshold=jm_single().DUST_THRESHOLD,
@ -789,12 +789,21 @@ class JMClientProtocolFactory(protocol.ClientFactory):
def setClient(self, client):
self.proto_client = client
def getClient(self):
return self.proto_client
def buildProtocol(self, addr):
return self.protocol(self, self.client)
def get_mchannels(self, mode):
""" A transparent wrapper that allows override,
so that a script can return a customised set of
message channel configs; currently used for testing
multiple bots on regtest.
"""
return get_mchannels(mode)
def start_reactor(host, port, factory=None, snickerfactory=None,
bip78=False, jm_coinjoin=True, ish=True,
daemon=False, rs=True, gui=False): #pragma: no cover

188
jmclient/jmclient/configure.py

@ -141,53 +141,99 @@ rpc_password = password
# information.
rpc_wallet_file =
## SERVER 1/3) Darkscience IRC (Tor, IP)
[MESSAGING:onion]
# onion based message channels must have the exact type 'onion'
# (while the section name above can be MESSAGING:whatever), and there must
# be only ONE such message channel configured (note the directory servers
# can be multiple, below):
type = onion
socks5_host = localhost
socks5_port = 9050
# the tor control configuration.
# for most people running the tor daemon
# on Linux, no changes are required here:
tor_control_host = localhost
# or, to use a UNIX socket
# tor_control_host = unix:/var/run/tor/control
tor_control_port = 9051
# the host/port actually serving the hidden service
# (note the *virtual port*, that the client uses,
# is hardcoded to 80):
onion_serving_host = 127.0.0.1
onion_serving_port = 8080
# directory node configuration
#
# This is mandatory for directory nodes (who must also set their
# own *.onion:port as the only directory in directory_nodes, below),
# but NOT TO BE USED by non-directory nodes (which is you, unless
# you know otherwise!), as it will greatly degrade your privacy.
# (note the default is no value, don't replace it with "").
hidden_service_dir =
#
# This is a comma separated list (comma can be omitted if only one item).
# Each item has format host:port ; both are required, though port will
# be 80 if created in this code.
# for MAINNET:
directory_nodes = 3kxw6lf5vf6y26emzwgibzhrzhmhqiw6ekrek3nqfjjmhwznb2moonad.onion:80,qqd22cwgygaxcy6vdw6mzwkyaxg5urb4ptbc5d74nrj25phspajxjbqd.onion:80
# for SIGNET (testing network):
# directory_nodes = rr6f6qtleiiwic45bby4zwmiwjrj3jsbmcvutwpqxjziaydjydkk5iad.onion:80,k74oyetjqgcamsyhlym2vgbjtvhcrbxr4iowd4nv4zk5sehw4v665jad.onion:80
# This setting is ONLY for developer regtest setups,
# running multiple bots at once. Don't alter it otherwise
regtest_count = 0,0
## IRC SERVER 1: Darkscience IRC (Tor, IP)
################################################################################
[MESSAGING:server1]
# by default the legacy format without a `type` field is
# understood to be IRC, but you can, optionally, add it:
# type = irc
channel = joinmarket-pit
port = 6697
usessl = true
# For traditional IP (default):
host = irc.darkscience.net
socks5 = false
# For traditional IP:
#host = irc.darkscience.net
#socks5 = false
# For Tor (recommended as clearnet alternative):
#host = darkirc6tqgpnwd3blln3yfv5ckl47eg7llfxkmtovrv7c7iwohhb6ad.onion
#socks5 = true
#socks5_host = localhost
#socks5_port = 9050
host = darkirc6tqgpnwd3blln3yfv5ckl47eg7llfxkmtovrv7c7iwohhb6ad.onion
socks5 = true
socks5_host = localhost
socks5_port = 9050
## SERVER 2/3) hackint IRC (Tor, IP)
## IRC SERVER 2: ILITA IRC (optional IRC alternate, Tor only)
################################################################################
[MESSAGING:server2]
channel = joinmarket-pit
port = 6667
usessl = false
socks5 = true
socks5_host = localhost
# For traditional IP (default):
host = irc.hackint.org
port = 6697
usessl = true
socks5 = false
# For Tor (recommended as clearnet alternative):
#host = ncwkrwxpq2ikcngxq3dy2xctuheniggtqeibvgofixpzvrwpa77tozqd.onion
#port = 6667
#usessl = false
#socks5 = true
#socks5_host = localhost
#socks5_port = 9050
host = ilitafrzzgxymv6umx2ux7kbz3imyeko6cnqkvy4nisjjj4qpqkrptid.onion
socks5_port = 9050
## SERVER 3/3) ILITA IRC (Tor - disabled by default)
## IRC SERVER 3: (backup) hackint IRC (Tor, IP)
################################################################################
#[MESSAGING:server3]
#channel = joinmarket-pit
# channel = joinmarket-pit
# For traditional IP:
## host = irc.hackint.org
## port = 6697
## usessl = true
## socks5 = false
# For Tor (default):
#host = ncwkrwxpq2ikcngxq3dy2xctuheniggtqeibvgofixpzvrwpa77tozqd.onion
#port = 6667
#usessl = false
#socks5 = true
#socks5_host = localhost
# For Tor (recommended):
#host = ilitafrzzgxymv6umx2ux7kbz3imyeko6cnqkvy4nisjjj4qpqkrptid.onion
#socks5_port = 9050
[LOGGING]
@ -488,7 +534,7 @@ def set_config(cfg, bcint=None):
global_singleton.bc_interface = bcint
def get_irc_mchannels():
def get_mchannels(mode="TAKER"):
SECTION_NAME = 'MESSAGING'
# FIXME: remove in future release
if jm_single().config.has_section(SECTION_NAME):
@ -499,34 +545,64 @@ def get_irc_mchannels():
return _get_irc_mchannels_old()
SECTION_NAME += ':'
irc_sections = []
for s in jm_single().config.sections():
if s.startswith(SECTION_NAME):
irc_sections.append(s)
assert irc_sections
req_fields = [("host", str), ("port", int), ("channel", str), ("usessl", str)]
irc_fields = [("host", str), ("port", int), ("channel", str), ("usessl", str),
("socks5", str), ("socks5_host", str), ("socks5_port", int)]
onion_fields = [("type", str), ("directory_nodes", str), ("regtest_count", str),
("socks5_host", str), ("socks5_port", int),
("tor_control_host", str), ("tor_control_port", int),
("onion_serving_host", str), ("onion_serving_port", int),
("hidden_service_dir", str)]
configs = []
for section in irc_sections:
def get_irc_section(s):
server_data = {}
# check if socks5 is enabled for tor and load relevant config if so
try:
server_data["socks5"] = jm_single().config.get(section, "socks5")
server_data["socks5"] = jm_single().config.get(s, "socks5")
except NoOptionError:
server_data["socks5"] = "false"
if server_data["socks5"].lower() == 'true':
server_data["socks5_host"] = jm_single().config.get(section, "socks5_host")
server_data["socks5_port"] = jm_single().config.get(section, "socks5_port")
server_data["socks5_host"] = jm_single().config.get(s, "socks5_host")
server_data["socks5_port"] = jm_single().config.get(s, "socks5_port")
for option, otype in req_fields:
val = jm_single().config.get(section, option)
for option, otype in irc_fields:
val = jm_single().config.get(s, option)
server_data[option] = otype(val)
server_data['btcnet'] = get_network()
configs.append(server_data)
return configs
return server_data
def get_onion_section(s):
onion_data = {}
for option, otype in onion_fields:
try:
val = jm_single().config.get(s, option)
except NoOptionError:
continue
onion_data[option] = otype(val)
# the onion messaging section must specify whether
# to serve an onion:
onion_data["serving"] = mode == "MAKER"
onion_data['btcnet'] = get_network()
# Just to allow a dynamic set of var:
onion_data["section-name"] = s
return onion_data
onion_sections = []
irc_sections = []
for section in jm_single().config.sections():
if not section.startswith(SECTION_NAME):
continue
if jm_single().config.has_option(section, "type"):
channel_type = jm_single().config.get(section, "type").lower()
if channel_type == "onion":
onion_sections.append(get_onion_section(section))
elif channel_type == "irc":
irc_sections.append(get_irc_section(section))
else:
irc_sections.append(get_irc_section(section))
assert irc_sections or onion_sections
assert len(onion_sections) < 2
return irc_sections + onion_sections
def _get_irc_mchannels_old():
fields = [("host", str), ("port", int), ("channel", str), ("usessl", str),
@ -655,28 +731,6 @@ def load_program_config(config_path="", bs=None, plugin_services=[]):
"settings and restart joinmarket.", "info")
sys.exit(EXIT_FAILURE)
#These are left as sanity checks but currently impossible
#since any edits are overlays to the default, these sections/options will
#always exist.
# FIXME: This check is a best-effort attempt. Certain incorrect section
# names can pass and so can non-first invalid sections.
for s in required_options: #pragma: no cover
# check for sections
avail = None
if not global_singleton.config.has_section(s):
for avail in global_singleton.config.sections():
if avail.startswith(s):
break
else:
raise Exception(
"Config file does not contain the required section: " + s)
# then check for specific options
k = avail or s
for o in required_options[s]:
if not global_singleton.config.has_option(k, o):
raise Exception("Config file does not contain the required "
"option '{}' in section '{}'.".format(o, k))
loglevel = global_singleton.config.get("LOGGING", "console_log_level")
try:
set_logging_level(loglevel)

7
jmclient/jmclient/wallet_rpc.py

@ -159,6 +159,9 @@ class JMWalletDaemon(Service):
# can be shut down cleanly:
self.coinjoin_connection = None
def get_client_factory(self):
return JMClientProtocolFactory(self.taker)
def activate_coinjoin_state(self, state):
""" To be set when a maker or taker
operation is initialized; they cannot
@ -1003,13 +1006,13 @@ class JMWalletDaemon(Service):
self.taker = Taker(self.services["wallet"], schedule,
max_cj_fee = max_cj_fee,
callbacks=(self.filter_orders_callback,
None, self.taker_finished))
None, self.taker_finished))
# TODO ; this makes use of a pre-existing hack to allow
# selectively disabling the stallMonitor function that checks
# if transactions went through or not; here we want to cleanly
# destroy the Taker after an attempt is made, successful or not.
self.taker.testflag = True
self.clientfactory = JMClientProtocolFactory(self.taker)
self.clientfactory = self.get_client_factory()
dhost, dport = self.check_daemon_ready()

4
jmclient/test/test_client_protocol.py

@ -167,9 +167,9 @@ def end_test():
class JMTestServerProtocol(JMBaseProtocol):
@JMInit.responder
def on_JM_INIT(self, bcsource, network, irc_configs, minmakers,
def on_JM_INIT(self, bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location):
show_receipt("JMINIT", bcsource, network, irc_configs, minmakers,
show_receipt("JMINIT", bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location)
d = self.callRemote(JMInitProto,
nick_hash_length=1,

1
jmdaemon/jmdaemon/__init__.py

@ -4,6 +4,7 @@ from .protocol import *
from .enc_wrapper import as_init_encryption, decode_decrypt, \
encrypt_encode, init_keypair, init_pubkey, get_pubkey, NaclError
from .irc import IRCMessageChannel
from .onionmc import OnionMessageChannel
from jmbase.support import get_log
from .message_channel import MessageChannel, MessageChannelCollection
from .orderbookwatch import OrderbookWatch

30
jmdaemon/jmdaemon/daemon_protocol.py

@ -7,8 +7,9 @@ from .enc_wrapper import (as_init_encryption, init_keypair, init_pubkey,
from .protocol import (COMMAND_PREFIX, ORDER_KEYS, NICK_HASH_LENGTH,
NICK_MAX_ENCODED, JM_VERSION, JOINMARKET_NICK_HEADER,
COMMITMENT_PREFIXES)
from .irc import IRCMessageChannel
from .irc import IRCMessageChannel
from .onionmc import OnionMessageChannel
from jmbase import (is_hs_uri, get_tor_agent, JMHiddenService,
get_nontor_agent, BytesProducer, wrapped_urlparse,
bdict_sdict_convert, JMHTTPResource)
@ -474,7 +475,7 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
self.factory = factory
self.jm_state = 0
self.restart_mc_required = False
self.irc_configs = None
self.chan_configs = None
self.mcc = None
#Default role is TAKER; must be overriden to MAKER in JMSetup message.
self.role = "TAKER"
@ -503,7 +504,7 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
d.addErrback(self.defaultErrback)
@JMInit.responder
def on_JM_INIT(self, bcsource, network, irc_configs, minmakers,
def on_JM_INIT(self, bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location):
"""Reads in required configuration from client for a new
session; feeds back joinmarket messaging protocol constants
@ -517,20 +518,25 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
self.dust_threshold = int(dust_threshold)
#(bitcoin) network only referenced in channel name construction
self.network = network
if irc_configs == self.irc_configs:
if chan_configs == self.chan_configs:
self.restart_mc_required = False
log.msg("New init received did not require a new message channel"
" setup.")
else:
if self.irc_configs:
if self.chan_configs:
#close the existing connections
self.mc_shutdown()
self.irc_configs = irc_configs
self.chan_configs = chan_configs
self.restart_mc_required = True
mcs = [IRCMessageChannel(c,
daemon=self,
realname='btcint=' + bcsource)
for c in self.irc_configs]
mcs = []
for c in self.chan_configs:
if "type" in c and c["type"] == "onion":
mcs.append(OnionMessageChannel(c, daemon=self))
else:
# default is IRC; TODO allow others
mcs.append(IRCMessageChannel(c,
daemon=self,
realname='btcint=' + bcsource))
self.mcc = MessageChannelCollection(mcs)
OrderbookWatch.set_msgchan(self, self.mcc)
#register taker-specific msgchan callbacks here
@ -946,7 +952,8 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
for a new transaction; effectively means any previous
incomplete transaction is wiped.
"""
self.jm_state = 0 #uninited
self.jm_state = 0
self.mcc.set_nick(nick)
if self.restart_mc_required:
self.mcc.run()
self.restart_mc_required = False
@ -954,7 +961,6 @@ class JMDaemonServerProtocol(amp.AMP, OrderbookWatch):
#if we are not restarting the MC,
#we must simulate the on_welcome message:
self.on_welcome()
self.mcc.set_nick(nick)
def transfer_commitment(self, commit):
"""Send this commitment via privmsg to one (random)

8
jmdaemon/jmdaemon/message_channel.py

@ -259,13 +259,13 @@ class MessageChannelCollection(object):
for x in self.available_channels()
if mc == x.hostid]
if len(matching_channels) != 1: #pragma: no cover
#this can happen if an IRC goes down shortly before a message
#this can happen if a m-channel goes down shortly before a message
#is supposed to be sent. There used to be an exception raise.
#to prevent a crash (especially in makers), we just inform
#the user about it for now
log.error("Tried to communicate on this IRC server but "
log.error("Tried to communicate on this message channel but "
"failed: " + str(mc))
log.error("You might have to comment out this IRC server "
log.error("You might have to comment out this message channel"
"in joinmarket.cfg and restart.")
log.error("No action needed for makers / yield generators!")
# todo: add logic to continue on other available mc
@ -444,7 +444,7 @@ class MessageChannelCollection(object):
if (not self.on_welcome_announce_id) and self.on_welcome:
self.on_welcome_announce_id = reactor.callLater(60, self.on_welcome_setup_finished,)
else:
log.info("All IRC servers connected, starting execution.")
log.info("All message channels connected, starting execution.")
if self.on_welcome_announce_id:
self.on_welcome_announce_id.cancel()
self.on_welcome_setup_finished()

1358
jmdaemon/jmdaemon/onionmc.py

File diff suppressed because it is too large Load Diff

8
jmdaemon/test/test_daemon_protocol.py

@ -7,7 +7,7 @@ from jmdaemon.daemon_protocol import JMDaemonServerProtocol
from jmdaemon.protocol import NICK_HASH_LENGTH, NICK_MAX_ENCODED, JM_VERSION,\
JOINMARKET_NICK_HEADER
from jmbase import get_log
from jmclient import (load_test_config, jm_single, get_irc_mchannels)
from jmclient import (load_test_config, jm_single, get_mchannels)
from twisted.python.log import msg as tmsg
from twisted.python.log import startLogging
from twisted.internet import protocol, reactor, task
@ -59,11 +59,11 @@ class JMTestClientProtocol(JMBaseProtocol):
def clientStart(self):
self.sigs_received = 0
irc = get_irc_mchannels()
chan_configs = [get_mchannels()[0]]
d = self.callRemote(JMInit,
bcsource="dummyblockchain",
network="dummynetwork",
irc_configs=irc,
chan_configs=chan_configs,
minmakers=2,
maker_timeout_sec=3,
dust_threshold=27300,
@ -212,7 +212,7 @@ class JMDaemonTestServerProtocol(JMDaemonServerProtocol):
return super().on_JM_REQUEST_OFFERS()
@JMInit.responder
def on_JM_INIT(self, bcsource, network, irc_configs, minmakers,
def on_JM_INIT(self, bcsource, network, chan_configs, minmakers,
maker_timeout_sec, dust_threshold, blacklist_location):
self.maker_timeout_sec = maker_timeout_sec
self.dust_threshold = int(dust_threshold)

6
jmdaemon/test/test_irc_messaging.py

@ -6,7 +6,7 @@ from twisted.trial import unittest
from twisted.internet import reactor, task
from jmdaemon import IRCMessageChannel, MessageChannelCollection
#needed for test framework
from jmclient import (load_test_config, get_irc_mchannels, jm_single)
from jmclient import (load_test_config, get_mchannels, jm_single)
si = 1
class DummyDaemon(object):
@ -95,7 +95,7 @@ def junk_fill(mc):
def getmc(nick):
dm = DummyDaemon()
mc = DummyMC(get_irc_mchannels()[0], nick, dm)
mc = DummyMC(get_mchannels()[0], nick, dm)
mc.register_orderbookwatch_callbacks(on_order_seen=on_order_seen)
mc.register_taker_callbacks(on_pubkey=on_pubkey)
mc.on_connect = on_connect
@ -108,7 +108,7 @@ class TrialIRC(unittest.TestCase):
def setUp(self):
load_test_config()
print(get_irc_mchannels()[0])
print(get_mchannels()[0])
jm_single().maker_timeout_sec = 1
dm, mc, mcc = getmc("irc_publisher")
dm2, mc2, mcc2 = getmc("irc_receiver")

4
jmdaemon/test/test_orderbookwatch.py

@ -2,7 +2,7 @@ import pytest
from jmdaemon.orderbookwatch import OrderbookWatch
from jmdaemon import IRCMessageChannel, fidelity_bond_cmd_list
from jmclient import get_irc_mchannels, load_test_config
from jmclient import get_mchannels, load_test_config
from jmdaemon.protocol import JM_VERSION, ORDER_KEYS
from jmbase.support import hextobin
from jmclient.fidelity_bond import FidelityBondProof
@ -24,7 +24,7 @@ def on_welcome(x):
def get_ob():
load_test_config()
dm = DummyDaemon()
mc = DummyMC(get_irc_mchannels()[0], "test", dm)
mc = DummyMC(get_mchannels()[0], "test", dm)
ob = OrderbookWatch()
ob.on_welcome = on_welcome
ob.set_msgchan(mc)

4
scripts/joinmarket-qt.py

@ -912,10 +912,10 @@ class SpendTab(QWidget):
daemon=daemon,
gui=True)
else:
#This will re-use IRC connections in background (daemon), no restart
#This will re-use message channels in background (daemon), no restart
self.clientfactory.getClient().client = self.taker
self.clientfactory.getClient().clientStart()
mainWindow.statusBar().showMessage("Connecting to IRC ...")
mainWindow.statusBar().showMessage("Connecting to message channels ...")
def takerInfo(self, infotype, infomsg):
if infotype == "INFO":

68
scripts/obwatch/ob-watcher.py

@ -44,8 +44,9 @@ if 'matplotlib' in sys.modules:
import matplotlib.pyplot as plt
from jmclient import jm_single, load_program_config, calc_cj_fee, \
get_irc_mchannels, add_base_options
from jmdaemon import OrderbookWatch, MessageChannelCollection, IRCMessageChannel
get_mchannels, add_base_options
from jmdaemon import (OrderbookWatch, MessageChannelCollection,
OnionMessageChannel, IRCMessageChannel)
#TODO this is only for base58, find a solution for a client without jmbitcoin
import jmbitcoin as btc
from jmdaemon.protocol import *
@ -737,32 +738,32 @@ class ObBasic(OrderbookWatch):
def request_orderbook(self):
self.msgchan.request_orderbook()
class ObIRCMessageChannel(IRCMessageChannel):
"""A customisation of the message channel
to allow receipt of privmsgs without the
verification hooks in client-daemon communication."""
def on_privmsg(self, nick, message):
if len(message) < 2:
return
if message[0] != COMMAND_PREFIX:
log.debug('message not a cmd')
return
cmd_string = message[1:].split(' ')[0]
if cmd_string not in offername_list:
log.debug('non-offer ignored')
return
#Ignore sigs (TODO better to include check)
sig = message[1:].split(' ')[-2:]
#reconstruct original message without cmd pref
rawmessage = ' '.join(message[1:].split(' ')[:-2])
for command in rawmessage.split(COMMAND_PREFIX):
_chunks = command.split(" ")
try:
self.check_for_orders(nick, _chunks)
self.check_for_fidelity_bond(nick, _chunks)
except:
pass
"""An override for MessageChannel classes,
to allow receipt of privmsgs without the
verification hooks in client-daemon communication."""
def on_privmsg(inst, nick, message):
if len(message) < 2:
return
if message[0] != COMMAND_PREFIX:
log.debug('message not a cmd')
return
cmd_string = message[1:].split(' ')[0]
if cmd_string not in offername_list:
log.debug('non-offer ignored')
return
#Ignore sigs (TODO better to include check)
sig = message[1:].split(' ')[-2:]
#reconstruct original message without cmd pref
rawmessage = ' '.join(message[1:].split(' ')[:-2])
for command in rawmessage.split(COMMAND_PREFIX):
_chunks = command.split(" ")
try:
inst.check_for_orders(nick, _chunks)
inst.check_for_fidelity_bond(nick, _chunks)
except:
pass
def get_dummy_nick():
@ -805,7 +806,16 @@ def main():
load_program_config(config_path=options.datadir)
check_and_start_tor()
hostport = (options.host, options.port)
mcs = [ObIRCMessageChannel(c) for c in get_irc_mchannels()]
mcs = []
chan_configs = get_mchannels()
for c in chan_configs:
if "type" in c and c["type"] == "onion":
mcs.append(OnionMessageChannel(c))
else:
# default is IRC; TODO allow others
mcs.append(IRCMessageChannel(c))
IRCMessageChannel.on_privmsg = on_privmsg
OnionMessageChannel.on_privmsg = on_privmsg
mcc = MessageChannelCollection(mcs)
mcc.set_nick(get_dummy_nick())
taker = ObBasic(mcc, hostport)

340
test/e2e-coinjoin-test.py

@ -0,0 +1,340 @@
#! /usr/bin/env python
'''Creates wallets and yield generators in regtest,
then runs both them and a JMWalletDaemon instance
for the taker, injecting the newly created taker
wallet into it and running sendpayment once.
Number of ygs is configured in the joinmarket.cfg
with `regtest-count` in the `ln-onion` type MESSAGING
section.
See notes below for more detail on config.
Run it like:
pytest \
--btcroot=/path/to/bitcoin/bin/ \
--btcpwd=123456abcdef --btcconf=/blah/bitcoin.conf \
-s test/e2e-coinjoin-test.py
'''
from twisted.internet import reactor, defer
from twisted.web.client import readBody, Headers
from common import make_wallets
import pytest
import random
import json
from datetime import datetime
from jmbase import (get_nontor_agent, BytesProducer, jmprint,
get_log, stop_reactor)
from jmclient import (YieldGeneratorBasic, load_test_config, jm_single,
JMClientProtocolFactory, start_reactor, SegwitWallet, get_mchannels,
SegwitLegacyWallet, JMWalletDaemon)
from jmclient.wallet_utils import wallet_gettimelockaddress
from jmclient.wallet_rpc import api_version_string
log = get_log()
# For quicker testing, restrict the range of timelock
# addresses to avoid slow load of multiple bots.
# Note: no need to revert this change as test runs
# in isolation.
from jmclient import FidelityBondMixin
FidelityBondMixin.TIMELOCK_ERA_YEARS = 2
FidelityBondMixin.TIMELOCK_EPOCH_YEAR = datetime.now().year
FidelityBondMixin.TIMENUMBERS_PER_PUBKEY = 12
wallet_name = "test-onion-yg-runner.jmdat"
mean_amt = 2.0
directory_node_indices = [1]
def get_onion_messaging_config_regtest(run_num: int, dns=[1], hsd="", mode="TAKER"):
""" Sets a onion messaging channel section for a regtest instance
indexed by `run_num`. The indices to be used as directory nodes
should be passed as `dns`, as a list of ints.
"""
def location_string(directory_node_run_num):
return "127.0.0.1:" + str(
8080 + directory_node_run_num)
if run_num in dns:
# means *we* are a dn, and dns currently
# do not use other dns:
dns_to_use = [location_string(run_num)]
else:
dns_to_use = [location_string(a) for a in dns]
dn_nodes_list = ",".join(dns_to_use)
log.info("For node: {}, set dn list to: {}".format(run_num, dn_nodes_list))
cf = {"type": "onion",
"btcnet": "testnet",
"socks5_host": "127.0.0.1",
"socks5_port": 9050,
"tor_control_host": "127.0.0.1",
"tor_control_port": 9051,
"onion_serving_host": "127.0.0.1",
"onion_serving_port": 8080 + run_num,
"hidden_service_dir": "",
"directory_nodes": dn_nodes_list,
"regtest_count": "1, 1"}
if mode == "MAKER":
cf["serving"] = True
else:
cf["serving"] = False
if run_num in dns:
# only directories need to use fixed hidden service directories:
cf["hidden_service_dir"] = hsd
return cf
class RegtestJMClientProtocolFactory(JMClientProtocolFactory):
i = 1
def set_directory_nodes(self, dns):
# a list of integers representing the directory nodes
# for this test:
self.dns = dns
def get_mchannels(self, mode="TAKER"):
# swaps out any existing onionmc configs
# in the config settings on startup, for one
# that's indexed to the regtest counter var:
default_chans = get_mchannels(mode=mode)
new_chans = []
onion_found = False
hsd = ""
for c in default_chans:
if "type" in c and c["type"] == "onion":
onion_found = True
if c["hidden_service_dir"] != "":
hsd = c["hidden_service_dir"]
continue
else:
new_chans.append(c)
if onion_found:
new_chans.append(get_onion_messaging_config_regtest(
self.i, self.dns, hsd, mode=mode))
return new_chans
class JMWalletDaemonT(JMWalletDaemon):
def check_cookie(self, request):
if self.auth_disabled:
return True
return super().check_cookie(request)
class TWalletRPCManager(object):
""" Base class for set up of tests of the
Wallet RPC calls using the wallet_rpc.JMWalletDaemon service.
"""
# the port for the jmwallet daemon
dport = 28183
# the port for the ws
wss_port = 28283
def __init__(self):
# a client connnection object which is often but not always
# instantiated:
self.client_connector = None
self.daemon = JMWalletDaemonT(self.dport, self.wss_port, tls=False)
self.daemon.auth_disabled = True
# because we sync and start the wallet service manually here
# (and don't use wallet files yet), we won't have set a wallet name,
# so we set it here:
self.daemon.wallet_name = wallet_name
def start(self):
r, s = self.daemon.startService()
self.listener_rpc = r
self.listener_ws = s
def get_route_root(self):
addr = "http://127.0.0.1:" + str(self.dport)
addr += api_version_string
return addr
def stop(self):
for dc in reactor.getDelayedCalls():
dc.cancel()
d1 = defer.maybeDeferred(self.listener_ws.stopListening)
d2 = defer.maybeDeferred(self.listener_rpc.stopListening)
if self.client_connector:
self.client_connector.disconnect()
# only fire if everything is finished:
return defer.gatherResults([d1, d2])
@defer.inlineCallbacks
def do_request(self, agent, method, addr, body, handler, token=None):
if token:
headers = Headers({"Authorization": ["Bearer " + self.jwt_token]})
else:
headers = None
response = yield agent.request(method, addr, headers, bodyProducer=body)
yield self.response_handler(response, handler)
@defer.inlineCallbacks
def response_handler(self, response, handler):
body = yield readBody(response)
# these responses should always be 200 OK.
#assert response.code == 200
# handlers check the body is as expected; no return.
yield handler(body)
return True
def test_start_yg_and_taker_setup(setup_onion_ygrunner):
"""Set up some wallets, for the ygs and 1 taker.
Then start LN and the ygs in the background, then fire
a startup of a wallet daemon for the taker who then
makes a coinjoin payment.
"""
if jm_single().config.get("POLICY", "native") == "true":
walletclass = SegwitWallet
else:
# TODO add Legacy
walletclass = SegwitLegacyWallet
start_bot_num, end_bot_num = [int(x) for x in jm_single().config.get(
"MESSAGING:onion", "regtest_count").split(",")]
num_ygs = end_bot_num - start_bot_num
# specify the number of wallets and bots of each type:
wallet_services = make_wallets(num_ygs + 1,
wallet_structures=[[1, 3, 0, 0, 0]] * (num_ygs + 1),
mean_amt=2.0,
walletclass=walletclass)
#the sendpayment bot uses the last wallet in the list
wallet_service = wallet_services[end_bot_num - 1]['wallet']
jmprint("\n\nTaker wallet seed : " + wallet_services[end_bot_num - 1]['seed'])
# for manual audit if necessary, show the maker's wallet seeds
# also (note this audit should be automated in future, see
# test_full_coinjoin.py in this directory)
jmprint("\n\nMaker wallet seeds: ")
for i in range(start_bot_num, end_bot_num):
jmprint("Maker seed: " + wallet_services[i - 1]['seed'])
jmprint("\n")
wallet_service.sync_wallet(fast=True)
ygclass = YieldGeneratorBasic
# As per previous note, override non-default command line settings:
options = {}
for x in ["ordertype", "txfee_contribution", "txfee_contribution_factor",
"cjfee_a", "cjfee_r", "cjfee_factor", "minsize", "size_factor"]:
options[x] = jm_single().config.get("YIELDGENERATOR", x)
ordertype = options["ordertype"]
txfee_contribution = int(options["txfee_contribution"])
txfee_contribution_factor = float(options["txfee_contribution_factor"])
cjfee_factor = float(options["cjfee_factor"])
size_factor = float(options["size_factor"])
if ordertype == 'reloffer':
cjfee_r = options["cjfee_r"]
# minimum size is such that you always net profit at least 20%
#of the miner fee
minsize = max(int(1.2 * txfee_contribution / float(cjfee_r)),
int(options["minsize"]))
cjfee_a = None
elif ordertype == 'absoffer':
cjfee_a = int(options["cjfee_a"])
minsize = int(options["minsize"])
cjfee_r = None
else:
assert False, "incorrect offertype config for yieldgenerator."
txtype = wallet_service.get_txtype()
if txtype == "p2wpkh":
prefix = "sw0"
elif txtype == "p2sh-p2wpkh":
prefix = "sw"
elif txtype == "p2pkh":
prefix = ""
else:
assert False, "Unsupported wallet type for yieldgenerator: " + txtype
ordertype = prefix + ordertype
for i in range(start_bot_num, end_bot_num):
cfg = [txfee_contribution, cjfee_a, cjfee_r, ordertype, minsize,
txfee_contribution_factor, cjfee_factor, size_factor]
wallet_service_yg = wallet_services[i - 1]["wallet"]
wallet_service_yg.startService()
yg = ygclass(wallet_service_yg, cfg)
clientfactory = RegtestJMClientProtocolFactory(yg, proto_type="MAKER")
# This ensures that the right rpc/port config is passed into the daemon,
# for this specific bot:
clientfactory.i = i
# This ensures that this bot knows which other bots are directory nodes:
clientfactory.set_directory_nodes(directory_node_indices)
nodaemon = jm_single().config.getint("DAEMON", "no_daemon")
daemon = bool(nodaemon)
#rs = True if i == num_ygs - 1 else False
start_reactor(jm_single().config.get("DAEMON", "daemon_host"),
jm_single().config.getint("DAEMON", "daemon_port"),
clientfactory, daemon=daemon, rs=False)
reactor.callLater(1.0, start_test_taker, wallet_services[end_bot_num - 1]['wallet'], end_bot_num, num_ygs)
reactor.run()
@defer.inlineCallbacks
def start_test_taker(wallet_service, i, num_ygs):
# this rpc manager has auth disabled,
# and the wallet_service is set manually,
# so no unlock etc.
mgr = TWalletRPCManager()
mgr.daemon.services["wallet"] = wallet_service
# because we are manually setting the wallet_service
# of the JMWalletDaemon instance, we do not follow the
# usual flow of `initialize_wallet_service`, we do not set
# the auth token or start the websocket; so we must manually
# sync the wallet, including bypassing any restart callback:
def dummy_restart_callback(msg):
log.warn("Ignoring rescan request from backend wallet service: " + msg)
mgr.daemon.services["wallet"].add_restart_callback(dummy_restart_callback)
mgr.daemon.wallet_name = wallet_name
mgr.daemon.services["wallet"].startService()
def get_client_factory():
clientfactory = RegtestJMClientProtocolFactory(mgr.daemon.taker,
proto_type="TAKER")
clientfactory.i = i
clientfactory.set_directory_nodes(directory_node_indices)
return clientfactory
mgr.daemon.get_client_factory = get_client_factory
# before preparing the RPC call to the wallet daemon,
# we decide a coinjoin destination, counterparty count and amount.
# Choosing a destination in the wallet is a bit easier because
# we can query the mixdepth balance at the end.
coinjoin_destination = mgr.daemon.services["wallet"].get_internal_addr(4)
cj_amount = 22000000
def n_cps_from_n_ygs(n):
if n > 4:
return n - 2
if n > 2:
return 2
assert False, "Need at least 3 yield generators to test"
n_cps = n_cps_from_n_ygs(num_ygs)
# once the taker is finished we sanity check before
# shutting down:
def dummy_taker_finished(res, fromtx=False,
waittime=0.0, txdetails=None):
jmprint("Taker is finished")
# check that the funds have arrived.
mbal = mgr.daemon.services["wallet"].get_balance_by_mixdepth()[4]
assert mbal == cj_amount
jmprint("Funds: {} sats successfully arrived into mixdepth 4.".format(cj_amount))
stop_reactor()
mgr.daemon.taker_finished = dummy_taker_finished
mgr.start()
agent = get_nontor_agent()
addr = mgr.get_route_root()
addr += "/wallet/"
addr += mgr.daemon.wallet_name
addr += "/taker/coinjoin"
addr = addr.encode()
body = BytesProducer(json.dumps({"mixdepth": "1",
"amount_sats": cj_amount,
"counterparties": str(n_cps),
"destination": coinjoin_destination}).encode())
yield mgr.do_request(agent, b"POST", addr, body,
process_coinjoin_response)
def process_coinjoin_response(response):
json_body = json.loads(response.decode("utf-8"))
print("coinjoin response: {}".format(json_body))
@pytest.fixture(scope="module")
def setup_onion_ygrunner():
load_test_config()
jm_single().bc_interface.tick_forward_chain_interval = 10
jm_single().bc_interface.simulate_blocks()

42
test/regtest_joinmarket.cfg

@ -16,6 +16,7 @@ network = testnet
rpc_wallet_file = jm-test-wallet
[MESSAGING:server1]
type = irc
host = localhost
hostid = localhost1
channel = joinmarket-pit
@ -26,6 +27,7 @@ socks5_host = localhost
socks5_port = 9150
[MESSAGING:server2]
type = irc
host = localhost
hostid = localhost2
channel = joinmarket-pit
@ -35,8 +37,46 @@ socks5 = false
socks5_host = localhost
socks5_port = 9150
[MESSAGING:onion]
# onion based message channels must have the exact type 'onion'
# (while the section name above can be MESSAGING:whatever), and there must
# be only ONE such message channel configured (note the directory servers
# can be multiple, below):
type = onion
socks5_host = localhost
socks5_port = 9050
# the tor control configuration:
tor_control_host = localhost
# or, to use a UNIX socket
# control_host = unix:/var/run/tor/control
tor_control_port = 9051
# the host/port actually serving the hidden service
# (note the *virtual port*, that the client uses,
# is hardcoded to 80):
onion_serving_host = 127.0.0.1
onion_serving_port = 8080
# This is mandatory for directory nodes (who must also set their
# own .onion:port as the only directory in directory_nodes, below),
# but NOT TO BE USED by non-directory nodes (which is you, unless
# you know otherwise!), as it will greatly degrade your privacy.
#
# Special handling on regtest, so just ignore and let the code handle it:
hidden_service_dir = ""
# This is a comma separated list (comma can be omitted if only one item).
# Each item has format host:port
# On regtest we are going to increment the port numbers served from, with
# the value used here as the starting value:
directory_nodes = localhost:8081
# this is not present in default real config
# and is specifically used to flag tests:
# means we use indices 1,2,3,4,5:
regtest_count=1,5
[TIMEOUT]
maker_timeout_sec = 15
maker_timeout_sec = 10
[LOGGING]
console_log_level = DEBUG
[POLICY]
# for dust sweeping, try merge_algorithm = gradual

3
test/ygrunner.py

@ -96,7 +96,7 @@ class DeterministicMaliciousYieldGenerator(YieldGeneratorBasic):
"num_ygs, wallet_structures, fb_indices, mean_amt, malicious, deterministic",
[
# 1sp 3yg, honest makers, one maker has FB:
(3, [[1, 3, 0, 0, 0]] * 4, [1, 2], 2, 0, False),
(3, [[1, 3, 0, 0, 0]] * 4, [], 2, 0, False),
# 1sp 3yg, malicious makers reject on auth and on tx 30% of time
#(3, [[1, 3, 0, 0, 0]] * 4, 2, 30, False),
# 1 sp 9 ygs, deterministically malicious 50% of time
@ -173,6 +173,7 @@ def test_start_ygs(setup_ygrunner, num_ygs, wallet_structures, fb_indices,
ygclass = DeterministicMaliciousYieldGenerator
else:
ygclass = MaliciousYieldGenerator
for i in range(num_ygs):
cfg = [txfee_contribution, cjfee_a, cjfee_r, ordertype, minsize,
txfee_contribution_factor, cjfee_factor, size_factor]

Loading…
Cancel
Save