See #1435.
Prior to this commit, if connections were immediately closed,
no backoff or jitter were applied to delay the next connection attempt,
resulting in too many connection attempts to directory nodes that the
client could not reach, and spam in the logs.
After this commit, we enforce that we always shut down the ClientService
on any disconnection trigger or connection failure, and only attempt to
reconnect with a manual backoff delay (4 seconds increasing by 50% for
20 attempts, plus a few seconds jitter), for directory nodes, while for
non-directory nodes we always give up on any failure. Max delay is a bit
above 3 hours.
Prior to this commit, if a message (like a disconnection control
message), was to be sent to live peers, regarding another peer that did
not have a nick field set, a crash would occur in directory nodes
sending this message, because the function
OnionPeer.get_nick_peerlocation_ser() assumed the existence of the nick
field, and raised the OnionPeerError exception if it did not exist.
After this commit, the raising of this Exception is caught when calling
OnionPeer.get_nick_peerlocation_ser, so that a peer in this state is
just not added to the list of peers to be sent to live peers. This
condition is caused specifically by the fact that said peer has not
completed a handshake (which must always pass the value of the nick),
and so the other live peers will not yet be aware of it.
Prior to this commit, we timed out the attempts to connect to individual
directory nodes after 10s, which proves to be too optimistic for
connections to onions, as failures are often seen. After this commit, we
allow 60s (note that this time includes handshake completion).
Fixes#1306.
Prior to this commit, it was possible (unlikely) for a peer to exist in
the active_directories nick in OnionMessageChannel, but to have False
entries (i.e. disconnected) for every directory, meaning that the random
choice from those directories raised an Exception which was not caught,
causing a crash.
This is now fixed by checking whether the list of directories to be
chosen from randomly, is empty, and if so raising the correct Exception
type, namely OnionDirectoryPeerNotFound.
Fixes#1259.
This is done to support envisaged experimentation with significantly
larger counterparty counts than the current usually 10-12 maximum. IRC
notionally supports very large messages because we split messages into
chunks, but realistically, throttling usually causes
timeouts/interruptions if we try to use extremely large amounts of text.
In Joinmarket's communication protocol, the only time we need very large
messages is specifically to allow transfer of fully signed transactions
that need to be pushed to the bitcoin network. (See JMPushTx). So this
bump to the maximum allowable line length is specifically to support
that message, in case people want to try coinjoins with 20-40
participants. The comment indicates an extremely rough guesstimate of
maximum supported counterparty count based on this limit.
Note that users can always just broadcast the final tx themselves, of
course.
Prior to this commit, passive observation of orderbook updates by bots
such as ob-watcher was not able to keep track of nicks/bots that had
disconnected and were no longer available, because non-directory peers
do not get the network level disconnection event.
After this commit, the directory nodes forward a variant of the
"peerlist" control message, that was previously used only to send
connection information from one non-directory peer to another (so that
they could establish a new p2p connection if desired). Now a new version
of that message adds an extra "D" field to indicate that this message is
being used to inform that the relevant peer has disconnected from this
directory node.
Bubbling up the on_nick_leave event: it is already the case that the
final on_nick_leave callback is only triggered by the disappearance of a
nick from all available message channels (so that as long as a nick is
perceived as being in at least one available message channel, its
current offers are still maintained in the local database); this logic
is now repeated one layer down, because the disconnection of a nick from
one directory node may not mean it is shut down, so we only pass the
on_nick_leave callback from the OnionMessageChannel object up to the
MessageChannelCollection callback when the active_directories dict
indicates that this particular nick is no longer available on any of our
configured directory nodes.
Original testing setup of onion message channels just sent messages on
the first directory node peer in our list. This fixes that TODO.
After this commit, we keep a map of which directory nodes we have seen a
given Joinmarket nick on, and we keep track of whether those directory
nodes are available (connected) or not. Then when we want to privmsg
that nick, if we do not currently have a direct connection, we choose
randomly from the set of directory nodes on which that nick has been
seen, and which are currently live/connected.
Prior to this commit, a taker bot that received the orderbook on the
onion message channel would attempt to connect to every maker bot that
had sent an offer, which is not just inefficient but appears to trigger
rate limiting in Tor itself, throwing errors in the SOCKS proxy, at
numbers of around 80 makers, in testing (but could easily happen at
lower levels).
After this commit, we only opportunistically attempt to connect to
makers (using the same OnionPeer.try_to_connect method as before) when
we have already decided to send them a message (e.g. chosen to fill
their offer). This is done by calling try_to_connect only in
OnionMessageChannel._privmsg, and only to the right kind of peer.
The downside of this is that we will send more messages via directory
(at least one additional, for each maker), but clearly the alternative
is not viable.
As part of this update, we ensure that connection attempts are not
accidentally duplicated (see OnionPeer.connecting).
Prior to this commit, running ob-watcher with
an onion-type message channel configured would result
in the bot attempting to connect to all the makers,
which is bad. This happened because the internal logic
of the onion message channel is that receivers of privmsgs
will be sent peer info from the directory, so that they can
immediately respond p2p if they succeed in outward connecting.
But for bots who do not intend to engage in a coinjoin interactive
protocol, like ob-watchers, this is absolutely not the desired outcome.
After this commit, a bot can specify mode "PASSIVE" in the call to
get_mchannels(), which results in the OnionMessageChannel object only
creating non-directory remote peer objects of type OnionPeerPassive,
instead of OnionPeer, which means they never try to connect to those
remote peers.
Also, exports JMMakerClientProtocol for custom directory node scripts
(stored in the custom-scripts repo).
Modify default config with 2 signet and mainnet directory nodes to
start.
Handles unreachable directory nodes with a human readable error and
adjusts connection timeouts to be realistic.
Changes wording in Qt notifications from "IRC" to message channel.
Updates docs, new directory node information.
In the previous commit, all peers served an onion.
After this commit, taker client instances will automatically
send a config var to the jmdaemon backend that instructs
the OnionMessageChannel instance to not start an onion service,
and the handshake messages sent by these peers replace the
onion location with a placeholder string NOT-SERVING-ONION.
Directories and maker peers will not therefore to connect outbound
to them, but privmsging still happens p2p with connections from
takers to makers after the directory has communicated their
reachable .onion addresses.
This change reduces the configuration requirement for takers and
is better for their privacy and security (without sacrificing
the gain we get from having p2p connections).
The above comments re: takers also apply to ob-watcher bots.
This commit also fixes a large number of minor bugs and errors in
documentation, as well as many Python cleanups after review from
@PulpCattel. A few concrete items are:
It fixes the ob-watcher functionality to work with the new subclass
of MessageChannel (OnionMessageChannel).
It corrects the on_nick_leave trigger to make dynamic nick switching
between MessageChannels (as implemented in MessageChannelCollection)
work correctly.
It corrects the order of events in the add_peer workflow to ensure that
a handshake can always be sent so that the activation of the connection
always works.
It sets a default messaging config with onion, 2 active IRC servers and
one inactive IRC server. The onion config has 2 signet directory nodes,
so this will change to mainnet after the PR is merged to master.
Joinmarket bots run their own onion services allowing inbound connections.
Both takers and makers connect to other makers at the mentioned
onion services, over Tor.
Directory nodes run persistent onion services allowing peers to
find other (maker) peers to connect to, and also forwarding
messages where necessary.
This is implemented as an alternative to IRC, i.e. a new
implementation of the abstract class MessageChannel, in onionmc.py.
Note that using both this *and* IRC servers is supported; Joinmarket
supports multiple, redundant different communication methods,
simultaneously.
Messaging is done with a derived class of twisted's LineReceiver,
and there is an additional layer of syntax, similar to but not the
same as the IRC syntax for ensuring that messages are passed with
the same J5.. nick as is used on IRC. This allows us to keep the
message signing logic the same as before. As well as Joinmarket line
messages, we use additional control messages to communicate peer lists,
and to manage connections.
Peers which send messages not conforming to the syntax are dropped.
See https://github.com/JoinMarket-Org/JoinMarket-Docs/pull/12 for
documentation of the syntax.
Connections to directory nodes are robust as for IRC servers, in
that we use a ReconnectingClientFactory to keep trying to re-establish
broken connections with exponential backoff. Connections to maker
peers do not require this feature, as they will often disconnect
in normal operation.
Multiple directory nodes can and should be configured by bots.
zebra-lucky
roshii
Adam Gibson
Kristaps Kaupe
bisqubutor
Adam Gibson