Fixes#1214.
Prior to this commit, if a broadcast of a transaction failed locally, or
failed after attempting remote broadcast then falling back to local via
the function jmclient.taker.Taker.handle_unbroadcast_transaction, then
the version of the on_finished_callback found in
jmclient.wallet_rpc.JMWalletDaemon incorrectly asserted that the fromtx
keyword argument must be True, thus failing to update the state of the
daemon to CJ_NOT_RUNNING.
After this commit that assertion is removed, and the coinjoin_in_process
value returned over the API is correctly False after either a
successful, or an unsuccessful, finishing of the coinjoin.
Fixes#1213.
Prior to this commit, if the YieldGeneratorService were started via a
call to /maker/start over the RPC-API, the early check of whether coins
were present in the wallet to allow starting the maker, incorrectly read
the contents of the dict returned by
jmclient.wallet.UTXOManager.get_balance_by_mixdepth, which returns
entries for mixdepths with coins which are all frozen, with a total
value of zero, even if the include_disabled keyword argument is set to
False. Thus only checking for whether the length of the returned dict
was zero was incorrect.
After this commit, we will stop early, as intended, for a wallet which
has no mixdepths with any non-frozen coins.
Fixes#1212.
Before this commit, transactions were kept monitored in
WalletService.transaction_monitor after first being seen, only if the
list of confirmed callbacks was non empty, but this meant that the logic
is process_new_tx was not executed, meaning the height field of the utxo
was never updated from INF_HEIGHT, so the utxo would continue to be seen
as having zero confirmations, even after blocks are mined.
After this commit, we add any transaction that was seen to the
active_txs dict, so that the height field, and thus the number of
confirmations reported, is correct even if the caller never registered a
confirmed callback.
`:80` needs to be provided otherwise parsing will fail with
Failed to load directory nodes: InvalidLocationStringError('3kxw6lf5vf6y26emzwgibzhrzhmhqiw6ekrek3nqfjjmhwznb2moonad.onion')
This is correct in the auto-generated configuration file, but not in the
document.
Also mention that even for UNIX sockets, it's necessary to specify `tor_control_port`.
Prior to this commit, running ob-watcher with
an onion-type message channel configured would result
in the bot attempting to connect to all the makers,
which is bad. This happened because the internal logic
of the onion message channel is that receivers of privmsgs
will be sent peer info from the directory, so that they can
immediately respond p2p if they succeed in outward connecting.
But for bots who do not intend to engage in a coinjoin interactive
protocol, like ob-watchers, this is absolutely not the desired outcome.
After this commit, a bot can specify mode "PASSIVE" in the call to
get_mchannels(), which results in the OnionMessageChannel object only
creating non-directory remote peer objects of type OnionPeerPassive,
instead of OnionPeer, which means they never try to connect to those
remote peers.
Also, exports JMMakerClientProtocol for custom directory node scripts
(stored in the custom-scripts repo).
Modify default config with 2 signet and mainnet directory nodes to
start.
Handles unreachable directory nodes with a human readable error and
adjusts connection timeouts to be realistic.
Changes wording in Qt notifications from "IRC" to message channel.
Updates docs, new directory node information.
In the previous commit, all peers served an onion.
After this commit, taker client instances will automatically
send a config var to the jmdaemon backend that instructs
the OnionMessageChannel instance to not start an onion service,
and the handshake messages sent by these peers replace the
onion location with a placeholder string NOT-SERVING-ONION.
Directories and maker peers will not therefore to connect outbound
to them, but privmsging still happens p2p with connections from
takers to makers after the directory has communicated their
reachable .onion addresses.
This change reduces the configuration requirement for takers and
is better for their privacy and security (without sacrificing
the gain we get from having p2p connections).
The above comments re: takers also apply to ob-watcher bots.
This commit also fixes a large number of minor bugs and errors in
documentation, as well as many Python cleanups after review from
@PulpCattel. A few concrete items are:
It fixes the ob-watcher functionality to work with the new subclass
of MessageChannel (OnionMessageChannel).
It corrects the on_nick_leave trigger to make dynamic nick switching
between MessageChannels (as implemented in MessageChannelCollection)
work correctly.
It corrects the order of events in the add_peer workflow to ensure that
a handshake can always be sent so that the activation of the connection
always works.
It sets a default messaging config with onion, 2 active IRC servers and
one inactive IRC server. The onion config has 2 signet directory nodes,
so this will change to mainnet after the PR is merged to master.
Joinmarket bots run their own onion services allowing inbound connections.
Both takers and makers connect to other makers at the mentioned
onion services, over Tor.
Directory nodes run persistent onion services allowing peers to
find other (maker) peers to connect to, and also forwarding
messages where necessary.
This is implemented as an alternative to IRC, i.e. a new
implementation of the abstract class MessageChannel, in onionmc.py.
Note that using both this *and* IRC servers is supported; Joinmarket
supports multiple, redundant different communication methods,
simultaneously.
Messaging is done with a derived class of twisted's LineReceiver,
and there is an additional layer of syntax, similar to but not the
same as the IRC syntax for ensuring that messages are passed with
the same J5.. nick as is used on IRC. This allows us to keep the
message signing logic the same as before. As well as Joinmarket line
messages, we use additional control messages to communicate peer lists,
and to manage connections.
Peers which send messages not conforming to the syntax are dropped.
See https://github.com/JoinMarket-Org/JoinMarket-Docs/pull/12 for
documentation of the syntax.
Connections to directory nodes are robust as for IRC servers, in
that we use a ReconnectingClientFactory to keep trying to re-establish
broken connections with exponential backoff. Connections to maker
peers do not require this feature, as they will often disconnect
in normal operation.
Multiple directory nodes can and should be configured by bots.
* Migrate callbacks registered under provisional (tx-output tuples) keys
as soon as their txids are known. Leave a txid breadcrumb in place so
register_callbacks and check_callback_called can find it and the
migrated list of callbacks.
* Invoke callbacks via list comprehensions, retaining only the callbacks
that return False. The old code was buggy, as it was removing elements
from the callback lists while iterating over them, which would cause
callbacks to be skipped.
* The existing code would fail to call any "confirmed" callbacks for a
remove-only transaction if no "unconfirmed" callbacks had been
registered for that transaction, and it would discontinue monitoring
of a transaction after just one "confirmed" callback had returned
True, even if other "confirmed" callbacks returned False to remain
registered. This commit overhauls the logic to fix all of these bugs.
* Delete emptied callback lists from the dicts to be tidy.
The transaction_monitor function in WalletService was polling the most
recent 100 transactions from the BlockchainInterface every 5 seconds.
This was egregious and inefficient. Introduce a _yield_new_transactions
function that remembers the newest previously yielded transaction
between invocations and yields only transactions newer than that one.
Use the new function in transaction_monitor.
Avoid repeatedly deserializing confirmed transactions that remain in the
set of monitored transactions after being confirmed either due to
"confirmed" callbacks that return False or because callbacks have timed
out and have been unregistered without cleaning the set of monitored
transactions. Change the active_txids list to an active_txs dict whose
keys are the txids and whose values are the deserialized transactions.
Use this new dict as a cache to avoid repeatedly deserializing confirmed
transactions.
Alter the BitcoinCoreInterface._yield_transactions generator to be
nicer in cases where only a small number of transactions need to be
examined. Geometrically ramp the requested number of transactions in
each successive request, starting from 1. Since new transactions can
appear in the Bitcoin Core wallet between RPC calls, overlap successive
request ranges and resume yielding transactions after finding the
previously yielded transaction to avoid yielding duplicates.
The listtransactions RPC of Bitcoin Core can return the same (txid,vout)
as both a "category":"receive" and a "category":"send". Therefore, use
(txid,vout,category) as the sync key to ensure that all distinct
elements are yielded.
Delete the unused wallet_name parameter from _yield_transactions, per
suggestion by Adam Gibson.
Prior to this commit, the file commitmentlist, which stores
commitments (podle) for makers to let them blacklist/prevent
reuse was stored in the local directory for the script, which
allowed remote running of jmdaemon but was very unhelpful for
any situation where multiple bots are running at once, e.g.
in testing or using multiple wallets against the same codebase.
This could result in incorrect rejection of coinjoins.
After this commit, by default, the commitmentlist file is stored
in datadir/cmtdata/commitmentlist, so it will be local to any
custom data directory as would be the case for running multiple
wallets on the same machine. A user can set the POLICY variable
commitment_list_location to "." to revert to the previous behaviour.
If a user has many UTXOs but wants to spend just one, they would have
to freeze all the UTXOs one-by-one except for one, then after sending
do the same again unfreezing all UTXOs one-by-one. This new command
drastically speeds up this process.
This commit fixes an issue regarding a wrong taker state caused by
initiating a taker operation when no `max_cj_fee` value is present.
Before this change the taker state switches to `running` indefinitely.
After this change the config check is made before updating the state.
Use sats/kilo-vbyte
Use words for large numbers
Improve unit consistency
Improve newline consistency
Improve comment consistency
Improve capitalization consistency
Improve section consistency
Fixes#1170.
Prior to this commit, an attempt to spend using
the direct-send endpoint in the RPC API would
cause a backend crash because the NoFundsException
was not being caught.
This is now fixed.
Adds code to validate bech32m encoded taproot addresses
and scriptPubKeys. The underlying functions are in
python-bitcointx upcoming version 1.1.3.
Adds tests at both the basic address validation layer,
and also at the layer of sending transactions to
taproot outputs.
Fixes#1166.
Fixes#1173.
Prior to this commit, in tests, we were calling
JMWalletDaemon.stopService at exit of the python script,
outside the pytest context, leading to various errors because
the test framework had gone through tear down.
After this commit, no attempt to call stopService is made
at exit (this is not necessary, we already call wallet close
and there is no other persistence necessary).
Fixes#1168.
Before this commit, if an RPC client sent /maker/start
on an active wallet without confirmed coins, it would
raise NotEnoughCoins but this would happen *after* the
call to the setup function which resets the coinjoin
state to CJ_MAKER_RUNNING, even though the maker service
did not successfully start.
After this commit, the raise of NotEnoughCoins happens
first, preventing the update of coinjoin state.
Prior to this commit, a lock of one wallet followed by
an unlock of another, or an unlock of a new wallet, overriding
the old one, if it occurred during a running taker-side
coinjoin, would not reset the coinjoin_state to CJ_NOT_RUNNING,
resulting in an inability of the newly loaded wallet to function
correctly.
After this commit, all subservices, including the quasi-service
of taker-side coinjoin, are fully shut down whenever a wallet is
locked, or a new wallet is unlocked. This may be suboptimal (see
TODO) but is logical for now.
The main effect is to ensure that a new wallet will always start
in the correct coinjoin state (CJ_NOT_RUNNING).
Also worth noting, a running Taker will have its abort parameter
set to True on the wallet lock event, meaning that it will not
proceed to the next step on the next asynchronous message coming
from counterparties.
Fix shutdown of already open wallet on new unlock
Prior to this commit, if a JSON-RPC API client called
/unlock on a new wallet while an existing one was currently
loaded, the preexisting wallet would not have its wallet
service shut down, and thus the .lock file would not be removed,
resulting in an inability to open that wallet a second time.
After this commit, we correctly refer to the wallet service
internally such that its service does get shut down, and the
close event of the wallet occurs, removing the .lock file as
required.
Fix re-unlock of currently unlocked wallet
This behaviour was originally intended by #1136. If a wallet
is currently in an unlocked state in the daemon, but the user
loses the auth token, and tries to re-unlock, then before this
commit, the unlock operation would fail because there was an
unsuccessful attempt to open the wallet file.
After this commit, if the unlock request is applied to the wallet
with the same name as the one currently active, then the password
is authenticated by opening the wallet in read-only mode, preventing
the above issue from occurring.
Fix bug that lockwallet doesn't delete wallet_name
Before this commit, the member variable wallet_name in
the JMWalletDaemon object was not being reset to None
in a call to the lockwallet endpoint which meant that the
unlock event erroneously treated the wallet as already being
active. This corrects that error by setting walet_name to
None in lockwallet.
Fixes#1122. An RPC client can request the list
of coinjoins as shown in the file datadir/logs/yigen-statement.csv.
Authentication is not required, nor a specific wallet, as this is
a global list of all coinjoins done so far. A 404 is returned if
the file has not yet been created (no maker operations).
Fixes#1093. This adds a POST method freeze for a wallet,
in which the utxo must be specified as a standard hex txid:n
string in the body, along with a boolean value of 'freeze', to
toggle the frozen/unfrozen state of the given utxo in the wallet.
Fixes#1123. Prior to this commit, the unlockwallet
method erroneously waited for wallet sync before continuing
and returning the credential. This was not necessary as the
WalletService method startService calls wallet syncing
asynchronously(!) in request_wallet_sync. Hence this initial
syncing step is removed, allowing the RPC call to return
immediately.
Fixes#1143 (and possibly others). Before this commit,
(index plus gap limit) addresses are imported on sync,
and addresses used by maker/taker in coinjoin are imported,
but when a deposit occurred, bumping the index, further
addresses were not imported. The effect was that it was
possible, if doing a series of deposits to multiple
external addresses in a Qt session, to end up depositing
to an address that was not yet imported. And this results
in the user needing to rescan for Core+JM to recognize the
coins.
After this commit, we ensure all 'gap limit forwards'
addresses, which are displayed as potential deposit addresses
in Joinmarket-Qt, are imported before the display.
Fixes#1120. As per the discussion in that issue, while
we can only return 'processing started' by default, and
the client must monitor progress/state to see if the maker
service has connected successfully, nevertheless if the
maker startup is immediately invalidated by the fact that
the already-synced wallet does not contain confirmed coins,
we can and should return an error message instead of a 202.
This commit does that, adding to the spec a 409-No-Coins
response type which indicates to the API client that the
setup of the maker failed for this specific reason.
Fixes#1118.
Before this commit, the json serializtion of a
WalletEntry object was incorrect and missing some
fields. This is now fixed, and the WalletDisplayResponse
in the RPC spec .yaml file correctly reflects the
fields that are returned by the JMWalletDaemon in response
to the /display request.
Prior to this commit, if a user has attempted
a coinjoin with /taker/coinjoin but a problem
occurs (such as an unresponsive maker), they
are not able to abort the protocol but must wait
until the taker_finished event is triggered.
After this commit a GET request to the endpoint
defined as /wallet/walletname/taker/stop will
force-stop the coinjoin attempt (similar to
quitting the script in CLI). This will reset the
wallet daemon state to CJ_NOT_RUNNING.
Fixes#1121.
Prior to this commit, if a user
lost access to the authentication token for
a session with the wallet RPC, they would not
be able to lock the existing wallet before
authenticating again (getting a new token for
the same wallet, or a different one).
After this commit, a call to the /unlock endpoint
will succeed even if an existing wallet is currently
unlocked (whether a different one or the same one).
The existing wallet service, if present, is shut down,
if the attempt to authenticate a new wallet, with a
password is successful (otherwise nothing happens).
A test is added to make sure that this re-unlock can work.
This commit changes the underlying functions used in
jmbitcoin from the private and public key primitives
in coincurve and replaces them with equivalent
primitives CKey and CPubKey from python-bitcointx,
this removes the need to install coincurve and its
own bundled libsecp256k1 dynamic library.
Note that an additional pubkey_tweak_mul function
is exposed with ctypes from python-bitcointx's
bundled libsecp256k1 library.
Fixes#1043.
Prior to this commit, only keys/scripts/addresses
inside the scope of the current wallet script_map
(the keys cached by sync, according to persisted
index in wallet file, including gap limit) would
allow a successful signing operation, otherwise
an assertion was raised.
After this commit, signing can be done with any
arbitrary height index in the wallet (assuming a
valid path for this wallet).
Some of the return codes for the endpoints in
the RPC API were incorrect, these are fixed.
They are also checked in the tests.
Additionally, an extra test of the maker start/
stop function is added (though it tests only
state updates, not actual service start).
Fixes#1091.
Before this commit, if the unconfirmed transaction
event triggered the taker_finished callback in the
RPC-started coinjoin, the value of fromtx is
'unconfirmed' and not False, hence an assertion error
was raised unless the confirmed event came first
(which is only likely to happen on regtest). This
crashed the jmwalletd daemon backend.
After this commit, the assertion check is only that
the value is not True, which it must not be as long
as we only accept single-join schedules.
Adam Gibson
laanwj
Kristaps Kaupe
Adam Gibson
Matt Whitlock
unofficial bisq contributor
chris-belcher
Thebora Kompanioni
5F43CCDF
Dennis Reimann
Manas Gandy