Prior to this commit, the calls to get_new_addr in the functions in
the initial sync algo used for recovery, used the default value of the
argument validate_cache, which is True (because in normal running,
get_new_addr is used to derive addresses as destinations, for which it's
safer to not use the cache, and as one-off calls, are not
performance-sensitive). This caused initial sync to be very slow in
recovery, especially if using large gap limits (which is common).
After this commit, we set the argument validate_cache to False, as is
intended during initial sync. This allows the optimised performance from
caching to be in effect. See earlier PRs 1594 and 1614 for context.
Summary of changes:
* Add typehints to blockchaininterface.py and jsonrpc.py;
* Move all methods called by external code to BlockchainInterface base
class or add abstract methods there;
* More dummy abstract method overrides for DummyBlockchainInterface (for
tests);
* Alphabetical ordering of imports and other minor stuff;
* Behaviour change - previously fee estimation would fail if it could
not get mempoolminfee, now will go with default 10 sat/vB.
Co-authored-by: Pulp <51127079+PulpCattel@users.noreply.github.com>
Fixes#1082:
This commit allows recovery of a wallet from a seedphrase
with a new endpoint wallet/recover. 4 parameters are passed in,
the same three as for wallet/create but also a bip39 seedphrase
as the fourth argument.
This commit also adds a rescanblockchain RPC call and status:
This adds a new endpoint /rescanblockchain which is (Core) wallet specific
(due to underlying JM architecture). This action is spawned in a thread,
since the bitcoind RPC call is blocking and can take a very long time.
To monitor the status of the rescan, an extra field `rescanning` is
added to the /session endpoint.
Also adds test of rpc wallet recovery
Fixes#510.
Prior to this commit, if the call to
jmclient.BitcoinCoreInterface.get_current_blockheight() failed and
returned None, the function jmclient.WalletService.sync_unspent() would
simply ignore the failure and the wallet syncing process would complete,
without ever resetting and updating the wallet's utxo set, resulting in
a situation where users saw out of date utxo sets in their wallets (and
concomitant spending failures).
After this commit, if that blockheight access call fails 5 times, we
instead abandon the attempt to sync the wallet, and shut down the
application.
Before this commit, if Joinmarket-Qt or any other long running script
was started when a new utxo had been created in the wallet, but was not
yet in a block, the transition from unconfirmed to confirmed state would
not be registered, because when confirmation happened and the event was
seen in the transaction monitor, that txid was not in the active_txs
dict. This is now fixed.
Fixes#1212.
Before this commit, transactions were kept monitored in
WalletService.transaction_monitor after first being seen, only if the
list of confirmed callbacks was non empty, but this meant that the logic
is process_new_tx was not executed, meaning the height field of the utxo
was never updated from INF_HEIGHT, so the utxo would continue to be seen
as having zero confirmations, even after blocks are mined.
After this commit, we add any transaction that was seen to the
active_txs dict, so that the height field, and thus the number of
confirmations reported, is correct even if the caller never registered a
confirmed callback.
* Migrate callbacks registered under provisional (tx-output tuples) keys
as soon as their txids are known. Leave a txid breadcrumb in place so
register_callbacks and check_callback_called can find it and the
migrated list of callbacks.
* Invoke callbacks via list comprehensions, retaining only the callbacks
that return False. The old code was buggy, as it was removing elements
from the callback lists while iterating over them, which would cause
callbacks to be skipped.
* The existing code would fail to call any "confirmed" callbacks for a
remove-only transaction if no "unconfirmed" callbacks had been
registered for that transaction, and it would discontinue monitoring
of a transaction after just one "confirmed" callback had returned
True, even if other "confirmed" callbacks returned False to remain
registered. This commit overhauls the logic to fix all of these bugs.
* Delete emptied callback lists from the dicts to be tidy.
The transaction_monitor function in WalletService was polling the most
recent 100 transactions from the BlockchainInterface every 5 seconds.
This was egregious and inefficient. Introduce a _yield_new_transactions
function that remembers the newest previously yielded transaction
between invocations and yields only transactions newer than that one.
Use the new function in transaction_monitor.
Avoid repeatedly deserializing confirmed transactions that remain in the
set of monitored transactions after being confirmed either due to
"confirmed" callbacks that return False or because callbacks have timed
out and have been unregistered without cleaning the set of monitored
transactions. Change the active_txids list to an active_txs dict whose
keys are the txids and whose values are the deserialized transactions.
Use this new dict as a cache to avoid repeatedly deserializing confirmed
transactions.
Alter the BitcoinCoreInterface._yield_transactions generator to be
nicer in cases where only a small number of transactions need to be
examined. Geometrically ramp the requested number of transactions in
each successive request, starting from 1. Since new transactions can
appear in the Bitcoin Core wallet between RPC calls, overlap successive
request ranges and resume yielding transactions after finding the
previously yielded transaction to avoid yielding duplicates.
The listtransactions RPC of Bitcoin Core can return the same (txid,vout)
as both a "category":"receive" and a "category":"send". Therefore, use
(txid,vout,category) as the sync key to ensure that all distinct
elements are yielded.
Delete the unused wallet_name parameter from _yield_transactions, per
suggestion by Adam Gibson.
Fixes#1166.
Fixes#1173.
Prior to this commit, in tests, we were calling
JMWalletDaemon.stopService at exit of the python script,
outside the pytest context, leading to various errors because
the test framework had gone through tear down.
After this commit, no attempt to call stopService is made
at exit (this is not necessary, we already call wallet close
and there is no other persistence necessary).
Fixes#1143 (and possibly others). Before this commit,
(index plus gap limit) addresses are imported on sync,
and addresses used by maker/taker in coinjoin are imported,
but when a deposit occurred, bumping the index, further
addresses were not imported. The effect was that it was
possible, if doing a series of deposits to multiple
external addresses in a Qt session, to end up depositing
to an address that was not yet imported. And this results
in the user needing to rescan for Core+JM to recognize the
coins.
After this commit, we ensure all 'gap limit forwards'
addresses, which are displayed as potential deposit addresses
in Joinmarket-Qt, are imported before the display.
1. Moves the JMWalletDaemon service class into
the jmclient package (see the wallet_rpc.py module).
2. Adds dependencies "klein" and "autobahn" to the
jmclient package, as well as "pyjwt".
3. Adds another module websocketserver.py, using
autobahn, to allow the JMWalletDaemon service to
serve subscriptions over a websocket, for e.g.
transaction notifications.
4. Adds tests both for the websocket connection
and for the JSON-RPC HTTP connection.
JmwalletdWebSocketServerFactory.sendTxNotification
sends the json-ified transaction details using
jmbitcoin.human_readable_transaction (as is currently
used in our CLI), along with the txid.
Also adds a coinjoin state update event sent via
the websocket (switch from taker/maker/none).
Require authentication to connect to websocket.
5. Add OpenApi definition of API in yaml;
also auto-create human-readable API docs in markdown.
6. Add fidelity bond function to API
7. Add config read/write route to API
8. Remove snicker rpc calls temporarily
9. Updates to docoinjoin: corrects taker_finished
for this custom case, does not shut down at end.
10. Address detailed review comments of @PulpCattel.
The /utxos route is the equivalent of the showutxos
wallet tool method.
The heartbeat route /session allows a client to
make sure the backend is still running and in the
expected state (but see later commits for the
coinjoin state update via the websocket).
Fixes to sendpayment, the maker service,
the create wallet function.
sendpayment fix
Also substantially improved and made
functional the coinjoin route, a schedule
is now created and a complete taker-side
coinjoin is now possible.
This commit changes how timelocked addresses are created from the seed and bip32 tree.
A good thing to do would be to have each locktime (e.g. 1st jan 2020,
1st feb 2020, 1st march 2020, etc) actually use a different pubkey from
the HD tree (i.e. ("m/84'/1'/0'/2/0" + 1st jan 2020) ("m/84'/1'/0'/2/1" + 1st feb 2020), etc).
This now means that the sync code doesnt need to know what keys have been associated with
a fidelity bond to scan for the next one. Previously when a user funded a single timelocked
address, the wallet will generate _another_ pubkey and import _another_ ~960 addresses, so
funding one address would actually mean watching and generating ~1920 addresses not ~960.
This should help with the problem found by some people that fidelity bond wallets are slower
to sync. Other optimizations are possible but the structure of fidelity bond wallets will
probably be fixed for decades, so this change is worth doing now.
Prior to this commit, several test functions were using
"True" to flag internal and "False" to flag external for
the HD branch for the wallet, but we now use BaseWallet vars
ADDRESS_TYPE_[IN/EX]TERNAL (1/0), so this
is changed to explicitly reference those. There is no change
to the live code (which calls get_[internal/external]_addr).
In addition _index_cache updates in wallet are protected
with a wrapper function to ensure that the branch requested
is valid.
Tests pass both before and after this change.
See #772. This is likely a partial fix but that issue
may be more complex.
More generally, while we may have to process multiple
entries in the return of `listtransactions`, with the
same txid, because they may have different wallet labels,
we do not want to call `gettransaction` repeatedly on
the same txid in the same monitor loop call. Note however,
that we *do* need to call `gettransaction` again in
the next monitor loop, since the state (confirmations)
updates, so we cannot permanently cache those results.
Additionally removed redundant old_txs entries with set().
Prior to this commit, the list of used addresses,
which is required to check for address reuse, is populated
on startup in fast sync, and updated as new transactions
arrive; but if --recoversync is chosen, this list was not
originally getting populated. This commit corrects that bug.
Prior to this commit, in case an RPC failure occurred when
accesing the block height, the program would continue but the
wallet would be in an un-writeable state (for command line
programs, specifically yield generators; for Qt the shutdown
would occur).
This commit slightly cleans up the process of shutting down,
ensuring that duplicate shutdown calls do not result in
stack traces. It also ensures that also for command line
programs, the application will immediately shutdown if the
regular heartbeat call to query the block height fails, as this
risks inconsistencies in the wallet (though the previous
situation luckily did not result in this as the call to
BaseWallet.close() resulted in the wallet being read only).
A future PR should develop a more sophisticated approach to
RPC call failures that may allow the program to wait.
stopservice
Prior to this commit, if non-self broadcast was enabled
but the counterparty chosen did not broadcast, the transaction
would remain unbroadcast.
After this commit, the Taker checks, after the configured
value of TIMEOUT.unconfirm_timeout_sec (default 90s), the
Taker will broadcast the transaction.
Also amended default config comment for this function.
Update no-history-sync code:
This updates the new functionality in jmclient.wallet_utils
in the no-history-sync PR #444 to be compatible
with the python-bitcointx refactoring.
Remove all future/py2 compatibility code remaining:
This is in line with #525 and corrects erroneous
addition of more compatibility code.
Addresses all flake8 complaints (ununsed imports etc)
Addresses review of @dgpv
Addresses review of @kristapsk
Replaces core transaction, address, serialization
and sign functionality for Bitcoin with
python-bitcointx backend.
Removes bech32 and btscript
modules from jmbitcoin. Removes all string,
hex, binary conversion routines. A generic
hex/binary conversion now is added to jmbase.
Removes all transaction serialization and
deserialization routines. Removes the now
irrelevant test modules.
Remaining functions in jmbitcoin remove any parsing of
hex format, requiring callers to use binary only.
One additional test added, testing the remaining
function in secp256k1_transaction.py: the signing
of transactions. Deserialized form is now
bitcointx.CMutableTransaction.
For jmbase, in addition to the above, generic conversions
for utxos to and from strings is added, and a dynamic conversion
for AMP messages to binary-only. Within the code, utxos are
now only in (binarytxid, int) form, except where converted
for communcation.
Tthe largest part of the changes are
the modifications to jmbitcoin calls in jmclient;
as well as different encapsulation with CMutableTransaction,
there is also a removal of some but not all hex parsing;
it remains for rpc calls to Core and for AMP message
parsing. Backwards compatibility must be ensured so some
joinmarket protocol messages still use hex, and it is
also preserved in persistence of PoDLE data.
As part of this, some significant simplification of
certain legacy functions within the wallet has been done.
jmdaemon is entirely unaltered (save for one test which
simulates jmclient code).
Reasoning for this change: to ensure that Qt will show
a message and gracefully exit (or quit attempting to
load a wallet) in all 3 cases: on startup it show an
intelligible message if the RPC connection fails (as
before this PR), if the RPC fails while no wallet is
loaded and thus no wallet service is started, it should
show an intelligible error message when you attempt to
load a wallet and it fails, and finally it should show
an intelligible error message before quitting, if the rpc
connection fails during the period when the wallet is
already loaded.
By switching to an Exception instead of sys.exit, it does
mean that starting a yieldgenerator shows a stack trace,
but it also shows an intelligible error message (in red),
and this is command line, so UI requirements are less strong.
We preserve the "good" behaviour of no stack trace, but
only error message, if the rpc connection is lost during
running.
Fixes#442.
First, the CONNREFUSED socket error is handled in jsonrpc.
Second, we respond to this (but *not* to resets) with a reactor
shutdown in BitcoinCoreInterface.rpc(). This also necessitates
early-quitting in the calling function
(WalletService.transaction_monitor) since the reactor stop
will only stop future deferred calls, not the currently running
one. The obvious sys.exit approach is only used in startup,
because the reactor is not currently running at that point.
Also minor change to DummyBlockchainInterface for test.
Watchonly wallets use pubkeys instead of privkeys, but in a bit of
hack the functions previously called "_get_priv_from_path" would
actually return public keys for watchonly wallets. This could have
pretty terrible consequences one day, so functions like that have
been renamed to use the word "key" instead, which could be either
private or public.
Fidelity bond wallets are intended to be used when at a later date
using fidelity bonds to greatly increase joinmarket's resistance to
sybil attacks. This commit adds support for timelocked addresses.
It allows users to optionally create wallet which support such
addresses. The synchronization code is modified to also scan for
timelocked addresses. The keypairs of the timelocked addresses go in
the newly created 2nd address type, where before the zeroth index were
receive addresses and first index was change.
The locktime dates are fixed at the first of each month for the next 30
years. This means users dont need to remember any dates, and so just
their seed phrase and wallet type will still be enough to recover all
funds. Each keypair used for timelocking requires an additional 360
addresses to be scanned for, which isn't a problem for Bitcoin Core.
Fidelity bonds are only stored in the zeroth mixdepth, as they are not
used in repeated coinjoins so theres no point having them in multiple
mixdepths.
Timelocked addresses don't use the get_new_script() family of functions
because they all assume that one index equals one address, and that
therefore it's possible to ask for a "next" address. For timelocked
addresses knowing the index is not enough to know the address, the
timestamp must be known too. Also once one address made of
(index, timestamp) is used you mustn't use that index and pubkey again,
even though all the other timelocks for that index/pubkey are unused.
This is for privacy reasons, as its equivalent to address reuse.
zebra-lucky
Kristaps Kaupe
Adam Gibson
roshii
Matt Whitlock
abhishek0405
undeath
chris-belcher
jules23