Merge #475: Move all user data to home directory

8c8e6e2 Move all user data to home directory (Adam Gibson)

docs/JOINMARKET-QT-GUIDE.md

@@ -33,7 +33,7 @@ You will get the following error screen initially:
 ![](images/JMQrpcfailed.png)
 
 ... because your rpc connection to Bitcoin Core is not set up. A `joinmarket.cfg` file
-has been created in the directory you're running in. If you're running Bitcoin Core, open it and edit:
+has been created in your Joinmarket data directory (see [here](USAGE.md#data)). If you're running Bitcoin Core, open it and edit:
 
     [BLOCKCHAIN]
     rpc_user = yourusername-as-in-bitcoin.conf
@@ -41,9 +41,7 @@ has been created in the directory you're running in. If you're running Bitcoin C
     rpc_host = localhost #default usually correct 
     rpc_port = 8332 # default for mainnet
 
-If you're not using Core, you can set the value of `blockchain_source` in that section to `electrum-server` **but only do this for testing; it's not supported!**.
-
-Once the rpc connection is correct (or using Electrum), you will be presented with this start screen:
+Once the rpc connection is correct, you will be presented with this start screen:
 
 ![](images/JMQInitregtest.png)
 
@@ -67,8 +65,7 @@ Next, give a name for your wallet file:
 
 ![](images/JMQnewwalletfile.png)
 
-The wallet file will be saved with this name under the directory `wallets`, which is 
-created in the same location as where you have downloaded the application/binary.
+The wallet file will be saved with this name under the directory `wallets` in the same data directory as mentioned above.
 
 Joinmarket wallet file names are .json by default; this isn't strictly necessary, but better to 
 stick to that convention.

docs/USAGE.md

@@ -5,9 +5,13 @@ followed a manual installation as per [here](INSTALL.md)).
 
 # Contents
 
-1. [Configuring for Bitcoin Core](#configure)
+1. [Managing your Joinmarket data](#data)
 
-2. [Using the wallet-tool.py script](#wallet-tool)
+   a. [Portability](#portability)
+
+2. [Configuring for Bitcoin Core](#configure)
+
+3. [Using the wallet-tool.py script](#wallet-tool)
 
    a. [Creating a Wallet](#generate)
 
@@ -29,11 +33,51 @@ followed a manual installation as per [here](INSTALL.md)).
 
    j. [What is the Gap Limit](#gaplimit)
 
-3. [Try out a coinjoin; using sendpayment.py](#try-coinjoin)
+4. [Try out a coinjoin; using sendpayment.py](#try-coinjoin)
+
+5. [Running a "Maker" or "yield generator"](#run-maker)
+
+6. [Running the tumbler script to boost privacy of owned coins](#run-tumbler)
+
+<a name="data" />
+
+## Managing your Joinmarket data
 
-4. [Running a "Maker" or "yield generator"](#run-maker)
+First thing to do: go into `scripts/`, and run:
+
+        (jmvenv)$ python wallet-tool.py generate
 
-5. [Running the tumbler script to boost privacy of owned coins](#run-tumbler)
+This *should* quit with an error, because the connection to Bitcoin Core is not configured; we'll cover that in the next section.
+However, this first run will have automatically created a data directory.
+Locate the newly created file `joinmarket.cfg` which will be in your user home directory under `.joinmarket/`.
+So on Linux you should find it under `/home/username/.joinmarket/joinmarket.cfg`, and similarly for MacOS and Windows.
+You should see the following files and folders for an initial setup:
+
+```
+.joinmarket/
+    joinmarket.cfg
+    logs/
+    wallets/
+    cmtdata/
+```
+
+`joinmarket.cfg` is the main configuration file for Joinmarket and has a lot of settings, several of which you'll want to edit or at least examine.
+This will be discussed in several of the sections below.
+The `wallets/` directory is where wallet files, extension (by default) of `.jmdat` are stored after you create them. They are encrypted and store important information; without them, it is possible to recover your coins with the seedphrase, but can be a hassle, so keep the file safe.
+The `logs/` directory contains a log file for each bot you run (Maker or Taker), with debug information. You'll rarely need to read these files unless you encounter a problem; deleting them regularly is recommended (and never dangerous). However there are other log files kept here, in particular one called `yigen-statement.csv` which records all transactions your Maker bot does over time. This can be useful for keeping track. Additionall, tumbles have a `TUMBLE.schedule` and `TUMBLE.log` file here which can be very useful; don't delete these.
+The `cmtdata/` directory stores technical information that you will not need to read.
+
+<a name="portability" />
+
+### Portability
+
+It is possible to use a different data directory than the default mentioned above (`~/.joinmarket/` or equivalent). To do this, run any Joinmarket script (see below for descriptions of the main ones) with the flag `--datadir=/my/data/directory`.
+Then the above directory structure will be created there. If you move a wallet file from one directory to another it will work fine, as long as you are using the same instance of Bitcoin Core.
+
+The slightly more difficult case is moving to a new machine and/or a new Bitcoin Core instance. There, apart from the obvious of needing to change your BLOCKCHAIN configuration settings (see next section), you will encounter an additional hurdle when you move your `.jmdat` file to the new setup.
+Since the new Bitcoin Core instance doesn't have the addresses imported, you will need to do a rescan of the blockchain for some appropriate range of blocks.
+
+The worst case is if you recover only from seedphrase and on a new Core instance; then you not only don't have the addresses imported, but Joinmarket will have to search through the addresses to find all usages, which can be tricky and might require multiple rescans in theory. Use the `--recoversync` option of `wallet-tool.py` and use a large gap limit (`-g`) if the wallet was heavily used.
 
 <a name="configure" />
 
@@ -41,14 +85,9 @@ followed a manual installation as per [here](INSTALL.md)).
 
 Bitcoin Core is required to use Joinmarket; note that the node *can* be pruned.
 
-Configuring Joinmarket for Core is now reduced, since there is no longer any `walletnotify` used.
+Configuring Joinmarket for Core no longer needs any `walletnotify` setting.
 
-First thing to do: go into `scripts/`, and run:
-
-        (jmvenv)$ python wallet-tool.py generate
-
-This *should* quit with an error, because the rpc is not configured. Open the newly created file `joinmarket.cfg`,
-and edit:
+In the `joinmarket.cfg` file described above, edit this section (comments omitted; do read them):
 
     [BLOCKCHAIN]
     rpc_user = yourusername-as-in-bitcoin.conf
@@ -58,9 +97,10 @@ and edit:
 
 Note, you can also use a cookie file by setting, in this section, a variable `rpc_cookie_file` to the location of the file, as an alternative to using user/password.
 
-If you use Bitcoin Core's multiwallet feature, you can edit the value of `rpc_wallet_file` to your chosen wallet file.
+If you use Bitcoin Core's multiwallet feature, you can edit the value of `rpc_wallet_file` to your chosen wallet file. Just use a new line with `rpc_wallet_file = wallet2.dat` or similar.
+
+Then retry the `generate` command we mentioned above; it should now not error (see [below](#generate)).
 
-Then retry the same `generate` command; it should now not error (see [below](#generate)).
 If you still get rpc connection errors, make sure you can connect to your Core node using the command line first.
 
 <a name="wallet-tool" />

jmbase/jmbase/__init__.py

@@ -5,6 +5,7 @@ from builtins import *
 from .support import (get_log, chunks, debug_silence, jmprint,
                       joinmarket_alert, core_alert, get_password,
                       set_logging_level, set_logging_color,
-                      JM_WALLET_NAME_PREFIX)
+                      lookup_appdata_folder,
+                      JM_WALLET_NAME_PREFIX, JM_APP_NAME)
 from .commands import *
 

jmbase/jmbase/support.py

@@ -4,9 +4,12 @@ from builtins import * # noqa: F401
 
 import logging
 from getpass import getpass
+from os import path, environ
+import sys
 
 # global Joinmarket constants
 JM_WALLET_NAME_PREFIX = "joinmarket-wallet-"
+JM_APP_NAME = "joinmarket"
 
 # Exit status codes
 EXIT_SUCCESS = 0
@@ -124,3 +127,23 @@ def get_password(msg): #pragma: no cover
     if not isinstance(password, bytes):
         password = password.encode('utf-8')
     return password
+
+def lookup_appdata_folder(appname):
+    """ Given an appname as a string,
+    return the correct directory for storing
+    data for the given OS environment.
+    """
+    if sys.platform == 'darwin':
+        if "HOME" in environ:
+            data_folder = path.join(os.environ["HOME"],
+                                   "Library/Application support/",
+                                   appname) + '/'
+        else:
+            print("Could not find home folder")
+            os.exit()
+
+    elif 'win32' in sys.platform or 'win64' in sys.platform:
+        data_folder = path.join(environ['APPDATA'], appname) + '\\'
+    else:
+        data_folder = path.expanduser(path.join("~", "." + appname + "/"))
+    return data_folder

jmclient/jmclient/__init__.py

@@ -21,7 +21,7 @@ from .wallet import (Mnemonic, estimate_tx_fee, WalletError, BaseWallet, ImportW
 from .storage import (Argon2Hash, Storage, StorageError,
                       StoragePasswordError, VolatileStorage)
 from .cryptoengine import BTCEngine, BTC_P2PKH, BTC_P2SH_P2WPKH, EngineError
-from .configure import (
+from .configure import (load_test_config,
     load_program_config, get_p2pk_vbyte, jm_single, get_network, update_persist_config,
     validate_address, get_irc_mchannels, get_blockchain_interface_instance,
     get_p2sh_vbyte, set_config, is_segwit_mode, is_native_segwit_mode)
@@ -43,6 +43,11 @@ from .commitment_utils import get_utxo_info, validate_utxo_data, quit
 from .taker_utils import (tumbler_taker_finished_update, restart_waiter,
                              restart_wait, get_tumble_log, direct_send,
                              tumbler_filter_orders_callback)
+from .cli_options import (add_base_options, add_common_options,
+                          get_tumbler_parser, get_max_cj_fee_values,
+                          check_regtest, get_sendpayment_parser,
+                          get_default_max_absolute_fee,
+                          get_default_max_relative_fee)
 from .wallet_utils import (
     wallet_tool_main, wallet_generate_recover_bip39, open_wallet,
     open_test_wallet_maybe, create_wallet, get_wallet_cls, get_wallet_path,
@@ -50,6 +55,7 @@ from .wallet_utils import (
 from .wallet_service import WalletService
 from .maker import Maker, P2EPMaker
 from .yieldgenerator import YieldGenerator, YieldGeneratorBasic, ygmain
+
 # Set default logging handler to avoid "No handler found" warnings.
 
 try:

jmclient/jmclient/cli_options.py

@@ -7,6 +7,7 @@ from optparse import OptionParser, OptionValueError
 from configparser import NoOptionError
 
 import jmclient.support
+from jmbase import JM_APP_NAME
 from jmclient import jm_single, RegtestBitcoinCoreInterface, cryptoengine
 
 """This exists as a separate module for two reasons:
@@ -20,8 +21,37 @@ order_choose_algorithms = {
     'weighted_order_choose': '-W'
 }
 
+def add_base_options(parser):
+    """ Options for scripts common to all scripts
+    including maker, taker and non-coinjoin scripts.
+    See https://github.com/JoinMarket-Org/joinmarket-clientserver/issues/430
+    for more on how to further improve this.
+    Note that it's fine to have options here that are not used
+    in *all* scripts, as long as there is no conflict with another
+    usage of the option.
+    """
+    parser.add_option(
+        '--datadir',
+        dest='datadir',
+        default="",
+        help='Specify the path to a directory you want to use to store your user'
+        'data - wallets, logs and commitment files - and your joinmarket.cfg. '
+        'By default, the directory .' + JM_APP_NAME + ' is used.'
+    )
+    parser.add_option('--recoversync',
+                      action='store_true',
+                      dest='recoversync',
+                      default=False,
+                      help=('choose to do detailed wallet sync, '
+                            'used for recovering on new Core instance.'))
+    parser.add_option('--wallet-password-stdin',
+                      action='store_true',
+                      default=False,
+                      dest='wallet_password_stdin',
+                      help='Read wallet password from stdin')
 
 def add_common_options(parser):
+    add_base_options(parser)
     parser.add_option(
         '-f',
         '--txfee',
@@ -35,12 +65,6 @@ def add_common_options(parser):
         'confirmation target. This temporarily overrides the "tx_fees" setting '
         'in your joinmarket.cfg. Works the same way as described in it. Check '
         'it for examples.')
-    parser.add_option('--recoversync',
-                      action='store_true',
-                      dest='recoversync',
-                      default=False,
-                      help=('choose to do detailed wallet sync, '
-                            'used for recovering on new Core instance.'))
     parser.add_option(
         '-x',
         '--max-cj-fee-abs',
@@ -72,11 +96,6 @@ def add_common_options(parser):
              .format('random_under_max_order_choose',
                      ', '.join(order_choose_algorithms.keys())),
         dest='order_choose_fn')
-    parser.add_option('--wallet-password-stdin',
-                      action='store_true',
-                      default=False,
-                      dest='wallet_password_stdin',
-                      help='Read wallet password from stdin')
     add_order_choose_short_options(parser)
 
 

jmclient/jmclient/configure.py

@@ -13,7 +13,8 @@ from configparser import ConfigParser, NoOptionError
 import jmbitcoin as btc
 from jmclient.jsonrpc import JsonRpc
 from jmbase.support import (get_log, joinmarket_alert, core_alert, debug_silence,
-                            set_logging_level, jmprint, set_logging_color)
+                            set_logging_level, jmprint, set_logging_color,
+                            JM_APP_NAME, lookup_appdata_folder)
 from jmclient.podle import set_commitment_file
 
 log = get_log()
@@ -64,6 +65,8 @@ class AttributeDict(object):
 
 global_singleton = AttributeDict()
 global_singleton.JM_VERSION = 5
+global_singleton.APPNAME = JM_APP_NAME
+global_singleton.datadir = None
 global_singleton.nickname = None
 global_singleton.BITCOIN_DUST_THRESHOLD = 2730
 global_singleton.DUST_THRESHOLD = 10 * global_singleton.BITCOIN_DUST_THRESHOLD
@@ -423,14 +426,27 @@ def remove_unwanted_default_settings(config):
         if section.startswith('MESSAGING:'):
             config.remove_section(section)
 
-
-def load_program_config(config_path=None, bs=None):
+def load_program_config(config_path="", bs=None):
     global_singleton.config.readfp(io.StringIO(defaultconfig))
-    remove_unwanted_default_settings(global_singleton.config)
     if not config_path:
-        config_path = os.getcwd()
+        config_path = lookup_appdata_folder(global_singleton.APPNAME)
+    # we set the global home directory, but keep the config_path variable
+    # for callers of this function:
+    global_singleton.datadir = config_path
+    jmprint("User data location: " + global_singleton.datadir, "info")
+    if not os.path.exists(global_singleton.datadir):
+        os.makedirs(global_singleton.datadir)
+    # prepare folders for wallets and logs
+    if not os.path.exists(os.path.join(global_singleton.datadir, "wallets")):
+        os.makedirs(os.path.join(global_singleton.datadir, "wallets"))
+    if not os.path.exists(os.path.join(global_singleton.datadir, "logs")):
+        os.makedirs(os.path.join(global_singleton.datadir, "logs"))
+    if not os.path.exists(os.path.join(global_singleton.datadir, "cmtdata")):
+        os.makedirs(os.path.join(global_singleton.datadir, "cmtdata"))
     global_singleton.config_location = os.path.join(
-        config_path, global_singleton.config_location)
+        global_singleton.datadir, global_singleton.config_location)
+
+    remove_unwanted_default_settings(global_singleton.config)
     loadedFiles = global_singleton.config.read([global_singleton.config_location
                                                ])
     #Hack required for electrum; must be able to enforce a different
@@ -491,17 +507,28 @@ def load_program_config(config_path=None, bs=None):
     global_singleton.bc_interface = get_blockchain_interface_instance(
         global_singleton.config)
 
-    #set the location of the commitments file
+    # set the location of the commitments file; for non-mainnet a different
+    # file is used to avoid conflict
     try:
         global_singleton.commit_file_location = global_singleton.config.get(
             "POLICY", "commit_file_location")
     except NoOptionError: #pragma: no cover
         log.debug("No commitment file location in config, using default "
                   "location cmtdata/commitments.json")
+    if get_network() != "mainnet":
+        # no need to be flexible for tests; note this is used
+        # for regtest as well as testnet(3)
+        global_singleton.commit_file_location = "cmtdata/testnet_commitments.json"
     set_commitment_file(os.path.join(config_path,
                                          global_singleton.commit_file_location))
 
 
+def load_test_config(**kwargs):
+    if "config_path" not in kwargs:
+        load_program_config(config_path=".", **kwargs)
+    else:
+        load_program_config(**kwargs)
+
 ##########################################################
 ## Returns a tuple (rpc_user: String, rpc_pass: String) ##
 ##########################################################

jmclient/jmclient/wallet_utils.py

@@ -15,7 +15,8 @@ from itertools import islice
 from jmclient import (get_network, WALLET_IMPLEMENTATIONS, Storage, podle,
     jm_single, BitcoinCoreInterface, JsonRpcError, WalletError,
     VolatileStorage, StoragePasswordError, is_segwit_mode, SegwitLegacyWallet,
-    LegacyWallet, SegwitWallet, is_native_segwit_mode)
+    LegacyWallet, SegwitWallet, is_native_segwit_mode, load_program_config,
+    add_base_options, check_regtest)
 from jmclient.wallet_service import WalletService
 from jmbase.support import get_password, jmprint, EXIT_FAILURE, EXIT_ARGERROR
 
@@ -47,6 +48,7 @@ def get_wallettool_parser():
         '(freeze) Freeze or un-freeze a specific utxo. Specify mixdepth with -m.')
     parser = OptionParser(usage='usage: %prog [options] [wallet file] [method]',
                           description=description)
+    add_base_options(parser)
     parser.add_option('-p',
                       '--privkey',
                       action='store_true',
@@ -79,12 +81,6 @@ def get_wallettool_parser():
                       default=1,
                       help=('History method verbosity, 0 (least) to 6 (most), '
                             '<=2 batches earnings, even values also list TXIDs'))
-    parser.add_option('--recoversync',
-                      action='store_true',
-                      dest='recoversync',
-                      default=False,
-                      help=('choose to do detailed wallet sync, '
-                            'used for recovering on new Core instance.'))
     parser.add_option('-H',
                       '--hd',
                       action='store',
@@ -99,15 +95,9 @@ def get_wallettool_parser():
                       default=None,
                       help=("Key type when importing private keys.\n"
                             "If your address starts with '1' use 'standard', "
-                            "if your address starts with '3' use 'segwit-p2sh.\n"
-                            "Native segwit addresses (starting with 'bc') are"
+                            "if your address starts with '3' use 'segwit-p2sh'.\n"
+                            "Native segwit addresses (starting with 'bc') are "
                             "not yet supported."))
-    parser.add_option('--wallet-password-stdin',
-                      action='store_true',
-                      default=False,
-                      dest='wallet_password_stdin',
-                      help='Read wallet password from stdin')
-
 
     return parser
 
@@ -1203,9 +1193,9 @@ def wallet_sanity_check(wallet):
                         "is on '{}'.".format(get_network(), wallet.network))
 
 
-def get_wallet_path(file_name, wallet_dir):
-    # TODO: move default wallet path to ~/.joinmarket
-    wallet_dir = wallet_dir or 'wallets'
+def get_wallet_path(file_name, wallet_dir=None):
+    if not wallet_dir:
+        wallet_dir = os.path.join(jm_single().datadir, 'wallets')
     return os.path.join(wallet_dir, file_name)
 
 
@@ -1214,7 +1204,10 @@ def wallet_tool_main(wallet_root_path):
     """
     parser = get_wallettool_parser()
     (options, args) = parser.parse_args()
-
+    load_program_config(config_path=options.datadir)
+    check_regtest(blockchain_start=False)
+    # full path to the wallets/ subdirectory in the user data area:
+    wallet_root_path = os.path.join(jm_single().datadir, wallet_root_path)
     noseed_methods = ['generate', 'recover']
     methods = ['display', 'displayall', 'summary', 'showseed', 'importprivkey',
                'history', 'showutxos', 'freeze']

jmclient/jmclient/yieldgenerator.py

@@ -12,8 +12,8 @@ from twisted.python.log import startLogging
 from optparse import OptionParser
 from jmbase import get_log
 from jmclient import Maker, jm_single, load_program_config, \
-    JMClientProtocolFactory, start_reactor, \
-    calc_cj_fee, WalletService
+    JMClientProtocolFactory, start_reactor, calc_cj_fee, \
+    WalletService, add_base_options
 from .wallet_utils import open_test_wallet_maybe, get_wallet_path
 from jmbase.support import EXIT_ARGERROR
 
@@ -194,6 +194,7 @@ def ygmain(ygclass, txfee=1000, cjfee_a=200, cjfee_r=0.002, ordertype='swreloffe
     import sys
 
     parser = OptionParser(usage='usage: %prog [options] [wallet file]')
+    add_base_options(parser)
     parser.add_option('-o', '--ordertype', action='store', type='string',
                       dest='ordertype', default=ordertype,
                       help='type of order; can be either reloffer or absoffer')
@@ -212,20 +213,9 @@ def ygmain(ygclass, txfee=1000, cjfee_a=200, cjfee_r=0.002, ordertype='swreloffe
     parser.add_option('-g', '--gap-limit', action='store', type="int",
                       dest='gaplimit', default=gaplimit,
                       help='gap limit for wallet, default='+str(gaplimit))
-    parser.add_option('--recoversync',
-                      action='store_true',
-                      dest='recoversync',
-                      default=False,
-                      help=('choose to do detailed wallet sync, '
-                            'used for recovering on new Core instance.'))
     parser.add_option('-m', '--mixdepth', action='store', type='int',
                       dest='mixdepth', default=None,
                       help="highest mixdepth to use")
-    parser.add_option('--wallet-password-stdin',
-                      action='store_true',
-                      default=False,
-                      dest='wallet_password_stdin',
-                      help='Read wallet password from stdin')
     (options, args) = parser.parse_args()
     if len(args) < 1:
         parser.error('Needs a wallet')
@@ -249,9 +239,9 @@ def ygmain(ygclass, txfee=1000, cjfee_a=200, cjfee_r=0.002, ordertype='swreloffe
         sys.exit(EXIT_ARGERROR)
     nickserv_password = options.password
 
-    load_program_config()
+    load_program_config(config_path=options.datadir)
 
-    wallet_path = get_wallet_path(wallet_name, 'wallets')
+    wallet_path = get_wallet_path(wallet_name, None)
     wallet = open_test_wallet_maybe(
         wallet_path, wallet_name, options.mixdepth,
         wallet_password_stdin=options.wallet_password_stdin,

jmclient/test/test_blockchaininterface.py

@@ -9,7 +9,7 @@ from commontest import create_wallet_for_sync
 
 import pytest
 from jmbase import get_log
-from jmclient import load_program_config, jm_single
+from jmclient import load_test_config, jm_single
 
 log = get_log()
 
@@ -157,5 +157,5 @@ def test_imported_wallet_sync(setup_wallets, fast):
 
 @pytest.fixture(scope='module')
 def setup_wallets():
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface.tick_forward_chain_interval = 1

jmclient/test/test_client_protocol.py

@@ -5,7 +5,7 @@ from builtins import *
 '''test client-protocol interfacae.'''
 
 from jmbase import get_log
-from jmclient import load_program_config, Taker,\
+from jmclient import load_test_config, Taker,\
     JMClientProtocolFactory, jm_single, Maker, WalletService
 from jmclient.client_protocol import JMTakerClientProtocol
 from twisted.python.log import msg as tmsg
@@ -274,7 +274,7 @@ class TrialTestJMClientProto(unittest.TestCase):
         global clientfactory
         print("setUp()")
         params = [[False, False], [True, False], [False, True], [-1, False]]
-        load_program_config()
+        load_test_config()
         jm_single().maker_timeout_sec = 1
         self.port = reactor.listenTCP(28184, JMTestServerProtocolFactory())
         self.addCleanup(self.port.stopListening)
@@ -337,7 +337,7 @@ class TestMakerClientProtocol(unittest.TestCase):
         return d
 
     def setUp(self):
-        load_program_config()
+        load_test_config()
         factory = JMClientProtocolFactory(DummyMaker(), proto_type='MAKER')
         self.client = factory.buildProtocol(None)
         self.tr = proto_helpers.StringTransport()

jmclient/test/test_coinjoin.py

@@ -12,7 +12,7 @@ import pytest
 from twisted.internet import reactor
 
 from jmbase import get_log
-from jmclient import load_program_config, jm_single,\
+from jmclient import load_test_config, jm_single,\
     YieldGeneratorBasic, Taker, LegacyWallet, SegwitLegacyWallet,\
     NO_ROUNDING
 from jmclient.podle import set_commitment_file
@@ -319,7 +319,7 @@ def test_coinjoin_mixed_maker_addresses(monkeypatch, tmpdir, setup_cj,
 
 @pytest.fixture(scope='module')
 def setup_cj():
-    load_program_config()
+    load_test_config()
     jm_single().config.set('POLICY', 'tx_broadcast', 'self')
     jm_single().bc_interface.tick_forward_chain_interval = 5
     jm_single().bc_interface.simulate_blocks()

jmclient/test/test_commitment_utils.py

@@ -5,12 +5,12 @@ from builtins import * # noqa: F401
 from commontest import DummyBlockchainInterface
 import pytest
 
-from jmclient import (load_program_config, jm_single)
+from jmclient import (load_test_config, jm_single)
 from jmclient.commitment_utils import get_utxo_info, validate_utxo_data
 
 
 def test_get_utxo_info():
-    load_program_config()
+    load_test_config()
     jm_single().config.set("BLOCKCHAIN", "network", "mainnet")
     dbci = DummyBlockchainInterface()
     privkey = "L1RrrnXkcKut5DEMwtDthjwRcTTwED36thyL1DebVrKuwvohjMNi"

jmclient/test/test_configure.py

@@ -5,7 +5,7 @@ from builtins import * # noqa: F401
 
 import pytest
 import struct
-from jmclient import load_program_config, jm_single, get_irc_mchannels
+from jmclient import load_test_config, jm_single, get_irc_mchannels
 from jmclient.configure import (get_config_irc_channel, get_p2sh_vbyte,
                                 get_p2pk_vbyte, get_blockchain_interface_instance)
 
@@ -20,32 +20,32 @@ def test_attribute_dict():
 
 
 def test_load_config(tmpdir):
-    load_program_config(bs="regtest")
+    load_test_config(bs="regtest")
     jm_single().config_location = "joinmarket.cfg"
     with pytest.raises(SystemExit):
-        load_program_config(config_path=str(tmpdir), bs="regtest")
+        load_test_config(config_path=str(tmpdir), bs="regtest")
     jm_single().config_location = "joinmarket.cfg"
-    load_program_config()
+    load_test_config()
 
 
 def test_config_get_irc_channel():
-    load_program_config()
+    load_test_config()
     channel = "dummy"
     assert get_config_irc_channel(channel) == "#dummy-test"
     jm_single().config.set("BLOCKCHAIN", "network", "mainnet")
     assert get_config_irc_channel(channel) == "#dummy"
     get_irc_mchannels()
-    load_program_config()
+    load_test_config()
 
 
 def test_net_byte():
-    load_program_config()
+    load_test_config()
     assert struct.unpack(b'B', get_p2pk_vbyte())[0] == 0x6f
     assert struct.unpack(b'B', get_p2sh_vbyte())[0] == 196
 
 
 def test_blockchain_sources():
-    load_program_config()
+    load_test_config()
     for src in ["electrum", "dummy"]:
         jm_single().config.set("BLOCKCHAIN", "blockchain_source", src)
         if src=="electrum":
@@ -55,4 +55,4 @@ def test_blockchain_sources():
                 get_blockchain_interface_instance(jm_single().config)
         else:
             get_blockchain_interface_instance(jm_single().config)
-    load_program_config()
+    load_test_config()

jmclient/test/test_core_nohistory_sync.py

@@ -10,12 +10,12 @@ from commontest import create_wallet_for_sync
 
 import pytest
 from jmbase import get_log
-from jmclient import load_program_config
+from jmclient import load_test_config
 
 log = get_log()
 
 def test_fast_sync_unavailable(setup_sync):
-    load_program_config(bs="bitcoin-rpc-no-history")
+    load_test_config(bs="bitcoin-rpc-no-history")
     wallet_service = create_wallet_for_sync([0, 0, 0, 0, 0],
         ['test_fast_sync_unavailable'])
     with pytest.raises(RuntimeError) as e_info:
@@ -23,7 +23,7 @@ def test_fast_sync_unavailable(setup_sync):
 
 @pytest.mark.parametrize('internal', (False, True))
 def test_sync(setup_sync, internal):
-    load_program_config(bs="bitcoin-rpc-no-history")
+    load_test_config(bs="bitcoin-rpc-no-history")
     used_count = [1, 3, 6, 2, 23]
     wallet_service = create_wallet_for_sync(used_count, ['test_sync'],
         populate_internal=internal)

jmclient/test/test_maker.py

@@ -6,7 +6,7 @@ from builtins import * # noqa: F401
 
 import jmbitcoin as btc
 from jmclient import Maker, get_p2sh_vbyte, get_p2pk_vbyte, \
-    load_program_config, jm_single, WalletService
+    load_test_config, jm_single, WalletService
 import jmclient
 from commontest import DummyBlockchainInterface
 from test_taker import DummyWallet
@@ -179,4 +179,4 @@ def test_verify_unsigned_tx_nonsw_valid(setup_env_nodeps):
 def setup_env_nodeps(monkeypatch):
     monkeypatch.setattr(jmclient.configure, 'get_blockchain_interface_instance',
                         lambda x: DummyBlockchainInterface())
-    load_program_config()
+    load_test_config()

jmclient/test/test_payjoin.py

@@ -12,7 +12,7 @@ import pytest
 from twisted.internet import reactor
 from jmbase import get_log
 from jmclient import cryptoengine
-from jmclient import (load_program_config, jm_single,
+from jmclient import (load_test_config, jm_single,
                       P2EPMaker, P2EPTaker,
                       LegacyWallet, SegwitLegacyWallet, SegwitWallet)
 from commontest import make_wallets
@@ -150,7 +150,7 @@ def test_simple_payjoin(monkeypatch, tmpdir, setup_cj, wallet_cls,
 
 @pytest.fixture(scope='module')
 def setup_cj():
-    load_program_config()
+    load_test_config()
     jm_single().config.set('POLICY', 'tx_broadcast', 'self')
     jm_single().bc_interface.tick_forward_chain_interval = 5
     jm_single().bc_interface.simulate_blocks()

jmclient/test/test_podle.py

@@ -11,7 +11,7 @@ import json
 import pytest
 import copy
 from jmbase import get_log
-from jmclient import load_program_config, jm_single, generate_podle,\
+from jmclient import load_test_config, jm_single, generate_podle,\
     generate_podle_error_string, get_commitment_file, PoDLE,\
     get_podle_commitments, add_external_commitments, update_commitments
 from jmclient.podle import verify_all_NUMS, verify_podle, PoDLEError
@@ -217,7 +217,7 @@ def test_podle_error_string(setup_podle):
 
 @pytest.fixture(scope="module")
 def setup_podle(request):
-    load_program_config()
+    load_test_config()
     if not os.path.exists("cmtdata"):
         os.mkdir("cmtdata")
     prev_commits = False

jmclient/test/test_schedule.py

@@ -6,7 +6,7 @@ from builtins import * # noqa: F401
 
 import pytest
 from jmclient import (get_schedule, get_tumble_schedule,
-                      tweak_tumble_schedule, load_program_config)
+                      tweak_tumble_schedule, load_test_config)
 import os
 
 valids = """#sample for testing
@@ -38,7 +38,7 @@ invalids4 = """#sample for testing
 
 
 def test_get_schedule():
-    load_program_config()
+    load_test_config()
     tsf = "schedulefortesting"
     for s in [valids, invalids1, invalids2, invalids3, invalids4]:
         if os.path.exists(tsf):
@@ -123,7 +123,7 @@ def test_tumble_schedule(destaddrs, txcparams, mixdepthcount):
     ])
 def test_tumble_tweak(destaddrs, txcparams, mixdepthcount, lastcompleted,
                       makercountrange):
-    load_program_config()
+    load_test_config()
     options = get_options()
     options['mixdepthcount'] = mixdepthcount
     options['txcountparams'] = txcparams

jmclient/test/test_taker.py

@@ -13,7 +13,7 @@ import pytest
 import json
 import struct
 from base64 import b64encode
-from jmclient import load_program_config, jm_single, set_commitment_file,\
+from jmclient import load_test_config, jm_single, set_commitment_file,\
     get_commitment_file, SegwitLegacyWallet, Taker, VolatileStorage,\
     get_network, WalletService, NO_ROUNDING
 from taker_test_data import t_utxos_by_mixdepth, t_orderbook,\
@@ -490,6 +490,6 @@ def setup_taker(request):
     request.addfinalizer(cmtdatateardown)
     if not os.path.exists("cmtdata"):
             os.makedirs("cmtdata")
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface = DummyBlockchainInterface()
     jm_single().config.set("BLOCKCHAIN", "network", "testnet")

jmclient/test/test_tx_creation.py

@@ -13,7 +13,7 @@ from commontest import make_wallets, make_sign_and_push
 import jmbitcoin as bitcoin
 import pytest
 from jmbase import get_log
-from jmclient import load_program_config, jm_single,\
+from jmclient import load_test_config, jm_single,\
     get_p2pk_vbyte
 
 log = get_log()
@@ -283,4 +283,4 @@ def test_spend_p2wsh(setup_tx_creation):
 
 @pytest.fixture(scope="module")
 def setup_tx_creation():
-    load_program_config()
+    load_test_config()

jmclient/test/test_utxomanager.py

@@ -6,7 +6,7 @@ from jmclient.wallet import UTXOManager
 from test_storage import MockStorage
 import pytest
 
-from jmclient import load_program_config
+from jmclient import load_test_config
 import jmclient
 from commontest import DummyBlockchainInterface
 
@@ -126,4 +126,4 @@ def test_utxomanager_select(setup_env_nodeps):
 def setup_env_nodeps(monkeypatch):
     monkeypatch.setattr(jmclient.configure, 'get_blockchain_interface_instance',
                         lambda x: DummyBlockchainInterface())
-    load_program_config()
+    load_test_config()

jmclient/test/test_valid_addresses.py

@@ -1,7 +1,7 @@
 from __future__ import (absolute_import, division,
                         print_function, unicode_literals)
 from builtins import * # noqa: F401
-from jmclient.configure import validate_address, load_program_config
+from jmclient.configure import validate_address, load_test_config
 from jmclient import jm_single
 import json
 import pytest
@@ -77,4 +77,4 @@ def test_invalid_bech32_addresses():
 
 @pytest.fixture(scope="module")
 def setup_addresses():
-    load_program_config()
+    load_test_config()

jmclient/test/test_wallet.py

@@ -11,7 +11,7 @@ import pytest
 import jmbitcoin as btc
 from commontest import binarize_tx
 from jmbase import get_log
-from jmclient import load_program_config, jm_single, \
+from jmclient import load_test_config, jm_single, \
     SegwitLegacyWallet,BIP32Wallet, BIP49Wallet, LegacyWallet,\
     VolatileStorage, get_network, cryptoengine, WalletError,\
     SegwitWallet, WalletService
@@ -663,7 +663,7 @@ def test_wallet_mixdepth_decrease(setup_wallet):
 
 @pytest.fixture(scope='module')
 def setup_wallet():
-    load_program_config()
+    load_test_config()
     #see note in cryptoengine.py:
     cryptoengine.BTC_P2WPKH.VBYTE = 100
     jm_single().bc_interface.tick_forward_chain_interval = 2

jmclient/test/test_wallets.py

@@ -13,7 +13,7 @@ import json
 import pytest
 from jmbase import get_log
 from jmclient import (
-    load_program_config, jm_single,
+    load_test_config, jm_single,
     estimate_tx_fee, BitcoinCoreInterface, Mnemonic)
 from taker_test_data import t_raw_signed_tx
 testdir = os.path.dirname(os.path.realpath(__file__))
@@ -38,7 +38,7 @@ def do_tx(wallet_service, amount):
 
 
 def test_query_utxo_set(setup_wallets):
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface.tick_forward_chain_interval = 1
     wallet_service = create_wallet_for_sync([2, 3, 0, 0, 0],
                                     ["wallet4utxo.json", "4utxo", [2, 3]])
@@ -78,7 +78,7 @@ def test_pushtx_errors(setup_wallets):
     jm_single().bc_interface.jsonRpc.port = 18333
     assert not jm_single().bc_interface.pushtx(t_raw_signed_tx)
     #rebuild a valid jsonrpc inside the bci
-    load_program_config()
+    load_test_config()
 
 
 """Tests mainly for wallet.py"""
@@ -88,7 +88,7 @@ def test_absurd_fee(setup_wallets):
     jm_single().config.set("POLICY", "absurd_fee_per_kb", "1000")
     with pytest.raises(ValueError) as e_info:
         estimate_tx_fee(10, 2)
-    load_program_config()
+    load_test_config()
 
 
 def check_bip39_case(vectors, language="english"):
@@ -120,5 +120,5 @@ def test_bip39_vectors(setup_wallets):
 
 @pytest.fixture(scope="module")
 def setup_wallets():
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface.tick_forward_chain_interval = 2

jmclient/test/test_walletservice.py

@@ -6,7 +6,7 @@ from builtins import * # noqa: F401
 import os
 import pytest
 from jmbase import get_log
-from jmclient import load_program_config, jm_single, \
+from jmclient import load_test_config, jm_single, \
      WalletService
 from test_blockchaininterface import sync_test_wallet
 from test_wallet import fund_wallet_addr, get_populated_wallet
@@ -72,7 +72,7 @@ def test_address_reuse_freezing(setup_walletservice):
 
 @pytest.fixture(scope='module')
 def setup_walletservice(request):
-    load_program_config()
+    load_test_config()
     old_reuse_freeze_val = jm_single().config.getint("POLICY",
                                     "max_sats_freeze_reuse")
     def reset_config():

jmclient/test/test_yieldgenerator.py

@@ -4,7 +4,7 @@ from builtins import * # noqa: F401
 
 import unittest
 
-from jmclient import load_program_config, jm_single,\
+from jmclient import load_test_config, jm_single,\
     SegwitLegacyWallet, VolatileStorage, YieldGeneratorBasic, \
     get_network, WalletService
 
@@ -20,7 +20,7 @@ class CustomUtxoWallet(SegwitLegacyWallet):
         as given by the array.  (And the number of mixdepths from the array
         elements."""
 
-        load_program_config()
+        load_test_config()
 
         storage = VolatileStorage()
         super(CustomUtxoWallet, self).initialize(storage, get_network(),

jmdaemon/test/test_daemon_protocol.py

@@ -11,7 +11,7 @@ from jmdaemon.daemon_protocol import JMDaemonServerProtocol
 from jmdaemon.protocol import NICK_HASH_LENGTH, NICK_MAX_ENCODED, JM_VERSION,\
     JOINMARKET_NICK_HEADER
 from jmbase import get_log
-from jmclient import (load_program_config, jm_single, get_irc_mchannels)
+from jmclient import (load_test_config, jm_single, get_irc_mchannels)
 from twisted.python.log import msg as tmsg
 from twisted.python.log import startLogging
 from twisted.internet import protocol, reactor, task
@@ -294,7 +294,7 @@ class TrialTestJMDaemonProto(unittest.TestCase):
 
     def setUp(self):
         startLogging(sys.stdout)
-        load_program_config()
+        load_test_config()
         jm_single().maker_timeout_sec = 1
         self.port = reactor.listenTCP(28184, JMDaemonTestServerProtocolFactory())
         self.addCleanup(self.port.stopListening)
@@ -314,7 +314,7 @@ class TestJMDaemonProtoInit(unittest.TestCase):
     def setUp(self):
         global end_early
         end_early = True
-        load_program_config()
+        load_test_config()
         jm_single().maker_timeout_sec = 1
         self.port = reactor.listenTCP(28184, JMDaemonTest2ServerProtocolFactory())
         self.addCleanup(self.port.stopListening)

jmdaemon/test/test_irc_messaging.py

@@ -9,7 +9,7 @@ from twisted.trial import unittest
 from twisted.internet import reactor, task
 from jmdaemon import IRCMessageChannel, MessageChannelCollection
 #needed for test framework
-from jmclient import (load_program_config, get_irc_mchannels, jm_single)
+from jmclient import (load_test_config, get_irc_mchannels, jm_single)
 
 si = 1
 class DummyDaemon(object):
@@ -110,7 +110,7 @@ def getmc(nick):
 class TrialIRC(unittest.TestCase):
 
     def setUp(self):
-        load_program_config()
+        load_test_config()
         print(get_irc_mchannels()[0])
         jm_single().maker_timeout_sec = 1
         dm, mc, mcc = getmc("irc_publisher")

jmdaemon/test/test_orderbookwatch.py

@@ -7,7 +7,7 @@ import pytest
 
 from jmdaemon.orderbookwatch import OrderbookWatch
 from jmdaemon import IRCMessageChannel
-from jmclient import get_irc_mchannels, load_program_config
+from jmclient import get_irc_mchannels, load_test_config
 from jmdaemon.protocol import JM_VERSION, ORDER_KEYS
 class DummyDaemon(object):
     def request_signature_verify(self, a, b, c, d, e,
@@ -24,7 +24,7 @@ def on_welcome(x):
     print("Simulated on-welcome")
 
 def get_ob():
-    load_program_config()
+    load_test_config()
     dm = DummyDaemon()
     mc = DummyMC(get_irc_mchannels()[0], "test", dm)
     ob = OrderbookWatch()

scripts/add-utxo.py

@@ -16,13 +16,12 @@ import binascii
 from pprint import pformat
 
 from optparse import OptionParser
-from jmbase import jmprint
 import jmbitcoin as btc
 from jmclient import load_program_config, jm_single, get_p2pk_vbyte,\
     open_wallet, WalletService, add_external_commitments, update_commitments,\
     PoDLE, get_podle_commitments, get_utxo_info, validate_utxo_data, quit,\
-    get_wallet_path
-from jmbase.support import EXIT_SUCCESS, EXIT_FAILURE, EXIT_ARGERROR
+    get_wallet_path, add_base_options
+from jmbase.support import EXIT_SUCCESS, EXIT_FAILURE, EXIT_ARGERROR, jmprint
 
 
 def add_ext_commitments(utxo_datas):
@@ -79,6 +78,7 @@ def main():
                     
                     "Also note this ONLY works for standard (p2pkh or p2sh-p2wpkh) utxos."
     )
+    add_base_options(parser)
     parser.add_option(
         '-r',
         '--read-from-file',
@@ -147,14 +147,8 @@ def main():
         help='only validate the provided utxos (file or command line), not add',
         default=False
     )
-    parser.add_option('--recoversync',
-                      action='store_true',
-                      dest='recoversync',
-                      default=False,
-                      help=('choose to do detailed wallet sync, '
-                            'used for recovering on new Core instance.'))
     (options, args) = parser.parse_args()
-    load_program_config()
+    load_program_config(config_path=options.datadir)
     #TODO; sort out "commit file location" global so this script can
     #run without this hardcoding:
     utxo_data = []
@@ -178,7 +172,7 @@ def main():
     #Three options (-w, -r, -R) for loading utxo and privkey pairs from a wallet,
     #csv file or json file.
     if options.loadwallet:
-        wallet_path = get_wallet_path(options.loadwallet, None)
+        wallet_path = get_wallet_path(options.loadwallet)
         wallet = open_wallet(wallet_path, gap_limit=options.gaplimit)
         wallet_service = WalletService(wallet)
         while True:

scripts/convert_old_wallet.py

@@ -9,7 +9,7 @@ from hashlib import sha256
 from binascii import hexlify, unhexlify
 from collections import defaultdict
 from pyaes import AESModeOfOperationCBC, Decrypter
-
+from jmbase import JM_APP_NAME
 from jmclient import Storage, load_program_config
 from jmclient.wallet_utils import get_password, get_wallet_cls,\
     cli_get_wallet_passphrase_check, get_wallet_path
@@ -135,13 +135,22 @@ def main():
     parser.add_argument('old_wallet_file', type=open)
     parser.add_argument('--name', '-n', required=False, dest='name',
                         help="Name of the new wallet file. Default: [old wallet name].jmdat")
+    # hack; not using jmclient.add_base_options because ArgumentParser not OptionParser:
+    parser.add_argument(
+        '--datadir',
+        dest='datadir',
+        default="",
+        help='Specify the path to a directory you want to use to store your user'
+        'data - wallets, logs and commitment files - and your joinmarket.cfg. '
+        'By default, the directory .' + JM_APP_NAME + ' is used.'
+    )
 
     try:
         args = parser.parse_args()
     except Exception as e:
         print("Error: {}".format(e))
         return
-
+    load_program_config(config_path=args.datadir)
     data = parse_old_wallet(args.old_wallet_file)
 
     if not data:
@@ -157,5 +166,4 @@ def main():
 
 
 if __name__ == '__main__':
-    load_program_config()
     main()

scripts/joinmarket-qt.py

@@ -75,14 +75,12 @@ from jmclient import load_program_config, get_network, update_persist_config,\
     RegtestBitcoinCoreInterface, tumbler_taker_finished_update,\
     get_tumble_log, restart_wait, tumbler_filter_orders_callback,\
     wallet_generate_recover_bip39, wallet_display, get_utxos_enabled_disabled,\
-    NO_ROUNDING
+    NO_ROUNDING, get_max_cj_fee_values, get_default_max_absolute_fee, \
+    get_default_max_relative_fee
 from qtsupport import ScheduleWizard, TumbleRestartWizard, config_tips,\
     config_types, QtHandler, XStream, Buttons, OkButton, CancelButton,\
     PasswordDialog, MyTreeWidget, JMQtMessageBox, BLUE_FG,\
     donation_more_message
-# TODO refactor; these functions do not belong in cli_options:
-from cli_options import get_max_cj_fee_values, get_default_max_absolute_fee, \
-     get_default_max_relative_fee
 
 from twisted.internet import task
 
@@ -1452,11 +1450,13 @@ class JMMainWindow(QMainWindow):
         privkeys_fn = privkeys_fn_base
         # Updated to use json format, simply because L1354 writer
         # has some extremely weird behaviour cross Py2/Py3
-        while os.path.isfile(privkeys_fn + '.json'):
+        while os.path.isfile(os.path.join(jm_single().datadir,
+                                          privkeys_fn + '.json')):
             i += 1
             privkeys_fn = privkeys_fn_base + str(i)
         try:
-            with open(privkeys_fn + '.json', "wb") as f:
+            with open(os.path.join(jm_single().datadir,
+                                   privkeys_fn + '.json'), "wb") as f:
                 for addr, pk in private_keys.items():
                     #sanity check
                     if not addr == btc.pubkey_to_p2sh_p2wpkh_address(
@@ -1480,8 +1480,8 @@ class JMMainWindow(QMainWindow):
             return
 
         JMQtMessageBox(self,
-                       "Private keys exported to: " + privkeys_fn + '.json',
-                       title="Success")
+                       "Private keys exported to: " + os.path.join(jm_single().datadir,
+                        privkeys_fn) + '.json', title="Success")
 
     def seedEntry(self):
         d = QDialog(self)
@@ -1559,12 +1559,11 @@ class JMMainWindow(QMainWindow):
 
     def selectWallet(self, testnet_seed=None):
         if jm_single().config.get("BLOCKCHAIN", "blockchain_source") != "regtest":
-            current_path = os.path.dirname(os.path.realpath(__file__))
-            if os.path.isdir(os.path.join(current_path, 'wallets')):
-                current_path = os.path.join(current_path, 'wallets')
+            # guaranteed to exist as load_program_config was called on startup:
+            wallets_path = os.path.join(jm_single().datadir, 'wallets')
             firstarg = QFileDialog.getOpenFileName(self,
                                                    'Choose Wallet File',
-                                                   directory=current_path,
+                                                   wallets_path,
                                                    options=QFileDialog.DontUseNativeDialog)
             #TODO validate the file looks vaguely like a wallet file
             log.debug('Looking for wallet in: ' + str(firstarg))
@@ -1749,8 +1748,10 @@ class JMMainWindow(QMainWindow):
         '''
         if not seed:
             try:
+                # guaranteed to exist as load_program_config was called on startup:
+                wallets_path = os.path.join(jm_single().datadir, 'wallets')
                 success = wallet_generate_recover_bip39("generate",
-                                                   "wallets",
+                                                   wallets_path,
                                                    "wallet.jmdat",
                                                    callbacks=(self.displayWords,
                                                               None,
@@ -1806,6 +1807,8 @@ def get_wallet_printout(wallet_service):
 ################################
 config_load_error = False
 try:
+    # note: uses default config_path value of "" always, i.e. user
+    # data is always in ~/.joinmarket for JM-QT
     load_program_config()
 except Exception as e:
     config_load_error = "Failed to setup joinmarket: "+repr(e)

scripts/obwatch/ob-watcher.py

@@ -445,8 +445,6 @@ def get_dummy_nick():
     return nick
 
 def main():
-    load_program_config(config_path='..')
-
     parser = OptionParser(
             usage='usage: %prog [options]',
             description='Runs a webservice which shows the orderbook.')
@@ -465,7 +463,7 @@ def main():
                       help='port to listen on, default=62601',
                       default=62601)
     (options, args) = parser.parse_args()
-
+    load_program_config(config_path=options.datadir)
     hostport = (options.host, options.port)
     mcs = [ObIRCMessageChannel(c) for c in get_irc_mchannels()]
     mcc = MessageChannelCollection(mcs)

scripts/receive-payjoin.py

@@ -10,8 +10,8 @@ from twisted.python.log import startLogging
 from jmbase import get_log, set_logging_level
 from jmclient import P2EPMaker, jm_single, load_program_config, \
     WalletService, JMClientProtocolFactory, start_reactor, \
-    open_test_wallet_maybe, get_wallet_path
-from cli_options import check_regtest
+    open_test_wallet_maybe, get_wallet_path, check_regtest, \
+    add_base_options
 from jmbase.support import EXIT_FAILURE, EXIT_ARGERROR
 from jmbitcoin import amount_to_sat
 
@@ -19,15 +19,10 @@ jlog = get_log()
 
 def receive_payjoin_main(makerclass):
     parser = OptionParser(usage='usage: %prog [options] [wallet file] [amount-to-receive]')
+    add_base_options(parser)
     parser.add_option('-g', '--gap-limit', action='store', type="int",
                       dest='gaplimit', default=6,
                       help='gap limit for wallet, default=6')
-    parser.add_option('--recoversync',
-                      action='store_true',
-                      dest='recoversync',
-                      default=False,
-                      help=('choose to do detailed wallet sync, '
-                            'used for recovering on new Core instance.'))
     parser.add_option('-m', '--mixdepth', action='store', type='int',
                       dest='mixdepth', default=0,
                       help="mixdepth to source coins from")
@@ -37,12 +32,7 @@ def receive_payjoin_main(makerclass):
                       type='int',
                       dest='amtmixdepths',
                       help='number of mixdepths in wallet, default 5',
-                      default=5)    
-    parser.add_option('--wallet-password-stdin',
-                      action='store_true',
-                      default=False,
-                      dest='wallet_password_stdin',
-                      help='Read wallet password from stdin')
+                      default=5)
 
     (options, args) = parser.parse_args()
     if len(args) < 2:
@@ -57,7 +47,7 @@ def receive_payjoin_main(makerclass):
     if receiving_amount < 0:
         parser.error("Receiving amount must be a positive number")
         sys.exit(EXIT_FAILURE)
-    load_program_config()
+    load_program_config(config_path=options.datadir)
 
     check_regtest()
 

scripts/sendpayment.py

@@ -17,13 +17,12 @@ import pprint
 from jmclient import Taker, P2EPTaker, load_program_config, get_schedule,\
     JMClientProtocolFactory, start_reactor, validate_address, jm_single,\
     estimate_tx_fee, direct_send, WalletService,\
-    open_test_wallet_maybe, get_wallet_path, NO_ROUNDING
+    open_test_wallet_maybe, get_wallet_path, NO_ROUNDING, \
+    get_sendpayment_parser, get_max_cj_fee_values, check_regtest
 from twisted.python.log import startLogging
 from jmbase.support import get_log, set_logging_level, jmprint, \
     EXIT_FAILURE, EXIT_ARGERROR
 
-from cli_options import get_sendpayment_parser, get_max_cj_fee_values, \
-     check_regtest
 import jmbitcoin as btc
 
 log = get_log()
@@ -53,7 +52,7 @@ def pick_order(orders, n): #pragma: no cover
 def main():
     parser = get_sendpayment_parser()
     (options, args) = parser.parse_args()
-    load_program_config()
+    load_program_config(config_path=options.datadir)
     if options.p2ep and len(args) != 3:
         parser.error("PayJoin requires exactly three arguments: "
                      "wallet, amount and destination address.")

scripts/sendtomany.py

@@ -13,7 +13,7 @@ from optparse import OptionParser
 import jmbitcoin as btc
 from jmbase import get_log, jmprint
 from jmclient import load_program_config, estimate_tx_fee, jm_single,\
-    get_p2pk_vbyte, validate_address, get_utxo_info,\
+    get_p2pk_vbyte, validate_address, get_utxo_info, add_base_options,\
     validate_utxo_data, quit
 
 
@@ -94,8 +94,9 @@ def main():
         help='input is p2pkh ("1" address), not segwit; if not used, input is assumed to be segwit type.',
         default=False
     )
+    add_base_options(parser)
     (options, args) = parser.parse_args()
-    load_program_config()
+    load_program_config(config_path=options.datadir)
     if len(args) < 2:
         quit(parser, 'Invalid syntax')
     u = args[0]

scripts/tumbler.py

@@ -12,12 +12,12 @@ from jmclient import Taker, load_program_config, get_schedule,\
     JMClientProtocolFactory, start_reactor, jm_single, get_wallet_path,\
     open_test_wallet_maybe, get_tumble_schedule,\
     schedule_to_text, estimate_tx_fee, restart_waiter, WalletService,\
-    get_tumble_log, tumbler_taker_finished_update,\
-    tumbler_filter_orders_callback, validate_address
+    get_tumble_log, tumbler_taker_finished_update, check_regtest, \
+    tumbler_filter_orders_callback, validate_address, get_tumbler_parser, \
+    get_max_cj_fee_values
+
 from jmbase.support import get_log, jmprint, EXIT_SUCCESS, \
     EXIT_FAILURE, EXIT_ARGERROR
-from cli_options import get_tumbler_parser, get_max_cj_fee_values, \
-     check_regtest
 
 log = get_log()
 logsdir = os.path.join(os.path.dirname(
@@ -32,7 +32,7 @@ def main():
     if len(args) < 1:
         jmprint('Error: Needs a wallet file', "error")
         sys.exit(EXIT_ARGERROR)
-    load_program_config()
+    load_program_config(config_path=options.datadir)
 
     check_regtest()
 

scripts/wallet-tool.py

@@ -3,11 +3,7 @@ from __future__ import (absolute_import, division,
 from builtins import * # noqa: F401
 
 from jmbase import jmprint
-from jmclient import load_program_config, wallet_tool_main
-from cli_options import check_regtest
+from jmclient import wallet_tool_main
 
 if __name__ == "__main__":
-    load_program_config()
-    check_regtest(blockchain_start=False)
-    #JMCS follows same convention as JM original; wallet is in "wallets" localdir
     jmprint(wallet_tool_main("wallets"), "success")

test/test_full_coinjoin.py

@@ -12,7 +12,7 @@ but certainly could be extended further.
 from common import make_wallets
 import pytest
 import sys
-from jmclient import YieldGeneratorBasic, load_program_config, jm_single,\
+from jmclient import YieldGeneratorBasic, load_test_config, jm_single,\
     sync_wallet, JMClientProtocolFactory, start_reactor, Taker, \
     random_under_max_order_choose
 from jmbase.support import get_log
@@ -142,6 +142,6 @@ def test_cj(setup_full_coinjoin, num_ygs, wallet_structures, mean_amt,
 
 @pytest.fixture(scope="module")
 def setup_full_coinjoin():
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface.tick_forward_chain_interval = 10
     jm_single().bc_interface.simulate_blocks()

test/test_segwit.py

@@ -11,7 +11,7 @@ from pprint import pformat
 import jmbitcoin as btc
 import pytest
 from jmbase import get_log
-from jmclient import load_program_config, jm_single, LegacyWallet
+from jmclient import load_test_config, jm_single, LegacyWallet
 log = get_log()
 
 
@@ -146,7 +146,7 @@ def test_spend_p2sh_p2wpkh_multi(setup_segwit, wallet_structure, in_amt, amount,
 
 @pytest.fixture(scope="module")
 def setup_segwit():
-    load_program_config()
+    load_test_config()
     jm_single().bc_interface.tick_forward_chain_interval = 1