In `build-electrum-git.sh`, the `-w` CLI arg is removed: it was apparently ignored as we are using a .spec file,
and pyinstaller 5.0+ is now raising a hard error (see https://github.com/pyinstaller/pyinstaller/issues/6660).
```
option(s) not allowed:
--console/--nowindowed/--windowed/--noconsole
makespec options not valid when a .spec file is given
```
-----
In ecc_fast.py, we don't sys.exit() anymore as pyinstaller 5.0+ tries to import electrum during the Analysis phase.
see https://github.com/pyinstaller/pyinstaller/pull/6171
```
57912 INFO: Looking for dynamic libraries
1746 INFO: gettext setting initial language to None
1932 ERROR: libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its depende
ncies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll' (or one of its dependenc
ies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencie
s). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor
syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find
module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
Traceback (most recent call last):
File "C:\python3\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\python3\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\python3\scripts\pyinstaller.exe\__main__.py", line 7, in <module>
File "C:\python3\lib\site-packages\PyInstaller\__main__.py", line 194, in _console_script_run
run()
File "C:\python3\lib\site-packages\PyInstaller\__main__.py", line 180, in run
run_build(pyi_config, spec_file, **vars(args))
File "C:\python3\lib\site-packages\PyInstaller\__main__.py", line 61, in run_build
PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
File "C:\python3\lib\site-packages\PyInstaller\building\build_main.py", line 1006, in main
build(specfile, distpath, workpath, clean_build)
File "C:\python3\lib\site-packages\PyInstaller\building\build_main.py", line 928, in build
exec(code, spec_namespace)
File "deterministic.spec", line 55, in <module>
a = Analysis([home+'run_electrum',
File "C:\python3\lib\site-packages\PyInstaller\building\build_main.py", line 428, in __init__
self.__postinit__()
File "C:\python3\lib\site-packages\PyInstaller\building\datastruct.py", line 184, in __postinit__
self.assemble()
File "C:\python3\lib\site-packages\PyInstaller\building\build_main.py", line 736, in assemble
isolated.call(find_binary_dependencies, self.binaries, self.binding_redirects, collected_packages)
File "C:\python3\lib\site-packages\PyInstaller\isolated\_parent.py", line 372, in call
return isolated.call(function, *args, **kwargs)
File "C:\python3\lib\site-packages\PyInstaller\isolated\_parent.py", line 302, in call
raise RuntimeError(f"Child process call to {function.__name__}() failed with:\n" + output)
RuntimeError: Child process call to find_binary_dependencies() failed with:
File "C:\python3\lib\site-packages\PyInstaller\isolated\_child.py", line 63, in run_next_command
output = function(*args, **kwargs)
File "C:\python3\lib\site-packages\PyInstaller\building\build_main.py", line 177, in find_binary_dependencies
__import__(package)
File "C:\python3\lib\site-packages\electrum\__init__.py", line 20, in <module>
from .wallet import Wallet
File "C:\python3\lib\site-packages\electrum\wallet.py", line 53, in <module>
from .bip32 import BIP32Node, convert_bip32_intpath_to_strpath, convert_bip32_strpath_to_intpath
File "C:\python3\lib\site-packages\electrum\bip32.py", line 11, in <module>
from . import constants
File "C:\python3\lib\site-packages\electrum\constants.py", line 30, in <module>
from . import bitcoin
File "C:\python3\lib\site-packages\electrum\bitcoin.py", line 35, in <module>
from . import ecc
File "C:\python3\lib\site-packages\electrum\ecc.py", line 39, in <module>
from .ecc_fast import _libsecp256k1, SECP256K1_EC_UNCOMPRESSED
File "C:\python3\lib\site-packages\electrum\ecc_fast.py", line 151, in <module>
sys.exit(f"Error: Failed to load libsecp256k1.")
SystemExit: Error: Failed to load libsecp256k1.
🗯 ERROR: build-electrum-git failed
```
Also, the -OO flag is removed from wine python, for similar reasons:
pyinstaller imports electrum, and in electrum/__init__.py, we raise
if -O is used: 9b1fb0e5fe/electrum/__init__.py (L40)
- repro builds to use fixed uid=1000 inside the container
- in case the file permissions leak into the binaries, they are still reproducible
- chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
- otherwise the local dev build would still interact with the fresh_clone dir
- due to e.g. recursive "find -exec touch",
- and even the "docker build" cmd itself would try to stat/read it
- see https://github.com/docker/for-linux/issues/380
- and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
- this way the local dev build does not need sudo
to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
- does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
- if the host userid is 1000, does not need sudo
- otherwise, needs sudo
closes https://github.com/spesmilo/electrum/issues/8261
reproducibility probably needs a hardcoded userid
Also, move the UID arg later in the dockerfiles, for better caching.
(if local dev build and repro build set different UIDs, the build caches
will diverge at that step)
The .dockerignore symlink in the project root dir is only picked up by the android build.
The android build has the project root as its build context for "docker build" --
the other builds have their own subdirectories as build context, e.g. contrib/build-linux/appimage.
- follow-up prev: CONTRIB was not defined
- rm folder signed/stripped if already exists (otherwise script early-exited silently)
- quote paths to guard against whitespace shenanigans
note: python 3.9.x is now in source-only mode, so could not update to latest...
it is time to investigate upgrading to python 3.10 in the win and mac binaries
see https://snapshot.debian.org/
> To access snapshots of suites using Valid-Until that are older than a dozen days, it is necessary to ignore the Valid-Until header within Release files, in order to prevent apt from disregarding snapshot entries ("Release file expired"). Use aptitude -o Acquire::Check-Valid-Until=false update or apt-get -o Acquire::Check-Valid-Until=false update for this purpose.
(thanks to Axel Gembe for pointing this out)
Instead of using pre-built binary wheels from PyPI.
We should tighten this more (re other requirements-*.txt files),
but there is no C compiler available inside the wine environment atm...
note: unclear where to put these files... `contrib/build-wine/` and `contrib/osx/`
contain binary-building-related files. maybe we could have a `doc/` folder
if the need for more similar files arise.
atm there are these two, plus maybe `contrib/docker_notes.md`.
- the windows build broke with 184e122c36
- upstream fix being pulled in is https://github.com/pyinstaller/pyinstaller/pull/6701
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing c:\electrum\contrib\build-wine\.cache\win32\pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <
module>
main()
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in m
ain
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 164, in p
repare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 188, in prepare_metadata_for_
build_wheel
self.run_setup()
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 249, in <module>
setup(
File "C:\python3\lib\site-packages\setuptools\__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "C:\python3\lib\site-packages\setuptools\_distutils\core.py", line 122, in setup
dist.parse_config_files()
File "C:\python3\lib\site-packages\setuptools\dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 167, in parse_configurat
ion
meta.parse()
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 417, in parse_section
self[name] = value
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 238, in __setitem__
value = parser(value)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\electrum\contrib\build-wine\.cache\win32\pyinstaller\PyInstaller.py", line 15, in <modu
le>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
🗯 ERROR: prepare-wine failed
```
see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1072669548
-----
pyinstaller 4.2 failed (during its runtime) to create exes (worked with cpython 3.9.10, but not with cpython 3.9.11):
```
80572 INFO: Processing module hooks...
80573 INFO: Loading module hook 'hook-certifi.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
80618 INFO: Loading module hook 'hook-cryptography.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
82879 INFO: Loading module hook 'hook-dns.rdata.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84147 INFO: Loading module hook 'hook-mnemonic.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84207 INFO: Loading module hook 'hook-pycparser.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84212 INFO: Loading module hook 'hook-usb1.py' from 'C:\\python3\\lib\\site-packages\\usb1\\__pyinstaller'...
84215 INFO: --- libusb1 pyinstaller hook ---
84226 INFO: Added libusb binaries: [('C:\\python3\\lib\\site-packages\\usb1\\libusb-1.0.dll', 'usb1')]
84228 INFO: Loading module hook 'hook-difflib.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
84237 INFO: Excluding import of doctest from module difflib
84237 INFO: Loading module hook 'hook-distutils.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
Unable to find "C:\python3\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
* apt-get install python3-dev
* apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)
🗯 ERROR: build-electrum-git failed
```
Looks like this might be fixed in pyinstaller 4.3 (we are using 4.2):
https://github.com/pyinstaller/pyinstaller/pull/5218
-----
While trying to update pyinstaller to have that fix, several issues found with versions 4.3-4.10,
now reported upstream and already fixed:
https://github.com/pyinstaller/pyinstaller/issues/6338https://github.com/pyinstaller/pyinstaller/issues/6339https://github.com/pyinstaller/pyinstaller/issues/6686
So the pyinstaller commit pinned here is from the stable "v4" branch, some commits after the 4.10 tag,
which has fixes for the above issues.
SomberNight
Sander van Grieken
ThomasV
gruve-p
Jamie C. Driver