With newer xcode (12+, I think), the default behaviour is to build
universal binaries that include native code for both arm64 and x86_64.
On older xcode, the default was to only target x86_64.
I am not sure why, but "hid.cpython-310-darwin.so" is not built
reproducibly when it is "universal".
(Even for consecutive builds done on the same machine.)
```
- + diff='Files /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so and /tmp/electrum_compare_dmg/dmg2/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so differ
```
This commit works around the reproducibly issue by making clang only
build for the x86_64 target. The binary should still be able to run
on arm64 macs using rosetta (same with previous versions of Electrum).
-----
The `lipo` tool can tell what archs a shared object is built for, e.g.:
```
$ lipo -info /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
Non-fat file: /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so is architecture: x86_64
```
For quick testing, we don't need to build the whole .app, only hid.so:
```
source /Users/vagrant/electrum/contrib/osx/build-venv/bin/activate
export CFLAGS="-g0"
python3 -m pip install -I --no-build-isolation --no-dependencies --no-binary :all: --no-warn-script-location hidapi==0.14.0
strip -x /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
cp /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so /Users/vagrant/wspace/tmp/try7/hid10.so
```
Re ARCHFLAGS env var, see https://stackoverflow.com/a/5808548
-----
Note: I've found several xcode build settings that looked relevant
but in the end were not useful, e.g. ARCHS, ONLY_ACTIVE_ARCH,
EXCLUDED_ARCHS, VALID_ARCHS.
- see https://developer.apple.com/documentation/technotes/tn3117-resolving-build-errors-for-apple-silicon
- see https://stackoverflow.com/a/64422757
```
602 WARNING: Failed to collect submodules for 'pkg_resources._vendor.pyparsing.diagram' because importing 'pkg_resources._vendor.pyparsing.diagram' raised: ModuleNotFoundError: No module named 'railroad'
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
5921 WARNING: collect_data_files - skipping data collection for module 'electrum.plugins' as it is not a package.
```
note: python 3.9.x is now in source-only mode, so could not update to latest...
it is time to investigate upgrading to python 3.10 in the win and mac binaries
note: the extra copies in make_osx.sh are needed because of the CI caching
(pyinstaller needs to see the .dylib's inside electrum/, e.g. when importing electrum during Analysis)
note: newer versions of pyinstaller (4.4+ ?) want to sign the bundle
themselves, in which case modifying the Info.plist file after
pyinstaller returns invalidates the sig.
This reverts commit 0c2a885c66.
Reverting for now due to reproducibility issues:
```
error: /Users/vagrant/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: can't open file: /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/libffi.8.dylib (No such file or directory)
```
`Electrum.app/Contents/MacOS/libffi.8.dylib` is included when building on one machine, but missing on the other...
Will have to investigate later.
related https://github.com/spesmilo/electrum/pull/7918
- the mac build broke with 184e122c36
- upstream fix is at https://github.com/pyinstaller/pyinstaller/pull/6701
however, we are using an old version of pyinstaller atm, so this workaround is easier
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing ./contrib/osx/.cache/pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
self.run_setup()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 75, in <module>
setup(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 122, in setup
dist.parse_config_files()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 167, in parse_configuration
meta.parse()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 417, in parse_section
self[name] = value
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 238, in __setitem__
value = parser(value)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/user/wspace/electrum/contrib/osx/.cache/pyinstaller/PyInstaller.py", line 16, in <module>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
This allows more freedom than using releases from PyPI.
(atm there is no released version that fixes https://github.com/pyinstaller/pyinstaller/pull/6701 )
Also, we now build the pyinstaller bootloader, just like in the windows build:
one fewer binary blob to trust.
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
When the monkey-patch was added, we were building pyinstaller onefile
binaries; but since then we changed to build pyinstaller onedir binaries
instead. So I believe there are no embedded files inside the main executable
anymore, so doing `codesign --deep *.app` (near the end of make_osx)
should be sufficient.
for sanity...
re touching these files: not sure they are really needed
(but the ~/Library/Python/ and ~/.pyenv folders are no longer there
on the build machine)
The .pyc files would get created when a .py module is imported,
which is done during the build for some source files by pyinstaller
(when it analyses imports).
I considered setting PYTHONDONTWRITEBYTECODE=1 in build_tools_util.sh,
to apply to all builds, but am not sure how it would affect the Android build,
where we actually want .pyc files included in the apk.
This commit ports the work of EchterAgo and cculianu from Electron-Cash,
to implement a new toolchain to scan qr codes.
Previously, on Linux and Win, we have been using zbar to access the camera
and read qrcodes; and on macOS we used CalinsQRReader (an objective-C
project by cculianu).
The new toolchain added here can use QtMultimedia to access the camera,
and then feed that image into zbar. When used this way, zbar needs
fewer dependencies and is easier to compile, in particular it can be
compiled for macOS.
The new toolchain works on all three platforms, with some caveats
(see code comments in related commits) -- so we also keep the end-to-end
zbar toolchain; but at least we can drop CalinsQRReader.
The related changes in Electron-Cash are spread over 50+ commits (several PRs and direct
pushes to master), but see in particular:
https://github.com/Electron-Cash/Electron-Cash/pull/1376
some other interesting links:
b2b737001c163224cf1f3b31e0fcb1eda015908ehttps://github.com/Electron-Cash/Electron-Cash/pull/1545052aa06c23
In some places in the script if some directory name had spaces it could cause issues.
I also added quotes around a few other directory names that currently are hardcoded with names without spaces as a way to prevent any future mishap
SomberNight
ThomasV
bitromortac
Alex Recuenco