Haven't checked if electrum-ecc compiles libsecp reproducibly.
For now let's just keep the old flow.
(but if we spent time on making that compilation reproducible,
the appimage and the macos builds could use it directly)
- For the notarisation, we were using "altool" (part of xcode), but Apple now fully deprecated that.
- need to migrate from altool to notarytool
https://developer.apple.com/news/?id=y5mjxqmn
- currently using macOS 10.14.6
and xcode 11.3.1
- notarytool requires xcode 13+
- xcode 13 requires macOS 11.3
https://en.wikipedia.org/wiki/Xcode#Version_comparison_table
==> created new build VM with macOS 11.7.10 and xcode 13.2
With newer xcode (12+, I think), the default behaviour is to build
universal binaries that include native code for both arm64 and x86_64.
On older xcode, the default was to only target x86_64.
I am not sure why, but "hid.cpython-310-darwin.so" is not built
reproducibly when it is "universal".
(Even for consecutive builds done on the same machine.)
```
- + diff='Files /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so and /tmp/electrum_compare_dmg/dmg2/Electrum.app/Contents/MacOS/hid.cpython-310-darwin.so differ
```
This commit works around the reproducibly issue by making clang only
build for the x86_64 target. The binary should still be able to run
on arm64 macs using rosetta (same with previous versions of Electrum).
-----
The `lipo` tool can tell what archs a shared object is built for, e.g.:
```
$ lipo -info /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
Non-fat file: /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so is architecture: x86_64
```
For quick testing, we don't need to build the whole .app, only hid.so:
```
source /Users/vagrant/electrum/contrib/osx/build-venv/bin/activate
export CFLAGS="-g0"
python3 -m pip install -I --no-build-isolation --no-dependencies --no-binary :all: --no-warn-script-location hidapi==0.14.0
strip -x /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so
cp /Users/vagrant/electrum/contrib/osx/build-venv/lib/python3.10/site-packages/hid.cpython-310-darwin.so /Users/vagrant/wspace/tmp/try7/hid10.so
```
Re ARCHFLAGS env var, see https://stackoverflow.com/a/5808548
-----
Note: I've found several xcode build settings that looked relevant
but in the end were not useful, e.g. ARCHS, ONLY_ACTIVE_ARCH,
EXCLUDED_ARCHS, VALID_ARCHS.
- see https://developer.apple.com/documentation/technotes/tn3117-resolving-build-errors-for-apple-silicon
- see https://stackoverflow.com/a/64422757
Ah ok, I give up for now... the prev does not really work.
The prior commit works on Windows but not on macOS.
On Windows, it would package all plugins as code and only as code.
On MacOS however, it would not package any plugins at all. And with this commit,
where I mark the plugins folder to be packaged as *data*, it packages all plugins as *both* code and data.
Not sure why.
Let's just package all plugins as both code and data, and ignore the code instances explicitly...
The hw_wallet and jade plugins were being included twice in the pyinstaller binaries.
They were apparently the only two plugins being picked as "modules" (a.pure),
which by default are included as compiled-bytecode.
In addition, we included "electrum/plugins" as data (a.data), which meant all
plugins in source form.
Instead of hacking around to fix the specific issue, this attempts a larger clean-up.
```
602 WARNING: Failed to collect submodules for 'pkg_resources._vendor.pyparsing.diagram' because importing 'pkg_resources._vendor.pyparsing.diagram' raised: ModuleNotFoundError: No module named 'railroad'
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
libsecp256k1 library failed to load. exceptions: [FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-2.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-1.dll'(or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'C:\\python3\\lib\\site-packages\\electrum\\libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-2.dll' (or one ofits dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-1.dll' (or one of its dependencies). Try using the full path with constructor syntax."), FileNotFoundError("Could not find module 'libsecp256k1-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.")]
5921 WARNING: collect_data_files - skipping data collection for module 'electrum.plugins' as it is not a package.
```
pyinstaller tries to import electrum and its different submodules at build-time
during the "Analysis" phase. sys._GUI_QT_VERSION was not getting set there,
and the resulting exception was blocking pyinstaller from discovering that
gui/common_qt is being used.
at runtime:
```
$ ./dist/Electrum.app/Contents/MacOS/run_electrum
1.53 | E | daemon.Daemon | GUI raised exception: Exception('Error loading trustedcoin plugin: ModuleNotFoundError("No module named \'electrum.gui.common_qt\'")'). shutting down.
1.53 | E | __main__ | daemon.run_gui errored
Traceback (most recent call last):
File "electrum/plugin.py", line 135, in load_plugin
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/vagrant/electrum/dist/Electrum.app/Contents/MacOS/electrum/plugins/trustedcoin/qt.py", line 51, in <module>
from .common_qt import TrustedcoinPluginQObject
File "/Users/vagrant/electrum/dist/Electrum.app/Contents/MacOS/electrum/plugins/trustedcoin/common_qt.py", line 16, in <module>
from electrum.gui.common_qt.plugins import PluginQObject
ModuleNotFoundError: No module named 'electrum.gui.common_qt'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run_electrum", line 456, in handle_cmd
d.run_gui()
File "electrum/daemon.py", line 617, in run_gui
self.gui_object = gui.ElectrumGui(config=self.config, daemon=self, plugins=self._plugins)
File "electrum/util.py", line 473, in do_profile
o = func(*args, **kw_args)
File "electrum/gui/qt/__init__.py", line 153, in __init__
self.plugins.load_plugin('trustedcoin')
File "electrum/plugin.py", line 138, in load_plugin
raise Exception(f"Error loading {name} plugin: {repr(e)}") from e
Exception: Error loading trustedcoin plugin: ModuleNotFoundError("No module named 'electrum.gui.common_qt'")
```
note: python 3.9.x is now in source-only mode, so could not update to latest...
it is time to investigate upgrading to python 3.10 in the win and mac binaries
note: the extra copies in make_osx.sh are needed because of the CI caching
(pyinstaller needs to see the .dylib's inside electrum/, e.g. when importing electrum during Analysis)
ThomasV had libffi installed via brew, and this side-effected the built electrum dmg.
`('libffi.8.dylib', '/usr/local/opt/libffi/lib/libffi.8.dylib', 'BINARY')`
this was the cause of 019d213325
based on 907e5e7009
This way users get more feedback from the OS when they attempt to launch the app on old macOS.
Co-authored-by: Calin Culianu <calin.culianu@gmail.com>
note: newer versions of pyinstaller (4.4+ ?) want to sign the bundle
themselves, in which case modifying the Info.plist file after
pyinstaller returns invalidates the sig.
This reverts commit 0c2a885c66.
Reverting for now due to reproducibility issues:
```
error: /Users/vagrant/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: can't open file: /tmp/electrum_compare_dmg/signed_app/Electrum.app/Contents/MacOS/libffi.8.dylib (No such file or directory)
```
`Electrum.app/Contents/MacOS/libffi.8.dylib` is included when building on one machine, but missing on the other...
Will have to investigate later.
related https://github.com/spesmilo/electrum/pull/7918
note: unclear where to put these files... `contrib/build-wine/` and `contrib/osx/`
contain binary-building-related files. maybe we could have a `doc/` folder
if the need for more similar files arise.
atm there are these two, plus maybe `contrib/docker_notes.md`.
SomberNight
ThomasV
Jamie C. Driver