A modern Python application packaging and distribution tool

Gregory Szorc

Last update: Jan 7, 2023

Related tags

Distribution PyOxidizer

Overview

PyOxidizer

PyOxidizer is a utility for producing binaries that embed Python. The over-arching goal of PyOxidizer is to make complex packaging and distribution problems simple so application maintainers can focus on building applications instead of toiling with build systems and packaging tools.

PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable. You can copy a single executable file to another machine and run a Python application contained within. It just works.

PyOxidizer exposes its lower level functionality for embedding self-contained Python interpreters as a tool and software library. So if you don't want to ship executables that only consist of a Python application, you can still use PyOxidizer to e.g. produce a library containing Python suitable for linking in any application or use PyOxidizer's embedding library directly for embedding Python in a larger application.

The Oxidizer part of the name comes from Rust: executables produced by PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations. If you don't know Rust, that's OK: PyOxidizer tries to make the existence of Rust nearly invisible to end-users.

While solving packaging and distribution problems is the primary goal of PyOxidizer, a side-effect of solving that problem with Rust is that PyOxidizer can serve as a bridge between these two languages. PyOxidizer can be used to easily add a Python interpreter to any Rust project. But the opposite is also true: PyOxidizer can also be used to add Rust to Python. Using PyOxidizer, you could bootstrap a new Rust project which contains an embedded version of Python and your application. Initially, your project is a few lines of Rust that instantiates a Python interpreter and runs Python code. Over time, functionality could be (re)written in Rust and your previously Python-only project could leverage Rust and its diverse ecosystem. Since PyOxidizer abstracts the Python interpreter away, this could all be invisible to end-users: you could rewrite an application from Python to Rust and people may not even know because they never see a libpython, .py files, etc.

Project Info

🏠 The official home of the PyOxidizer project is https://github.com/indygreg/PyOxidizer.

📔 Documentation (generated from the docs/ directory) is available at https://pyoxidizer.readthedocs.io/en/latest/index.html.

💬 The pyoxidizer-users mailing list is a forum for users to discuss all things PyOxidizer.

💰 If you want to financially contribute to PyOxidizer, do so on Patreon.

Comments

Build failure with 0.11.0 release on MacOS

It seems that latest release broke building targets on MacOS.

Context:

pyoxidizer 0.11.0
rust 1.50.0
macOS 10.14.6

The build target succeed with release 0.10.3.

Output (truncated):

pyoxidizer build --release --target-triple=x86_64-apple-darwin
resolving 1 targets
resolving target install
resolving target exe
resolving target dist
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20210303/cpython-3.8.8-x86_64-apple-darwin-pgo-20210303T0937.tar.zst", sha256: "b87aba6f6c6abed1365cee5b73e120fc1beca35414b535ffe944136943bdc8a3" }
Python distribution available at /Users/giovanni/Development/ProntoPro/cd-tool/./build/python_distributions/cpython-3.8.8-x86_64-apple-darwin-pgo-20210303T0937.tar.zst
reading data from Python distribution...

[...]

building with Rust 1.50.0
    Updating crates.io index
   Compiling memchr v2.3.4
   Compiling once_cell v1.7.2
   Compiling regex-syntax v0.6.22
   Compiling cc v1.0.67
   Compiling libc v0.2.87
   Compiling fs_extra v1.2.0
   Compiling byteorder v1.4.2
   Compiling anyhow v1.0.38
   Compiling encoding_rs v0.8.28
   Compiling proc-macro-hack v0.5.19
   Compiling cfg-if v1.0.0
   Compiling autocfg v1.0.1
   Compiling lazy_static v1.4.0
   Compiling smallvec v1.6.1
   Compiling either v1.6.1
   Compiling same-file v1.0.6
   Compiling quoted_printable v0.4.2
   Compiling base64 v0.12.3
   Compiling pyembed v0.11.0
   Compiling cpython v0.5.2
   Compiling tugger-file-manifest v0.1.0
   Compiling dunce v1.0.1
   Compiling thread_local v1.1.3
   Compiling base64 v0.10.1
   Compiling walkdir v2.3.1
   Compiling itertools v0.10.0
   Compiling embed-resource v1.6.1
   Compiling num-traits v0.2.14
   Compiling aho-corasick v0.7.15
   Compiling python-packed-resources v0.5.0
   Compiling jemalloc-sys v0.3.2
   Compiling alfred v0.1.0 (/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/alfred)
   Compiling memmap v0.7.0
   Compiling paste-impl v0.1.18
   Compiling charset v0.1.2
   Compiling mailparse v0.13.2
   Compiling regex v1.4.3
   Compiling paste v0.1.18
   Compiling spdx v0.3.6
   Compiling tugger-licensing v0.1.0
   Compiling python-packaging v0.4.0
   Compiling python3-sys v0.5.2
   Compiling jemallocator v0.3.2
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.0.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.1.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.10.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.11.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.2.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.3.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.4.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.5.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.6.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.7.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.8.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.9.rcgu.o" "-o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.4vn6ezcmupjx8p50.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/release/deps" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/build/jemalloc-sys-cc97a8e57aa21db6/out/build/lib" "-L" "/Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizer-build-exe-packaging1w160a" "-L" "/Users/giovanni/Development/ProntoPro/cd-tool/./build/python_distributions/python.b87aba6f6c6a/python/build/lib" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/artifacts" "-L" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libjemallocator-f37943904cb00a9d.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpyembed-b9a6fb91cb9a3657.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmemmap-c8c285284c6a0932.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libjemalloc_sys-d9a3757db5f656db.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libdunce-c5e9b78ee82be9f9.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcpython-e765973583055114.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpaste-d38ce5b5784be175.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libnum_traits-cf66d0025fde9c0d.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython_packaging-1f0435f87ba8cf06.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libitertools-a57eec996a8cc945.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libeither-7639319a24c1ce6b.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libwalkdir-3274d4d3dec30549.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libsame_file-312b7c3f4db734a3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython_packed_resources-77130fe057d6663e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmailparse-d81cbb87b0580c26.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libquoted_printable-76fd619fe8e7b795.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcharset-7b45e18ee5f18914.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libencoding_rs-3eb6f7d3e915f5c3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcfg_if-2c09d33489315c6b.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbase64-b1a9e17536c5510e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbase64-fdf9c4e974d9166a.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libtugger_licensing-402ca77557399c61.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libspdx-19c36274f2b04fa3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libregex-3dea78d0190721fd.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libthread_local-f0ed3e184ee41f51.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libonce_cell-2677ce9b9e668579.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libregex_syntax-8cbaaf8fb94140ae.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libaho_corasick-57a5daf663859cea.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmemchr-512039b4256b8b78.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/liblazy_static-be02bd543b1a672a.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libsmallvec-bf2bbad9b17c20a6.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libtugger_file_manifest-e421c134bef57653.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbyteorder-79457a5cbc751dc5.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libanyhow-19c0706a53df082e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython3_sys-c0c3d622228c777c.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/liblibc-6a12c3f69f3b87c2.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libstd-1356eb4a7f983e16.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-8ae047be503d7619.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libobject-ab8fafc452749532.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-f75e34ad235373a3.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libgimli-ed4b8c38c97ddcd6.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-9ca625a2899d3719.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-64c7f86b25c47576.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-4d1eb83250258253.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libunwind-46e0e02619331401.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-71dbab27ef745144.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/liblibc-8425a559f49d7df9.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/liballoc-3b6462f38288b0eb.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-820a42893aa0c85f.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcore-cfb9ec37e32acc9c.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-36c5611c149fc184.rlib" "-framework" "AppKit" "-framework" "ApplicationServices" "-framework" "Carbon" "-framework" "CoreFoundation" "-framework" "CoreGraphics" "-framework" "CoreServices" "-framework" "IOKit" "-framework" "SystemConfiguration" "-ldl" "-lncurses" "-lclang_rt.osx" "-lpthread" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-rdynamic"
  = note: ld: warning: directory not found for option '-L/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizer-build-exe-packaging1w160a'
          Undefined symbols for architecture x86_64:
            "___darwin_check_fd_set_overflow", referenced from:
                _seq2set in libpyembed-b9a6fb91cb9a3657.rlib(selectmodule.o)
                _set2list in libpyembed-b9a6fb91cb9a3657.rlib(selectmodule.o)
                _QueueFileEvents in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                _Tcl_CreateFileHandler in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                _Tcl_DeleteFileHandler in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                _TclUnixWaitForFile in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                _NotifierThreadProc in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                ...
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

I'm not able to test this on MacOS 11, it might be working. Also, same target on linux works.

opened by gi0baro 34

Addition of Mimalloc & Snmalloc as an allocators

I added Mimalloc as an optional feature on Windows systems since jemalloc is not really an option. Since Mimalloc was designed as a drop-in for malloc I decided the easiest route would be to make it the rust global allocator when the feature was enabled.

opened by ryancinsight 29
Switch to PyO3

I'm opening this PR so we have a better place to look at the code diff.

Current state is I'm able to build a working binary with PyO3 on Linux. macOS probably works as well. Windows will need more work (and likely some changes to PyO3's build system).

There's run-time crash with release builds on Linux. I think we were tickling an existing bug, however, as the crash is due to memory safety related to argv handling during Python interpreter initialization. It is a heisenbug and goes away if I sprinkle print statements around the suspect code (which I was doing because the stack trace was nearly worthless since a lot of functions were inlined). I may have to break out rr to debug it.

Anyway, the build system integration is the piece of this PR that still needs the most work. The Rust code speaking to PyO3 in the pyembed crate should be pretty solid and ready for review. All tests pass against the PyO3 branch I'm running. At present time, that branch only has commits for which there are open PRs in the PyO3 project.

CC @davidhewitt if you feel like looking at how I'm using PyO3. The main think I know I'm not doing right is I haven't implemented a GC visitor for types that contain references to other Python types. But that was a preexisting bug I think.

opened by indygreg 17
Tkinter module missing in build
Very happy to see a tool like this coming along. I'm having an issue with a small GUI application. It compiles fine. But it neither run with pyoxidizer run or as built binary.

Traceback (most recent call last): File "importlib.util", line 94, in find_spec File "gui", line 2, in <module> File "tkinter", line 36, in <module> ModuleNotFoundError: No module named '_tkinter' Segmentation fault: 11

This is on a Mac so I'm building x86_64-apple-darwin. I know my local python 2 seems to lack tkinter, my local python 3 does not. But from what I understand you pull other python builds. I'm guessing they don't have tkinter for some reason. Maybe not the most used extension, but quite useful to generate simple GUI.

Let me know if I can do anything to assist with figuring out the issue.
enhancement python-build-standalone
opened by lawik 16
Fix #337: Add __path__ filtering to OxidizedFinder's pkgutil.iter_modules support
This PR is a rewrite of and supersedes #340, and it fixes #337.

Summary

To make pkgutil.iter_modules(pkg.__path__) list only modules in pkg, I've added the OxidizedFinder.path_hook method, and, based on existing settings in OxidizedPythonInterpreterConfig, the OxidizedFinder instance in sys.meta_path has its bound path_hook method added to sys.path_hooks. When pkgutil.iter_modules passes pkg.__path__ to path_hook, path_hook returns an importlib.abc.PathEntryFinder object whose iter_modules method searches the OxidizedFinder instance for embedded resources whose names identify them as being in pkg.

Review

I think this is ready to go as long as you agree with me that none of the known bugs are blockers. All tests pass on my Mac except tests that were failing before in main at 2b4e66406f4fafdd7e09bc469878b239dcd55880. I'll run tests on Windows this weekend or early next week.

Known bugs

Multiple OxidizedFinder instances' path_hooks being in sys.path_hooks would interfere with each other. The first one in sys.path_hooks wins control over all paths under sys.executable.

The PathEntryFinder that OxidizedFinder.path_hook returns does not (yet) implement legacy methods find_loader and find_module, which were deprecated in Python 3.4. They wouldn't be hard to write, but I'm having trouble convincing myself there's any point. I have confirmed that nothing in the standard library relies on those methods. If we really want those methods, I think they can be added in future PRs, and we can crib Python's implementations.

OxidizedFinder.path_hook panics when a path contains an unpaired surrogate after the sys.executable part of the path because of dgrunwald/rust-cpython#246. I added a unittest.expectedFailure test case for a UnicodeDecodeError being raised instead.

My algorithm for separating the sys.executable part of the path from the pkg/mod part of the path has some trash strewn about it. The worst bit is where I make a couple of allocations (.collect<PathBuf>) where really none are needed. Open to suggestions about how to tighten that up.

UPDATE (12/22/20): Backwards incompatible change to achieve compatibility with CPython standard library

After some further testing I noticed that pkgutil.iter_modules() (i.e., with no path argument) was behaving differently when an OxidizedFinder was present on sys.meta_path. The standard library (vanilla CPython) yields only top-level modules (e.g., importlib but not importlib.abc). Moreover, the standard library only yields the last component of modules' __name__s:

>>> import importlib, pkgutil >>> [name for finder, name, ispkg in pkgutil.iter_modules(importlib.__path__)] ['_bootstrap', '_bootstrap_external', 'abc', 'machinery', 'metadata', 'resources', 'util']

02b9c2504960b3e53ea52eb2cf4224958f818d92 modifies OxidizedFinder.iter_modules and OxidizedFinder().path_hook(path).iter_modules to conform to the standard library's behavior. The documentation says almost nothing about a finder's iter_modules method, so the standard library is our best source of expected behavior.
opened by wkschwartz 14
apple-codesign: Can't disable main-binary executable segment for multiple binaries
Hi, thank you so much for your work on Notary support, this is super interesting. I tried it out recently from master.

I have a xar pkg that fails notarization. The rcodesign diff-signatures option was super helpful in researching my problem. I suspect my problem is caused because my xar pkg has 6 main binaries detected out of a total 14 Mach-O files. The notarization complaint only lists The signature of the binary is invalid. for the 6 binaries that were all detected as main binary.

I saw it's possible to control main-binary detection via the --executable-segment-flags option to rcodesign.

How can you specify multiple --executable-segment-flags ? I tried with multiple ordered arguments but the clap library parsing is complicated. Should it have .multiple_occurrences(true) in main.rs ?

How can you specify no flags at all for a SettingsScope? The FromStr for ExecutableSegmentFlags function doesn't allow empty-string or something to clear the flag. I could try with jit as a benign alternative perhaps.

bug apple-codesign
opened by mappu 13

`cargo fetch` can't pull down deps for v0.8.0

I'm pretty sure I'm missing something obvious, but this works at main and not at v0.8.0 (but also worked at 9b2631ce84817d177b8b0cbf16aeb2a7dab34e3f). We use cargo fetch to slurp down deps so we can satisfy requirements around non-use of the network during builds. Output:

augie% git checkout v0.8.0 > /dev/null
HEAD is now at 270a69e0 docs: document 0.8.0 release and link to blog post
augie% cargo fetch
    Updating crates.io index
error: failed to select a version for `pyembed`.
    ... required by package `oxidized-importer v0.8.0 (/Users/augie/Programming/pyoxidizer/oxidized-importer)`
versions that meet the requirements `^0.8.0` are: 0.8.0

the package `pyembed` links to the native library `pythonXY`, but it conflicts with a previous package which links to `pythonXY` as well:
package `pyembed v0.8.0 (/Users/augie/Programming/pyoxidizer/pyembed)`

failed to select a version for `pyembed` which could resolve this conflict

I'm confused because the pythonXY string doesn't show up many places and there don't seem to be any materially interesting changes around that between the tag and main. I'm happy to work on this, but I need some pointers to the right docs (either in pyoxidizer's code or in some Cargo docs) to get un-stuck. Thanks!

bug

opened by durin42 12

Cannot build app

Build: geadda2d

...
PyOxidizer does not set __file__ and this may create problems at run-time
See https://github.com/indygreg/PyOxidizer/issues/69 for more
package README.txt does not exist; excluding resources: ["README.txt"]
package bin does not exist; excluding resources: ["chardetect.exe"]
deriving custom importlib modules to support in-memory importing
building with Rust 1.42.0-nightly
    Updating crates.io index
error: no matching package named `pyembed` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
...

bug

opened by BiatuAutMiahn 12

multiphase initialisation suport of pyoxidizer

Hi,

This is a reduced example to the issue https://github.com/indygreg/PyOxidizer/issues/489

Thanks to @rgommers to point me to the right direction

I made a minimal example showing how cython modules can break pyoxidizer single file executables on windows.

https://github.com/franzhaas/pyoxidizer_cython_example

when importing fib, the repl crashes.

versions used are documented in pipenv.lock.

I am new to pyoxidizer, and would apreciate if I could get some guidnace how to tackle this. Is this a regression? Has that been working with older versions?

I do have access to working pyd files, which have been build out of hand crafted c code and will have a look at the difference between this and the cython c code, but this is likely not to work out....

Thanks for the grate tool!
bug

opened by franzhaas 9

Issue with Python distribution on Windows 10 with default init pyapp.

Firstly, thanks for this great tool - it's exactly what I've wanted to join the worlds of Rust and Python and I can't wait to get it going!

pyoxidizer itself builds fine, both via crate and the github master build, but I'm having an issue with the stand-alone python aspect on Windows 10 it sems. Here's the logs trying to build the default in init pyapp example.

...pyapp>pyoxidizer build
no existing PyOxidizer artifacts found
processing config file ...pyapp\pyoxidizer.toml
resolving Python distribution...
downloading https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
Python distribution available at ...pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
reading data from Python distribution...
thread 'main' panicked at 'unable to extract tar archive: Custom { kind: NotFound, error: TarError { desc: "failed to unpack `...pyapp\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\python.fd43554b5654\\python\\install\\Lib\\site-packages\\pip-19.1.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport\\__pycache__\\low_level.cpython-37.pyc`", io: Custom { kind: NotFound, error: TarError { desc: "failed to unpack `python/install/Lib/site-packages/pip-19.1.1-py3.7.egg/pip/_vendor/urllib3/contrib/_securetransport/_` into `...pyapp\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\python.fd43554b5654\\python\\install\\Lib\\site-packages\\pip-19.1.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport\\__pycache__\\low_level.cpython-37.pyc`", io: Os { code: 3, kind: NotFound, message: "The system cannot find the path specified." } } } } }', src\libcore\result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Backtrace enabled Version:

pyoxidizer build
no existing PyOxidizer artifacts found
processing config file ...pyapp\pyoxidizer.toml
resolving Python distribution...
Python distribution available at ...pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
reading data from Python distribution...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "unable to read Python license"', src\libcore\result.rs:997:5
stack backtrace:
   0: git_packbuilder_object_count
   1: git_packbuilder_object_count
   2: git_packbuilder_object_count
   3: git_packbuilder_object_count
   4: git_packbuilder_object_count
   5: git_packbuilder_object_count
   6: git_packbuilder_object_count
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: git_packbuilder_object_count
  16: git_packbuilder_object_count
  17: git_packbuilder_object_count
  18: <unknown>
  19: git_libgit2_version
  20: BaseThreadInitThunk
  21: RtlUserThreadStart

Any tips or pointers on this would be appreciated. Thanks!

opened by McSpidey 9

Panic signing a particular executable file

It seems like this particular binary causes apple-codesign to break for some reason:

signing cross-x86_64-centos-7-0.9.0-macos-aarch64/libexec/gcc/x86_64-linux-gnu/11.2.0/cc1plus in place
signing cross-x86_64-centos-7-0.9.0-macos-aarch64/libexec/gcc/x86_64-linux-gnu/11.2.0/cc1plus as a Mach-O binary
inferring default signing settings from Mach-O binary
preserving existing binary identifier in Mach-O
preserving code signature flags in existing Mach-O signature
setting binary identifier to cc1plus
parsing Mach-O
signing Mach-O binary at index 0
thread 'main' panicked at 'assertion failed: segment.fileoff == 0 || segment.fileoff == cursor.position()', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/apple-codesign-0.16.0/src/macho_signing.rs:166:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Here's the offending binary in a tarball: cc1plus.tar.gz

bug apple-codesign

opened by sfackler 8

Some questions about PyOxidizer
Regarding PyOxidizer, I have some questions to ask.

Does PyOxidizer support converting python code files into stand-alone executable binary files?

Does the stand-alone executable binary file follow the ELF and other executable file formats?

What is the principle of the pyoxidizer build process? Could you please introduce me?

Grateful!
opened by cyw3 0

thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)'

I'm running PyOxidizer within a ubi8-minimal container (based on RHEL8). I'm attempting to package https://github.com/fabioz/mu-repo/ as a standalone binary. I've installed PyOxidizer through pip3 in case that's helpful. And here's the full stacktrace of the error that I'm hitting

  thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.5.2+5.3.0-patched/build.rs:326:19
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
     1: core::panicking::panic_fmt
               at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
     2: build_script_build::execute
               at ./build.rs:326:19
     3: build_script_build::run
               at ./build.rs:319:5
     4: build_script_build::main
               at ./build.rs:257:5
     5: core::ops::function::FnOnce::call_once
               at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
warning: build failed, waiting for other jobs to finish...
error[PYOXIDIZER_PYTHON_EXECUTABLE]: adding PythonExecutable to FileManifest

    Caused by:
        0: building Python executable
        1: building executable with Rust project
        2: cargo build failed
      --> ./pyoxidizer.bzl:38:5
       |
    38 |     files.add_python_resource(".", exe)
       |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.to_file_manifest()


error: adding PythonExecutable to FileManifest

Caused by:
    0: building Python executable
    1: building executable with Rust project
    2: cargo build failed

This is the pyoxidizer.bzl file that I'm relying on: https://gist.github.com/wgordon17/f37e0500ba9998d0369d1c91ee1e3083

I don't have this issue when building on an ubuntu image, only the RHEL8-based image, and I can't figure out what the root cause is!

opened by wgordon17 0

build(deps): bump certifi from 2022.5.18.1 to 2022.12.7 in /docs
Bumps certifi from 2022.5.18.1 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies python
opened by dependabot[bot] 0
`pyoxidizer run` extra args are ignored

The PyOxidizer Getting Started docs suggest using pyoxidizer run -- [args]:

https://github.com/indygreg/PyOxidizer/blob/0defb06fd21c81dbde43ff8ffddd92ff133ec4c0/pyoxidizer/docs/pyoxidizer_getting_started.rst?plain=1#L330-L350

But it does not actually work. Looking at the code, pyoxidizer run does accept extra args, which get passed to projectmgmt::run as _extra_args:

https://github.com/indygreg/PyOxidizer/blob/0defb06fd21c81dbde43ff8ffddd92ff133ec4c0/pyoxidizer/src/projectmgmt.rs#L142-L171

However, this function does not use them so they just get lost. I looks like the regression was introduced in https://github.com/indygreg/PyOxidizer/commit/df0d459e044419369854918a03baa496d565608c released in 0.5.1. I tried to see quickly if I could fix the issue, but looks like starlark::eval::EvaluationContextBuilder does not support extra args at all? I don't know Rust or PyOxidizer enough to tell.

opened by pquentin 0

`importlib.import_module` raising ModuleNotFoundError with custom sys.path

Hello! I'm porting Rally (the official Elasticsearch benchmark tool) to PyOxidizer, and despite being stuck on 0.22.0 it works really well and required almost no change to our code. However Rally has this feature where it will load custom Python modules from another directory, for example https://github.com/elastic/rally-tracks/blob/master/eql/track.py.

Suppose /home/q/src/pyapp/other/path/amodule.py exists. The following CPython code does not raise an exception and loads the module:

import importlib
import sys

sys.path.insert(0, "/home/q/src/pyapp/other/path")
importlib.import_module("amodule")

But translated to PyOxidizer, I get an exception. Here's the pyoxidizer.bzl file that attempts to load the same module:

def make_exe():
    dist = default_python_distribution()
    policy = dist.make_python_packaging_policy()
    policy.resources_location = "in-memory"

    python_config = dist.make_python_interpreter_config()
    # Evaluate a string as Python code when the interpreter starts.
    python_config.run_command = """
import importlib
import sys

sys.path.insert(0, "/home/q/src/pyapp/other/path")
importlib.import_module("amodule")
"""

    return dist.to_python_executable(
        name="pyapp",
        packaging_policy=policy,
        config=python_config
    )


def make_install(exe):
    files = FileManifest()
    files.add_python_resource(".", exe)
    return files


register_target("exe", make_exe)
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()

But pyoxidizer run fails with:

installing files to /home/q/src/pyapp/./build/x86_64-unknown-linux-gnu/debug/install
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "importlib", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'amodule'
error: cargo run failed

I would appreciate any ideas to help me debug this, or even turns this into a reproducer using only oxidizer_importer.

opened by pquentin 0

mach not yet supported (pyoxidizer analyze)
I have a binary built by pyoxidizer (Mercurial). It's failing to invoke readline.get_completer and I'd like to inspect why. I thought pyoxidizer analyze might tell me more about what files are in the executable, but when I run it on the binary, I get this error:

$ pyoxidizer analyze /usr/local/bin/hg thread 'main' panicked at 'mach not yet supported', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tugger-binary-analysis-0.6.0/src/audit.rs:34:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

PyOxidizer is installed using pipx install pyoxidizer on macOS 13.0.1 on Apple Silicon.

$ pyoxidizer -V PyOxidizer 0.23.0
opened by jaraco 0