I need to add tests and docs for this, but I just wanted to post this before the weekend since I've actually gotten it working.

The key changes are:

1. Use pip download + wheel install instead of pip install for installing wheels. This way you can force installation into a directory even with incompatible wheels, and you can rely on pip download to get the correct wheels for a different platform.
2. Stop enforcing that --python exists, since you might be creating a .pyz file for a different machine.
3. Fix a bug in map_shared_objects where some internal .so files used by numpy would break the module_name splitting code.

If you want to try it out, just run:

shiv -p /path/to/python36/on/other/machine/python3 -o foo.pyz --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy

I didn't add any special args for cross-platform stuff, since we can rely on the pip download flags.

I ran into an edge case (ModuleNotFoundError) where I had an older version of a dependency installed into dist-packages, that was being used instead of the shiv packaged version, because the shiv site-packages was at the end of sys.path:

>>> import sys; sys.path
['./ebbp', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/tekumara/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/home/tekumara/.shiv/ebbp_921c47de-78d9-4ce4-a4cd-28426ed667e2/site-packages']

This change makes shiv zipapps more reliable, by placing shiv site-packages at the start of sys.path, rather than the end, eg:

>>> import sys; sys.path
['./ebbp', '/home/tekumara/.shiv/ebbp_921c47de-78d9-4ce4-a4cd-28426ed667e2/site-packages', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/tekumara/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

Hello,

I played a bit with shiv and noticed that it always writes the full path to a python interpreter (sys.executable or user provided one) in the zipapp's shebang. Which means, if I try to run the same pyz on a different machine where python3.6 is installed in a different location, it won't run.

https://github.com/linkedin/shiv/blob/29fb7c316fa7b1f9111cade730b2168a2c4f4a6e/src/shiv/cli.py#L56

https://github.com/linkedin/shiv/blob/29fb7c316fa7b1f9111cade730b2168a2c4f4a6e/src/shiv/builder.py#L39-L43

https://github.com/linkedin/shiv/blob/29fb7c316fa7b1f9111cade730b2168a2c4f4a6e/src/shiv/pip.py#L46

With pex, one can specify the interpreter to use either as absolute path or just the name of a Python interpreter within the environment.

It would be nice if shiv also allowed to do that.

Thank you

I'd like to be able to have a shiv option that would instruct the bootstrap code to clean out old directories in .shiv for the current app. Maybe:

shiv --delete-other-builds

With that option, when the first bootstrap of a build finished, the boostrap code would essentially run:

rm -rf ~/.shiv/the_app_name_*

Excluding the folder that contains the files for the currently running app.

As written, the example leaves the shiv pyz file with a shebang pointing at the (presumably temporary) virtual environment used to build it.

The example should probably include a -p "/usr/bin/env python3" argument to specify an explicit interpreter.

Actually, given that shiv pyz files are self-contained, there's no real benefit to them ever having shebangs that refer to a virtualenv - maybe it would be worth making the default shebang /usr/bin/env python3, or at least locate the "base" Python interpreter for a virtualenv - which can be done as

Path(getattr(sys, 'real_prefix', sys.base_prefix)) / Path(sys.executable).relative_to(sys.prefix)

I have a particular python script that uses subprocess to create child processes that execute another script. What is the recommended way of passing down the info about the shiv's site directory down to the child?

As an example, suppose I made a shiv like this:

$ shiv requests -o foo.shiv

I then make a parent.py that creates a child process:

import os
import subprocess
import sys

print("Parent sys.executable:", sys.executable)
print("Parent sys.path:", sys.path)
print("Parent sys.argv", sys.argv)
print("Parent PYTHONPATH", os.environ.get("PYTHONPATH"))

child = subprocess.run([
    sys.executable,
    'child.py',
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)

print(child)

And then I make a child.py that makes use of requests, like so:

import requests

print(requests)

When I run it, I get output that looks like this:

$ ./foo.shiv parent.py
Parent sys.executable: /Users/kedo/.pyenv/versions/3.6.5/bin/python
Parent sys.path: ['./foo.shiv', '/Users/kedo/.pyenv/versions/3.6.5/lib/python36.zip', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/lib-dynload', '/Users/kedo/.local/lib/python3.6/site-packages', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages', '/Users/kedo/Workspace/shiv/src', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/importlib_resources-0.8-py3.6.egg', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/wheel-0.31.1-py3.6.egg', '/Users/kedo/.shiv/foo_0550dd79-b895-48c0-91fa-11306e18f9cd/site-packages']
Parent sys.argv ['parent.py']
Parent PYTHONPATH None
CompletedProcess(args=['/Users/kedo/.pyenv/versions/3.6.5/bin/python', 'child.py'], returncode=1, stdout=b'', stderr=b'Traceback (most recent call last):\n  File "child.py", line 1, in <module>\n    import requests\nModuleNotFoundError: No module named \'requests\'\n')

Is there a non-hacky way for parent to know that it's running via a shiv, and that it needs to pass down info about the shiv to the child? Could we set some sort of environment variable in https://github.com/linkedin/shiv/blob/master/src/shiv/bootstrap/init.py#L86 that parent.py could check and use?

In build systems that allow products to build multiple Python packages (apps) together in parallel, such as PyGradle, the solution from #16 breaks the build of a package that does not use shiv. It was implemented in #27 but we did not notice until now the negative effect it has.

Since that solution is implemented only for homebrew Python, I believe it's unacceptable that it breaks standard Python builds and packages. We need to make this usage conditional, with some setting, so that users who need it can apply it, but the standard usage should not be doing this by default.

I'm inclined to just plain revert #27 to get the fix out ASAP. Users who need this change can use the releases of shiv that have it, and later can use the future release of shiv that implements this conditionally.

Can we execute this plan promptly? Thanks!

Here's a minimalist appveyor.yml file, running tox on python3.6 and python3.7 (64-bit only, I can add 32 if we want, though it's 2018).

Here's are the test failures I see with my own appveyor account linked to my anthrotype/shiv fork: https://ci.appveyor.com/project/anthrotype/shiv/build/1.0.4

To make this work in the upstream repo, @sixninetynine should first merge this to master then grant Appveyor access to this public repo.

Since this is under the linkedin github organanization, the way Appveyor works for such repositories is slightly more complicated than Travis CI. Read more here: https://www.appveyor.com/docs/team-setup/#setting-up-appveyor-account-for-github-organization

This commit loosens shiv's requirements when constructing a shebang line: rather than enforcing that the target interpreter must exist, we simply accept what the user provides.

We no longer append -sE to the shebang as these arguments can be provided to the value given to the --python argument.

Shiv will now exit(1) if the resulting shebang would exceed the BIN_PRM restrictions present on most unix-y OSes.

Fixes #17

After merging I'll cut a new release (0.0.26)

With shiv 0.0.14 from PyPI:

$ shiv aws -c aws -p $(which python3.6) -o blergh
 shiv! 🔪
Collecting aws
Collecting fabric>=1.6 (from aws)
Collecting boto (from aws)
  Using cached https://files.pythonhosted.org/packages/bd/b7/a88a67002b1185ed9a8e8a6ef15266728c2361fcb4f1d02ea331e4c7741d/boto-2.48.0-py2.py3-none-any.whl
Collecting prettytable>=0.7 (from aws)
Collecting paramiko<3.0,>=1.10 (from fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/3e/db/cb7b6656e0e7387637ce850689084dc0b94b44df31cc52e5fc5c2c4fd2c1/paramiko-2.4.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/74/8e/a6c0d340972d9e2f1a405aaa3f2460950b4c0337f92db0291a4355974529/PyNaCl-1.2.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting bcrypt>=3.1.3 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/7e/59/d48fd712941da1a5d6490964a37bb3de2e526965b6766273f6a7049ee590/bcrypt-3.1.4-cp36-cp36m-macosx_10_6_intel.whl
Collecting cryptography>=1.5 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/40/87/acdcf84ce6d25a7db1c113f4b9b614fd8d707b7ab56fbf17cf18cd26a627/cryptography-2.2.2-cp34-abi3-macosx_10_6_intel.whl
Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/8e/be/40b1bc2c3221acdefeb9dab6773d43cda7543ed0d8c8df8768f05af2d01e/cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl
Collecting six (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
Installing collected packages: pyasn1, pycparser, cffi, six, pynacl, bcrypt, idna, asn1crypto, cryptography, paramiko, fabric, boto, prettytable, aws
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 335, in run
    use_user_site=options.use_user_site,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 49, in install_given_reqs
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 748, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 961, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/wheel.py", line 216, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/locations.py", line 165, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

Pip install failed!

Here's the packages installed systemwide alongside shiv:

click==6.7
importlib-resources==0.5
pip==10.0.1
setuptools==39.1.0
shiv==0.0.14
wheel==0.31.0

OS X, Python 3.6.5 from Homebrew.

It is useful to be able to launch subprocesses with the exact same PYTHONPATH as the parent process, but it doesn't seem possible with shiv at the moment.

For example, if I have a virtualenv, and I either activate it or set PATH to use it, Python subprocesses will import from the virtualenv.

Shiv's --extend-pythonpath is close to that behavior, but not sufficient. Specifically, when the option is enabled, PYTHONPATH for subprocesses is extended (https://github.com/linkedin/shiv/blob/526ebbe2ee472e8af6604c32535b5d954fa2e7ca/src/shiv/bootstrap/init.py#L167-L169). That is, if there's already Python package X in the pre-existing PYTHONPATH, it will be used instead of X from the shiv.

The parent process, in contrast, adds the shiv site-packages to the start of sys.path: https://github.com/linkedin/shiv/blob/526ebbe2ee472e8af6604c32535b5d954fa2e7ca/src/shiv/bootstrap/init.py#L171-L173

I can think of two ways to address this:

1. Change --extend-pythonpath to match shiv's internal behavior. This is not backwards-compatible, however.
2. Add a new option (--override-pythonpath maybe?). Potentially this would also involve deprecating the old option, but perhaps some people have a reason for that variant too.

If you have a preference, I can submit a PR with your preferred option.

Suppose I have 3 files : requirements.txt, module1.py and module2.py I need to bake a single shiv executable with all the dependencies specified inside requirements.txt and later I want to use the same shiv executable for running module1.py as well as module2.py

can we do something like:

• bake shiv using shiv requirements.txt -o shiv.pyz
• to run module1.py, export SHIV_ENTRY_POINT=module1:main and then ./shiv.pyz should run the module1 script
• to run module2.py, export SHIV_ENTRY_POINT=module2:main and then ./shiv.pyz should run the module2 script

does this work? do we have any alternative to do that?

Thanks, Sudarshan

I'm trying to create a zipapp for a simple Django project but I'm getting this error when trying to run it like this ./solution.pyz or python3 solution.pyz:

Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)

The project structure looks like this:

And this is the content of the main.py and build.sh files: main.py

import os
import sys

import django

# setup django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "solution.settings")
django.setup()

try:
    production = sys.argv[1] == "production"
except IndexError:
    production = False

if production:
    import gunicorn.app.wsgiapp as wsgi

    # This is just a simple way to supply args to gunicorn
    sys.argv = [".", "solution.wsgi", "--bind=0.0.0.0:80"]

    wsgi.run()
else:
    from django.core.management import call_command

    call_command("runserver")

build.sh

#!/usr/bin/env bash

# clean old build
rm -r dist solution.pyz

# include the dependencies from `pip freeze`
pip install -r <(pip freeze) --target dist/

# or, if you're using pipenv
# pip install -r  <(pipenv lock -r) --target dist/

# specify which files to be included in the build
# You probably want to specify what goes here
cp -r \
-t dist \
core manage.py db.sqlite3 solution

# finally, build!
shiv --site-packages dist --compressed -p '/usr/bin/env python3' -o solution.pyz -e solution.main

I have an issue with the way Shiv runs ConfigParser for a specific module I need.

configparser.DuplicateOptionError: While reading from PosixPath('/tmp/tmpjl8xhapm/pyroute2-0.7.2.dist-info/entry_points.txt') [line 56]: option 'conntrack' in section 'pyroute2' already exists

pyroute2 has multiple entries of conntrack with different capitalisation. Conntrack/conntrack. Excerpt from pyroute2 setup.cfg

conntrack = pyroute2.conntrack
--
45 | iproute = pyroute2.iproute
… |  
61 | NetlinkDumpInterrupted = pyroute2.netlink.exceptions:NetlinkDumpInterrupted
62 | Conntrack = pyroute2.conntrack:Conntrack

Since this is a linux module it shouldn't be an issue to handle this, but from what I can understand ConfigParser changes everything to lowercase because of Windows.

https://stackoverflow.com/a/19359720/837767 This answer explains it, and shows a fix for the issue.

Any way to implement this into shiv ?

I am trying to build an app for scancode-toolkit and while the build works fine, the built app is failing to run with this error:

$ python3 scancode
....python3: can't find '__main__' module in 'scancode'

I scratched my head and I think this may be subject to these issues https://github.com/python/cpython/issues/77140 and https://github.com/python/cpython/issues/89739

scancode-toolkit happens to contain a good number of data files (~ 68K)

What I did as a proof was to:

1. unzip the built app
2. remove enough files to go below the 65K limit
3. rezip and run the app... and things work fine

Since shiv depends on the boostrapping mechanisms of Python zip importer, this is dependent on @thatch Zip64 support in import being merged in https://github.com/python/cpython/pull/94146 ... therefore I am not sure that there is anything to do here beyond some documentation.

When creating a pyz file with dependencies "requests pip_system_certs", the file "pip_system_certs.pth" extracted in the ~/.shiv/pyzfolder/site-packages is not handled (tested by running import requests; requests.get("https://some.website.needing.system.certificate.store") which triggers a '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)' error)

Would it be possible to make the ~/.shiv/pyzfolder/site-packages folder be considered as one of the directory with .pth that are handled ?

Or automatically run the following code (will handle the .pth files in the folder) during the bootstrap ?

import site
site.addsitedir("~/.shiv/pyzfolder/site-packages")
site.main()