It seems like the first line in the par has an incomplete path to the wrapper script:

$ head bazel-bin/compiler/compiler.par
#!bazel_tools/tools/python/py2wrapper.sh
PK

Some python packages have shared object files that are required to be extracted in order for the system to execute them. This patch extracts all packages that include shared objects to a tmp dir so that the so files can be run properly.

In my PR to add PIP support to Bazel you can reproduce this as shown below.

On a system with:

$ pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)

(I must have gotten this through apt-get or some other super-slow release channel)

If I bazel run the py_binary form, no warning:

rules_python$ bazel run rules_python:piptool --  --name foo --input $PWD/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
INFO: Analysed target //rules_python:piptool (0 packages loaded).
INFO: Found 1 target...
Target //rules_python:piptool up-to-date:
  bazel-bin/rules_python/piptool
INFO: Elapsed time: 0.184s, Critical Path: 0.00s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/rules_python/piptool --name foo --input /home/mattmoor/rules_python/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
Collecting futures>=3.1 (from -r /home/mattmoor/rules_python/examples/helloworld/requirements.txt (line 1))
  File was already downloaded /tmp/pip/futures-3.1.1-py2-none-any.whl
Skipping futures, due to already being wheel.

If I bazel run the .par target, I get a warning:

rules_python$ bazel run rules_python:piptool.par --  --name foo --input $PWD/examples/helloworld/requirements.txt --
directory /tmp/pip/ --output /tmp/requirements.bzl
INFO: Analysed target //rules_python:piptool.par (0 packages loaded).
INFO: Found 1 target...
Target //rules_python:piptool.par up-to-date:
  bazel-bin/rules_python/piptool.par
INFO: Elapsed time: 0.204s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/rules_python/piptool.par --name foo --input /home/mattmoor/rules_python/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
Collecting futures>=3.1 (from -r /home/mattmoor/rules_python/examples/helloworld/requirements.txt (line 1))
  File was already downloaded /tmp/pip/futures-3.1.1-py2-none-any.whl
Skipping futures, due to already being wheel.
You are using pip version 1.5.4, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Happy to provide more info as needed.

Tested: bazel0.20 test //... --all_incompatible_changes

• Reformatted file with Buildifier
• Applied Buildifier automated fixes
• Fixed other issues (depsets are not iterable, tools are separate from other inputs)

Progress towards https://github.com/bazelbuild/rules_python/issues/134

In case the Python runtime used to build a par is not exactly the same path as the target interpreter, this allows the interpreter to be explicitly set instead of inferred from the stub file.

Example:

par_binary(
    name = "mybinary",
    srcs = ["mybinary.py"],
    compiler_args = ["--interpreter", "/usr/bin/env python3.6"],
)

This pull request simply add default shell envs to par_* rules.

Under a lot of customized environments, the process-wrapper and sandbox of Bazel will not carry the shell envs by default. So very often, par rules will fail for this reason. This PR add the default_shell_env to the rule.

Fixes #98 by detecting the default toolchain wrappers and replacing with env versions. I'm perhaps not totally happy with how it's done, as any change in the path to these wrappers would break things again, so open to suggestions. Also, for Python 2 should we use /usr/bin/python or /usr/bin/python2, I prefer the latter but don't know how this behaves on all systems?

Also, now that we have PyInfo provider, the import paths are passed directly from Starlark to the compiler, removing a bit of fragility in parsing the stub file for those. However, I can't see a way to pass the interpreter directly, as this doesn't appear to be available to Starlark. This removed a TODO with reference b/29227737, if that means anything internally?

Tested with both the default toolchain and a custom one to ensure the correct shebang ends up at the top of the par file. It might be worth adding a test that excercises this, but until the toolchains flag is flipped in CI the test will always fail.

This updates the par_*** macros to propagate tags to the underlying rules. This is important, as currently the manual tag doesn't work fro the par_*** macros.

If the main python script contains from __future__ import ... statements, running par file causes the following error:

$ ./train.par
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 165, in _run_module_as_main
    mod_name, loader, code, fname = _get_main_module_details(_Error)
  File "/usr/lib/python2.7/runpy.py", line 133, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/lib/python2.7/runpy.py", line 119, in _get_module_details
    code = loader.get_code(mod_name)
  File "./train.par/__main__.py", line 74
SyntaxError: from __future__ imports must occur at the beginning of the file

This seems to be due to the boilerplate added by subpar compiler before the __future statements.:

# Boilerplate added by subpar/compiler/python_archive.py
from subpar.runtime import support as _
_.setup(import_roots=[u'__main__', u'six_archive'])
del _
# End boilerplate

This makes it possible to run py_tests as .par files, which is occasionally useful for testing code that behaves differently when called from within a zipped executable. It also enables the uncommon (but valid, I'm pretty sure) scenario of having a par_binary definition with testonly=True set.

On some systems I see errors like the following:

benley@alnitak:~/pm/pi-k8s$ bazel build  @subpar//compiler:compiler.par --verbose_failures --strategy=PythonCompile=standalone                               
INFO: Found 1 target...           
ERROR: /home/benley/.cache/bazel/_bazel_benley/9e259f4d6c3d097d1daa32f0372be769/external/subpar/compiler/BUILD:40:1: Building par file @subpar//compiler:compiler.par failed: compiler failed: error executing command                                                                
  (cd /home/benley/.cache/bazel/_bazel_benley/9e259f4d6c3d097d1daa32f0372be769/execroot/pi-k8s && \                                        
  exec env - \                    
  bazel-out/host/bin/external/subpar/compiler/compiler --manifest_file bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler.par_SOURCES --outputpar bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler.par --stub_file bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler external/subpar/compiler/compiler.py): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.                    
Traceback (most recent call last):
  File "bazel-out/host/bin/external/subpar/compiler/compiler", line 168, in <module>                                                       
    Main()                        
  File "bazel-out/host/bin/external/subpar/compiler/compiler", line 148, in Main                                                           
    raise AssertionError('Could not find python binary: ' + PYTHON_BINARY)                                                                 
AssertionError: Could not find python binary: python                 
Target @subpar//compiler:compiler.par failed to build                
INFO: Elapsed time: 0.103s, Critical Path: 0.03s

On others it works.

The difference appears to be in the behaviour of env - across linux distributions. On Ubuntu and Debian, the resulting "empty" environment still has $PATH set to a default value:

benley@aram:~$ env - bash -c set
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="i586-pc-linux-gnu")
BASH_VERSION='4.3.30(1)-release'
DIRSTACK=()
EUID=1009
GROUPS=()
HOSTNAME=aram
HOSTTYPE=i586
IFS=$' \t\n'
MACHTYPE=i586-pc-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
PPID=6844
PS4='+ '
PWD=/home/benley
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=1009
_=bash

(same output on ubuntu 16.04)

On at least one other distro (NixOS linux), there seems to be no such default PATH:

benley@alnitak:~$ env - bash -c set
BASH=/run/current-system/sw/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="4" [2]="12" [3]="1" [4]="release" [5]="x86_64-unknown-linux-gnu")
BASH_VERSION='4.4.12(1)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOSTNAME=alnitak
HOSTTYPE=x86_64
IFS=$' \t\n'
MACHTYPE=x86_64-unknown-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/no-such-path
PPID=3931
PS4='+ '
PWD=/home/benley/pm/pi-k8s
SHELL=/run/current-system/sw/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=1000
_=bash

It's not entirely clear to me why PATH gets set like that, but it certainly breaks the compiler's build.

Bumps setuptools from 5.5.1 to 65.5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This change includes a fix to #39

• Remove the stubfile parsing when the interpreter is specified.
• And use the interpreter as the first argument while running the compiler.par on windows, because compiler.par is not recognised as win32 application and it requires the python to invoke it. For others, the shebang is specified in the file and it works.

Ubuntu 20.04 uses Python 3, which might be what's causing this.

Lots of test failures, see https://buildkite.com/bazel/subpar/builds/7413#7f8773f9-2385-4244-8a00-da4c4dd9a2ab

This is very unlikely to be fixed due to Subpar's maintenance status but it's good to document it here to potentially save others some debugging time.

Any .par written by Subpar using zipfile will be ZIP64 format whenever the file size exceeds ZIP64_LIMIT = (1 << 31) - 1. See https://github.com/python/cpython/blob/ffd87b7093109c279caf8e3ca060f408a102388a/Lib/zipfile.py#L56

As of Python 3.10 zipimport does not support ZIP64. The module function in question is _read_directory: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/zipimport.py#L408-L468

On attempting to run the subpar like python3 foo.par you will get an obscure error:

can't find '__main__' module in /foo.par

If you do python3 -vv foo.par you will see a logline:

# zipimport: found 0 names in /abs/path/to/foo.par

Python's zipimport._read_directory has failed to find any files because it has incorrectly calculated the byte offset for the first ZIP record file. It has incorrectly calculated the byte offset because it does not understand ZIP64.

So if you see this error check the size of your .par.

par_size = ...
max_bytes = (1 << 31) -1
if max_bytes < par_size:
    print("bad news")

cc @ducthienbui97 who completed this bug investigation.

Subpar fails when py_runtime is provided a label for interpreter location, see: https://github.com/google/subpar/issues/98#issuecomment-529904795

@thundergolfer brings up the point that for nix builds, or other hermetic python builds, subpar will not work. Ideally subpar should bundle the interpreter binary into the par for these instances.

I see the subpar is not longer being actively maintained, so no big deal. I think it's still important to have this issue open in case other people run into the same issue.

It seems like --stub_file is only parsed to derive which interpreter is being used. Correct me if I'm wrong, but it seems like passing in --interpreter should short-circuit parsing that file. Seems like this will allow for using in-workspace py_runtime definitions as long as the interpreter is specified explicitly in the parfile() rule.

I might be missing some important context though.