pip-run - dynamic dependency loader for Python

Jason R. Coombs

Last update: Dec 14, 2022

Related tags

Package Management pip-run

Overview

https://readthedocs.org/projects/pip-run/badge/?version=latest

pip-run provides on-demand temporary package installation for a single interpreter run.

It replaces this series of commands (or their Windows equivalent):

$ virtualenv --python pythonX.X --system-site-packages $temp/env
$ $temp/env/bin/pip install pkg1 pkg2 -r reqs.txt
$ $temp/env/bin/python ...
$ rm -rf $temp/env

With this single-line command:

$ pythonX.X -m pip-run pkg1 pkg2 -r reqs.txt -- ...

Features include

Downloads missing dependencies and makes their packages available for import.
Installs packages to a special staging location such that they're not installed after the process exits.
Relies on pip to cache downloads of such packages for reuse.
Leaves no trace of its invocation (except files in pip's cache).
Supersedes installed packages when required.
Relies on packages already satisfied [1].
Re-uses the pip tool chain for package installation.

pip-run is not intended to solve production dependency management, but does aim to address the other, one-off scenarios around dependency management:

trials and experiments
build setup
test runners
just in time script running
interactive development
bug triage

pip-run is a compliment to Pip and Virtualenv and Setuptools, intended to more readily address the on-demand needs.

[1]	Except when a requirements file is used.

Installation

pip-run is meant to be installed in the system site packages alongside pip, though it can also be installed in a virtualenv.

Usage

as script launcher
as runtime dependency context manager
as interactive interpreter in dependency context
as module launcher (akin to python -m)

Invoke pip-run from the command-line using the console entry script (simply pip-run) or using the module executable ( python -m pip-run). This latter usage is particularly convenient for testing a command across various Python versions.

Parameters following pip-run are passed directly to pip install, so pip-run numpy will install numpy (reporting any work done during the install) and pip-run -q -r requirements.txt will quietly install all the requirements listed in a file called requirements.txt. Any environment variables honored by pip are also honored.

Following the parameters to pip install, one may optionally include a -- after which any parameters will be passed to a Python interpreter in the context.

Examples

The examples folder in this project includes some examples demonstrating the power and usefulness of the project. Read the docs on those examples for instructions.

In many of these examples, the option -q is passed to pip-run to suppress the output from pip.

Module Script Runner

Perhaps the most powerful usage of pip-run is its ability to invoke executable modules and packages via runpy (aka python -m):

$ pip-run -q pycowsay -- -m pycowsay "moove over, pip-run"

  -------------------
< moove over, pip-run >
  -------------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Interactive Interpreter

pip-run also offers a painless way to run a Python interactive interpreter in the context of certain dependencies:

$ /clean-install/python -m pip-run -q boto
>>> import boto
>>>

Command Runner

Note that everything after the -- is passed to the python invocation, so it's possible to have a one-liner that runs under a dependency context:

$ python -m pip-run -q requests -- -c "import requests; print(requests.get('https://pypi.org/project/pip-run').status_code)"
200

As long as pip-run is installed in each of Python environments on the system, this command can be readily repeated on the other python environments by specifying the relevant interpreter:

$ python2.7 -m pip-run ...

or on Windows:

$ py -2.7 -m pip-run ...

Experiments and Testing

Because pip-run provides a single-command invocation, it is great for experiments and rapid testing of various package specifications.

Consider a scenario in which one wishes to create an environment where two different versions of the same package are installed, such as to replicate a broken real-world environment. Stack two invocations of pip-run to get two different versions installed:

$ pip-run -q keyring==21.8.0 -- -m pip-run -q keyring==22.0.0 -- -c "import importlib.metadata, pprint; pprint.pprint([dist._path for dist in importlib.metadata.distributions() if dist.metadata['name'] == 'keyring'])"
[PosixPath('/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-run-a3xvd267/keyring-22.0.0.dist-info'),
PosixPath('/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-run-1fdjsgfs/keyring-21.8.0.dist-info')]

Script Runner

Let's say you have a script that has a one-off purpose. It's either not part of a library, where dependencies are normally declared, or it is normally executed outside the context of that library. Still, that script probably has dependencies, say on requests. Here's how you can use pip-run to declare the dependencies and launch the script in a context where those dependencies have been resolved.

First, add a __requires__ directive at the head of the script:

#!/usr/bin/env python

__requires__ = ['requests']

import requests

req = requests.get('https://pypi.org/project/pip-run')
print(req.status_code)

Then, simply invoke that script with pip-run:

$ python -m pip-run -q -- myscript.py
200

The format for requirements must follow PEP 508.

pip-run also recognizes a global __index_url__ attribute. If present, this value will supply --index-url to pip with the attribute value, allowing a script to specify a custom package index:

#!/usr/bin/env python

__requires__ = ['my_private_package']
__index_url__ = 'https://my.private.index/'

import my_private_package
...

Supplying parameters to Pip

If you've been using pip-run, you may have defined some requirements in the __requires__ of a script, but now you wish to install those to a more permanent environment. pip-run provides a routine to facilitate this case:

$ python -m pip_run.read-deps script.py
my_dependency

If you're on Unix, you may pipe this result directly to pip:

$ pip install $(python -m pip_run.read-deps script.py)

And since pipenv uses the same syntax, the same technique works for pipenv:

$ pipenv install $(python -m pip_run.read-deps script.py)

How Does It Work

pip-run effectively does the following:

pip install -t $TMPDIR
PYTHONPATH=$TMPDIR python
cleanup

For specifics, see pip_run.run().

Limitations

Due to limitations with pip, pip-run cannot run with "editable" (-e) requirements.
pip-run uses a sitecustomize module to ensure that .pth files in the requirements are installed. As a result, any environment that has a sitecustomize module will find that module masked when running under pip-run.

Comparison with pipx

The pipx project is another mature project with similar goals. Both projects expose a project and its dependencies in ephemeral environments. The main difference is pipx primarily exposes Python binaries (console scripts) from those environments whereas pip-run exposes a Python context (including runpy scripts).

Feature	pip-run	pipx
user-mode operation	✓	✓
invoke console scripts		✓
invoke runpy modules	✓
run standalone scripts	✓
interactive interpreter with deps	✓
re-use existing environment	✓
ephemeral environments	✓	✓
persistent environments		✓
PEP 582 support		✓
Specify optional dependencies	✓
Python 2 support	✓

Comparison with virtualenvwrapper mktmpenv

The mkvirtualenv project attempts to address some of the use-cases that pip-run solves, especially with the mktmpenv command, which destroys the virtualenv after deactivation. The main difference is that pip-run is transient only for the invocation of a single command, while mktmpenv lasts for a session.

Feature	pip-run	mktmpenv
create temporary package environment	✓	✓
re-usable across python invocations		✓
portable		✓
one-line invocation	✓
multiple interpreters in session	✓
run standalone scripts		✓
interactive interpreter with deps	✓	✓
re-use existing environment	✓
ephemeral environments	✓	✓
persistent environments		✓

Integration

The author created this package with the intention of demonstrating the capability before integrating it directly with pip in a command such as pip run. After proposing the change, the idea was largely rejected in pip 3971.

If you would like to see this functionality made available in pip, please upvote or comment in that ticket.

Versioning

pip-run uses semver, so you can use this library with confidence about the stability of the interface, even during periods of great flux.

Testing

Invoke tests with tox.

Comments

9.4.0: pytest is failing in two units

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
I'm running build in build env which is cu off from access to the public network

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0, configfile: pytest.ini
collected 37 items

pip_run/commands.py ...                                                                                                                                                               [  8%]
pip_run/deps.py ..                                                                                                                                                                    [ 13%]
pip_run/persist.py .                                                                                                                                                                  [ 16%]
pip_run/read-deps.py .                                                                                                                                                                [ 18%]
pip_run/scripts.py ..                                                                                                                                                                 [ 24%]
tests/test_deps.py ..........                                                                                                                                                         [ 51%]
tests/test_launch.py ..x..                                                                                                                                                            [ 64%]
tests/test_scripts.py F..........F.                                                                                                                                                   [100%]

========================================================================================= FAILURES ==========================================================================================
_____________________________________________________________________________________ test_pkg_imported _____________________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0')

    def test_pkg_imported(tmp_path):
        """
        Create a script that loads cython and ensure it runs.
        """
        jaraco.path.build(
            {
                'script': DALS(
                    """
                    import path
                    print("Successfully imported path.py")
                    """
                )
            },
            tmp_path,
        )
        script = tmp_path / 'script'
        pip_args = ['path.py']
        cmd = [sys.executable, '-m', 'pip-run'] + pip_args + ['--', str(script)]

>       out = subprocess.check_output(cmd, text=True)

tests/test_scripts.py:35:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25f3c10>, stdout = '', stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c073a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c172e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17700>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c178b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
    Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-2owvtu_x'), 'path.py')' returned non-zero exit status 1.
___________________________________________________________________________ test_pkg_loaded_from_alternate_index ____________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0')

    def test_pkg_loaded_from_alternate_index(tmp_path):
        """
        Create a script that loads cython from an alternate index
        and ensure it runs.
        """
        jaraco.path.build(
            {
                'script': DALS(
                    """
                    __requires__ = ['path.py']
                    __index_url__ = 'https://devpi.net/root/pypi/+simple/'
                    import path
                    print("Successfully imported path.py")
                    """
                )
            },
            tmp_path,
        )
        cmd = [sys.executable, '-m', 'pip-run', '-v', '--', str(tmp_path / 'script')]

>       out = subprocess.check_output(cmd, text=True)

tests/test_scripts.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25373a0>, stdout = 'Looking in indexes: https://devpi.net/root/pypi/+simple/\n', stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1580>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1730>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e18e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1a90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
    Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-xz8c_iz3'), '-v', '--index-url', 'https://devpi.net/root/pypi/+simple/', 'path.py')' returned non-zero exit status 1.
================================================================================== short test summary info ==================================================================================
XFAIL tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
FAILED tests/test_scripts.py::test_pkg_imported - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit sta...
FAILED tests/test_scripts.py::test_pkg_loaded_from_alternate_index - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero ...
=================================================================== 2 failed, 34 passed, 1 xfailed in 1647.11s (0:27:27) ====================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
asn1crypto                    1.5.1
attrs                         22.2.0
autocommand                   2.2.1
Babel                         2.11.0
bcrypt                        3.2.2
Brlapi                        0.8.3
build                         0.9.0
cffi                          1.15.1
charset-normalizer            3.0.1
contourpy                     1.0.6
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
exceptiongroup                1.0.0
extras                        1.0.0
fastjsonschema                2.16.1
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.18.0-unknown
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
importlib-resources           5.10.1
inflect                       5.0.0
iniconfig                     1.1.1
jaraco.classes                3.2.3
jaraco.context                4.2.0
jaraco.functools              3.5.2
jaraco.packaging              9.1.1
jaraco.path                   3.4.0
jaraco.text                   3.11.0
jaraco.tidelift               1.5.0
jeepney                       0.8.0
Jinja2                        3.1.2
jsonschema                    4.17.3
jupyter_core                  5.1.1
keyring                       23.11.0
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.2
more-itertools                9.0.0
nbformat                      5.7.0
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
path                          16.6.0
pbr                           5.9.0
pep517                        0.13.0
Pillow                        9.3.0
pip                           22.3.1
pkgutil_resolve_name          1.3.10
platformdirs                  2.6.0
pluggy                        1.0.0
ply                           3.11
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pyrsistent                    0.19.2
pytest                        7.2.0
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
requests                      2.28.1
requests-toolbelt             0.10.1
rpm                           4.17.0
rst.linker                    2.3.1
scour                         0.38.2
SecretStorage                 3.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
tomli                         2.0.1
tpm2-pkcs11-tools             1.33.7
tpm2-pytss                    1.1.0
traitlets                     5.8.0
typing_extensions             4.4.0
urllib3                       1.26.12
wheel                         0.38.4
zipp                          3.11.0

opened by kloczek 8

Shared redesign of embedded requirements feature
Background

In pypa/pip#3971, key members of the PyPA have leveled critiques of the embedded requirements feature, where a user can supply requirements and other common installer directives to signal to the tool how (best) to run the script. This feature emerged from the use-case where a script author would like to distribute a single file and have that file be runnable by any number of end users with minimal guidance, allowing the file to be published to a gist or alongside other scripts in a directory without needing additional context for executability.

Goals

the tool should be able to infer install requirements and index URL.

the tool must be able to parse these directives without executing the script (as executing the script should be able to rely on the presence of those items).

the syntax should be as intuitive as possible. As a corollary, the syntax should aim to re-use syntax familiar to the user (primarily the author, but also the end user).

(optional) the tool should be able to infer additional installer directives such as --extra-index-url or --quiet.
opened by jaraco 8
Git URLs no longer supported in __requires__?

Somewhere between versions 2.15.1 and 3.0, rwt stopped supporting (specifically, it appears to flat-out ignore) package specifiers of the form git+ssh://[email protected]/repo.git in __requires__ lists, though it continues to support them in requirements.txt files. I see no mention of this change in CHANGES.rst; was it intentional? If not, could we have the old behavior back?

opened by jwodder 8
app_paths dependency is a nuisance

Per https://github.com/jaraco/pip-run/commit/346392685903dfcd1414e438c734ac4263047665#r94301964, @mgorny reports that the app_paths dependency is problemmatic. Let's explore ways to fix that.

opened by jaraco 7
Environment caching
I have a number of scripts for my personal use and I found pip-run very useful for managing my dependencies. However it still takes several seconds for pip to install the packages, even if it already uses the download cache of pip.

I propose adding an option (e.g. --cache) where the directory containing the packages is not deleted after the program exits, and it can be reused as long as the dependencies list does not change. Similar tools (e.g. kotlin-main-kts) also have caching that makes rerunning a script practically free.

There are several things that need to be considered

The cached environments need to be managed, since they take up disk space (e.g. numpy takes up around 70MiB). A simple solution can be keep using /tmp and let the system manages it.

Identification of a cached environment. I think using the hash of the dependency list should work.

I think it is best to keep the functionality simple, and in the worst case just destroy and recreate the environment. But I do expect it to be able to avoid recreation of the environment if I run the same script twice.
opened by knthmn 7
Automatic dependency management

Hey @jaraco. I am reading about the project and for some reason I want to believe that rwt is able to figure out and fetch dependencies automatically, but so far I haven't found proof of that - we still need to create a custom "import" section just for rwt. In this respect rwt is still manual dependency management like http://docs.groovy-lang.org/latest/html/documentation/grape.html
enhancement

opened by techtonik 6
Change `-m pip-run` in README to `-m pip_run`

Recent versions of Python (not sure when it started; 3.9.0? 3.9.1?) no longer support using the -m option with an invalid module name. This PR thus updates the documentation to invoke pip-run with -m in a way that is guaranteed to work with all Python versions.

opened by jwodder 5

setup.py containing requires fails when run without rwt

hi @jaraco,

Here's a setup.py which defines __requires__ for use with rwt tool. It's taken straight from your README.rst for the sake of example.

#!/usr/bin/env python

__requires__ = ['setuptools', 'setuptools_scm']

from setuptools import setup

setup(
    name='test_rwt',
    use_scm_version=True,
    setup_requires=__requires__,
)

Now, if I try to run this without also invoking python -m rwt -- setup.py ..., like the usual:

$ python3 setup.py --help

I get this error from pkg_resources:

Traceback (most recent call last):
  File "setup_rwt.py", line 5, in <module>
    from setuptools import setup
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3018, in <module>
    @_call_aside
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3002, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3031, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in _build_master
    ws.require(__requires__)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 962, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 848, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

The error happens when a requirement listed in __requires__ (like setuptools_scm here) is not already installed in the current environment.

It seems like pkg_resources (imported by setuptools) also attempts to use the __requires__ variable, in WorkinSet._build_master method:

https://github.com/pypa/setuptools/blob/089cdeb489a0fa94d11b7307b54210ef9aa40511/pkg_resources/init.py#L640-L658

This behavior is documented in the pkg_resources docs: http://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects

All distributions available directly on sys.path will be activated automatically when pkg_resources is imported. This behaviour can cause version conflicts for applications which require non-default versions of those distributions. To handle this situation, pkg_resources checks for a requires attribute in the main module when initializing the default working set, and uses this to ensure a suitable version of each affected distribution is activated. For example:

__requires__ = ["CherryPy < 3"] # Must be set before pkg_resources import
import pkg_resources

In your REAME.rst, you say that this technique:

When invoked with rwt, the dependencies will be assured before the script is run, or if run with setuptools, the dependencies will be loaded using the older technique, so the script is backward compatible.

However, it appears to me that one cannot have such a backward-compatible setup.py which also takes advantage of rwt via the __requires__ variable, because once one uses that, it fails as one tries to run it in the old-fashioned way.

Am I missing something? Thank you for your help.

In the meantime I think I'll just use python3 -m rwt -r requirements/setup.txt -- setup.py ... instead of __requires__.

opened by anthrotype 5

Add a console entry point

I gave this a try, and my immediate thought was to run it as "rwt". I know the docs say to use "python -m rwt", but maybe add a console entry point as a convenience?

opened by pfmoore 5
double-dash is irritating, especially for script invocation

In https://github.com/pypa/pip/issues/3971#issuecomment-1320927420, @pfmoore has to say:

I find having to include the -- to separate the command parameters clumsy, and I'm not in love with details like the default being to print all of pip's output. Which all adds up to a general sense of "I never think to use it, because it's sort of alright, but irritating". Sorry, I know that's annoyingly vague, and I do think that the UI is the hardest part of this to get right. What I want is to be able to type pip-run my_script.py arg1 arg2 arg3, and have it run my script with all dependencies available. The other features of pip-run are secondary (to me).

In this issue, I'll focus on the double-dash behavior.

The reason for the -- is to clearly separate the parameters to the script from parameters to pip/pip-run. Since there are two concerns to deal with here, there needs to be clear separation. For me, 95% of the usage is on the install parameters side, with the right hand side accepting any argument to Python (sometimes no args (for a repl), sometimes -c, sometimes -m, and sometimes a script). This generalization means the tool has a wide variety of applications.

I'd like to improve the reported scenario, but I struggle a bit with how to do that without drastically degrading the current experience.

One possible improvement could be to allow the -- to be optional in the specific case where a script is the first parameter. That is, if the first parameter is an extant file, that's not a valid parameter to pip install, so pip-run could infer a -- before it. If someone wanted to pass arguments to the interpreter or to pip, they would need to bring back the --, but for convenience, a simple script invocation could omit it. WDYT?

opened by jaraco 4
Give pip_run.read-deps an option for newline-terminated output

Feature request: Give python -m pip_run.read-deps a --newline (or whatever you want to call it) option that would cause the individual requirements to be output on separate lines. This would make it easy to create requirements.txt files for __requires__-using scripts, especially when dealing with requirements with markers like importlib-metadata>=3.6; python_version < "3.10", which are currently output with spaces embedded, making it difficult to separate them from adjacent requirements.

opened by jwodder 4
[upstream] Better support for already-installed requirements
I have a scenario where if I make the following requirements.txt file:

cryptography<=3.2.1,>2.9.2 azure-storage-blob==12.8.1

and pass it with -r, everything is fine.

If I add this to the script i'm running, and remove the -r flag:

__requires__ = [ 'cryptography<=3.2.1,>2.9.2', 'azure-storage-blob>12', ]

I get an error because my site-packages has jwt 1.1.0 and cryptography 3.2.1 installed:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 3.4.7 which is incompatible.

What i'd like:

a flag that will give me the -r behavior without having to make another separate file

What I'd love:

a flag that will make it not matter what I have installed in site-packages at all. I just want a one line command that will run my script regardless of what might have been installed with pip on the machine in the past.
opened by fcrick 4