pip-run - dynamic dependency loader for Python

Overview
tests Code style: Black https://readthedocs.org/projects/pip-run/badge/?version=latest

pip-run provides on-demand temporary package installation for a single interpreter run.

It replaces this series of commands (or their Windows equivalent):

$ virtualenv --python pythonX.X --system-site-packages $temp/env
$ $temp/env/bin/pip install pkg1 pkg2 -r reqs.txt
$ $temp/env/bin/python ...
$ rm -rf $temp/env

With this single-line command:

$ pythonX.X -m pip-run pkg1 pkg2 -r reqs.txt -- ...

Features include

  • Downloads missing dependencies and makes their packages available for import.
  • Installs packages to a special staging location such that they're not installed after the process exits.
  • Relies on pip to cache downloads of such packages for reuse.
  • Leaves no trace of its invocation (except files in pip's cache).
  • Supersedes installed packages when required.
  • Relies on packages already satisfied [1].
  • Re-uses the pip tool chain for package installation.

pip-run is not intended to solve production dependency management, but does aim to address the other, one-off scenarios around dependency management:

  • trials and experiments
  • build setup
  • test runners
  • just in time script running
  • interactive development
  • bug triage

pip-run is a compliment to Pip and Virtualenv and Setuptools, intended to more readily address the on-demand needs.

[1] Except when a requirements file is used.

Installation

pip-run is meant to be installed in the system site packages alongside pip, though it can also be installed in a virtualenv.

Usage

  • as script launcher
  • as runtime dependency context manager
  • as interactive interpreter in dependency context
  • as module launcher (akin to python -m)

Invoke pip-run from the command-line using the console entry script (simply pip-run) or using the module executable ( python -m pip-run). This latter usage is particularly convenient for testing a command across various Python versions.

Parameters following pip-run are passed directly to pip install, so pip-run numpy will install numpy (reporting any work done during the install) and pip-run -q -r requirements.txt will quietly install all the requirements listed in a file called requirements.txt. Any environment variables honored by pip are also honored.

Following the parameters to pip install, one may optionally include a -- after which any parameters will be passed to a Python interpreter in the context.

Examples

The examples folder in this project includes some examples demonstrating the power and usefulness of the project. Read the docs on those examples for instructions.

In many of these examples, the option -q is passed to pip-run to suppress the output from pip.

Module Script Runner

Perhaps the most powerful usage of pip-run is its ability to invoke executable modules and packages via runpy (aka python -m):

$ pip-run -q pycowsay -- -m pycowsay "moove over, pip-run"

  -------------------
< moove over, pip-run >
  -------------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

cowsay example animation

Interactive Interpreter

pip-run also offers a painless way to run a Python interactive interpreter in the context of certain dependencies:

$ /clean-install/python -m pip-run -q boto
>>> import boto
>>>

Command Runner

Note that everything after the -- is passed to the python invocation, so it's possible to have a one-liner that runs under a dependency context:

$ python -m pip-run -q requests -- -c "import requests; print(requests.get('https://pypi.org/project/pip-run').status_code)"
200

As long as pip-run is installed in each of Python environments on the system, this command can be readily repeated on the other python environments by specifying the relevant interpreter:

$ python2.7 -m pip-run ...

or on Windows:

$ py -2.7 -m pip-run ...

Experiments and Testing

Because pip-run provides a single-command invocation, it is great for experiments and rapid testing of various package specifications.

Consider a scenario in which one wishes to create an environment where two different versions of the same package are installed, such as to replicate a broken real-world environment. Stack two invocations of pip-run to get two different versions installed:

$ pip-run -q keyring==21.8.0 -- -m pip-run -q keyring==22.0.0 -- -c "import importlib.metadata, pprint; pprint.pprint([dist._path for dist in importlib.metadata.distributions() if dist.metadata['name'] == 'keyring'])"
[PosixPath('/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-run-a3xvd267/keyring-22.0.0.dist-info'),
PosixPath('/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-run-1fdjsgfs/keyring-21.8.0.dist-info')]

Script Runner

Let's say you have a script that has a one-off purpose. It's either not part of a library, where dependencies are normally declared, or it is normally executed outside the context of that library. Still, that script probably has dependencies, say on requests. Here's how you can use pip-run to declare the dependencies and launch the script in a context where those dependencies have been resolved.

First, add a __requires__ directive at the head of the script:

#!/usr/bin/env python

__requires__ = ['requests']

import requests

req = requests.get('https://pypi.org/project/pip-run')
print(req.status_code)

Then, simply invoke that script with pip-run:

$ python -m pip-run -q -- myscript.py
200

The format for requirements must follow PEP 508.

pip-run also recognizes a global __index_url__ attribute. If present, this value will supply --index-url to pip with the attribute value, allowing a script to specify a custom package index:

#!/usr/bin/env python

__requires__ = ['my_private_package']
__index_url__ = 'https://my.private.index/'

import my_private_package
...

Supplying parameters to Pip

If you've been using pip-run, you may have defined some requirements in the __requires__ of a script, but now you wish to install those to a more permanent environment. pip-run provides a routine to facilitate this case:

$ python -m pip_run.read-deps script.py
my_dependency

If you're on Unix, you may pipe this result directly to pip:

$ pip install $(python -m pip_run.read-deps script.py)

And since pipenv uses the same syntax, the same technique works for pipenv:

$ pipenv install $(python -m pip_run.read-deps script.py)

How Does It Work

pip-run effectively does the following:

  • pip install -t $TMPDIR
  • PYTHONPATH=$TMPDIR python
  • cleanup

For specifics, see pip_run.run().

Limitations

  • Due to limitations with pip, pip-run cannot run with "editable" (-e) requirements.
  • pip-run uses a sitecustomize module to ensure that .pth files in the requirements are installed. As a result, any environment that has a sitecustomize module will find that module masked when running under pip-run.

Comparison with pipx

The pipx project is another mature project with similar goals. Both projects expose a project and its dependencies in ephemeral environments. The main difference is pipx primarily exposes Python binaries (console scripts) from those environments whereas pip-run exposes a Python context (including runpy scripts).

Feature pip-run pipx
user-mode operation
invoke console scripts  
invoke runpy modules  
run standalone scripts  
interactive interpreter with deps  
re-use existing environment  
ephemeral environments
persistent environments  
PEP 582 support  
Specify optional dependencies  
Python 2 support  

Comparison with virtualenvwrapper mktmpenv

The mkvirtualenv project attempts to address some of the use-cases that pip-run solves, especially with the mktmpenv command, which destroys the virtualenv after deactivation. The main difference is that pip-run is transient only for the invocation of a single command, while mktmpenv lasts for a session.

Feature pip-run mktmpenv
create temporary package environment
re-usable across python invocations  
portable  
one-line invocation  
multiple interpreters in session  
run standalone scripts  
interactive interpreter with deps
re-use existing environment  
ephemeral environments
persistent environments  

Integration

The author created this package with the intention of demonstrating the capability before integrating it directly with pip in a command such as pip run. After proposing the change, the idea was largely rejected in pip 3971.

If you would like to see this functionality made available in pip, please upvote or comment in that ticket.

Versioning

pip-run uses semver, so you can use this library with confidence about the stability of the interface, even during periods of great flux.

Testing

Invoke tests with tox.

Comments
  • 9.4.0: pytest is failing in two units

    9.4.0: pytest is failing in two units

    I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

    • python3 -sBm build -w --no-isolation
    • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
    • install .whl file in </install/prefix>
    • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
    • I'm running build in build env which is cu off from access to the public network

    Here is pytest output:

    + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
    + /usr/bin/pytest -ra -m 'not network'
    ==================================================================================== test session starts ====================================================================================
    platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0, configfile: pytest.ini
    collected 37 items
    
    pip_run/commands.py ...                                                                                                                                                               [  8%]
    pip_run/deps.py ..                                                                                                                                                                    [ 13%]
    pip_run/persist.py .                                                                                                                                                                  [ 16%]
    pip_run/read-deps.py .                                                                                                                                                                [ 18%]
    pip_run/scripts.py ..                                                                                                                                                                 [ 24%]
    tests/test_deps.py ..........                                                                                                                                                         [ 51%]
    tests/test_launch.py ..x..                                                                                                                                                            [ 64%]
    tests/test_scripts.py F..........F.                                                                                                                                                   [100%]
    
    ========================================================================================= FAILURES ==========================================================================================
    _____________________________________________________________________________________ test_pkg_imported _____________________________________________________________________________________
    
    tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0')
    
        def test_pkg_imported(tmp_path):
            """
            Create a script that loads cython and ensure it runs.
            """
            jaraco.path.build(
                {
                    'script': DALS(
                        """
                        import path
                        print("Successfully imported path.py")
                        """
                    )
                },
                tmp_path,
            )
            script = tmp_path / 'script'
            pip_args = ['path.py']
            cmd = [sys.executable, '-m', 'pip-run'] + pip_args + ['--', str(script)]
    
    >       out = subprocess.check_output(cmd, text=True)
    
    tests/test_scripts.py:35:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /usr/lib64/python3.8/subprocess.py:415: in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    input = None, capture_output = False, timeout = None, check = True
    popenargs = (['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script'],), kwargs = {'stdout': -1, 'text': True}
    process = <subprocess.Popen object at 0x7fa5b25f3c10>, stdout = '', stderr = None, retcode = 1
    
        def run(*popenargs,
                input=None, capture_output=False, timeout=None, check=False, **kwargs):
            """Run command with arguments and return a CompletedProcess instance.
    
            The returned instance will have attributes args, returncode, stdout and
            stderr. By default, stdout and stderr are not captured, and those attributes
            will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
            If check is True and the exit code was non-zero, it raises a
            CalledProcessError. The CalledProcessError object will have the return code
            in the returncode attribute, and output & stderr attributes if those streams
            were captured.
    
            If timeout is given, and the process takes too long, a TimeoutExpired
            exception will be raised.
    
            There is an optional argument "input", allowing you to
            pass bytes or a string to the subprocess's stdin.  If you use this argument
            you may not also use the Popen constructor's "stdin" argument, as
            it will be used internally.
    
            By default, all communication is in bytes, and therefore any "input" should
            be bytes, and the stdout and stderr will be bytes. If in text mode, any
            "input" should be a string, and stdout and stderr will be strings decoded
            according to locale encoding, or by "encoding" if set. Text mode is
            triggered by setting any of text, encoding, errors or universal_newlines.
    
            The other arguments are the same as for the Popen constructor.
            """
            if input is not None:
                if kwargs.get('stdin') is not None:
                    raise ValueError('stdin and input arguments may not both be used.')
                kwargs['stdin'] = PIPE
    
            if capture_output:
                if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                    raise ValueError('stdout and stderr arguments may not be used '
                                     'with capture_output.')
                kwargs['stdout'] = PIPE
                kwargs['stderr'] = PIPE
    
            with Popen(*popenargs, **kwargs) as process:
                try:
                    stdout, stderr = process.communicate(input, timeout=timeout)
                except TimeoutExpired as exc:
                    process.kill()
                    if _mswindows:
                        # Windows accumulates the output in a single blocking
                        # read() call run on child threads, with the timeout
                        # being done in a join() on those threads.  communicate()
                        # _after_ kill() is required to collect that and add it
                        # to the exception.
                        exc.stdout, exc.stderr = process.communicate()
                    else:
                        # POSIX _communicate already populated the output so
                        # far into the TimeoutExpired exception.
                        process.wait()
                    raise
                except:  # Including KeyboardInterrupt, communicate handled that.
                    process.kill()
                    # We don't call process.wait() as .__exit__ does that for us.
                    raise
                retcode = process.poll()
                if check and retcode:
    >               raise CalledProcessError(retcode, process.args,
                                             output=stdout, stderr=stderr)
    E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit status 1.
    
    /usr/lib64/python3.8/subprocess.py:516: CalledProcessError
    ----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c073a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c172e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17700>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c178b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
    ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
    ERROR: No matching distribution found for path.py
    WARNING: There was an error checking the latest version of pip.
    Traceback (most recent call last):
      File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
        __name__ == '__main__' and run()
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
        with deps.load(*deps.not_installed(pip_args)) as home:
      File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
        Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
      File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-2owvtu_x'), 'path.py')' returned non-zero exit status 1.
    ___________________________________________________________________________ test_pkg_loaded_from_alternate_index ____________________________________________________________________________
    
    tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0')
    
        def test_pkg_loaded_from_alternate_index(tmp_path):
            """
            Create a script that loads cython from an alternate index
            and ensure it runs.
            """
            jaraco.path.build(
                {
                    'script': DALS(
                        """
                        __requires__ = ['path.py']
                        __index_url__ = 'https://devpi.net/root/pypi/+simple/'
                        import path
                        print("Successfully imported path.py")
                        """
                    )
                },
                tmp_path,
            )
            cmd = [sys.executable, '-m', 'pip-run', '-v', '--', str(tmp_path / 'script')]
    
    >       out = subprocess.check_output(cmd, text=True)
    
    tests/test_scripts.py:176:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /usr/lib64/python3.8/subprocess.py:415: in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    input = None, capture_output = False, timeout = None, check = True
    popenargs = (['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script'],), kwargs = {'stdout': -1, 'text': True}
    process = <subprocess.Popen object at 0x7fa5b25373a0>, stdout = 'Looking in indexes: https://devpi.net/root/pypi/+simple/\n', stderr = None, retcode = 1
    
        def run(*popenargs,
                input=None, capture_output=False, timeout=None, check=False, **kwargs):
            """Run command with arguments and return a CompletedProcess instance.
    
            The returned instance will have attributes args, returncode, stdout and
            stderr. By default, stdout and stderr are not captured, and those attributes
            will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
            If check is True and the exit code was non-zero, it raises a
            CalledProcessError. The CalledProcessError object will have the return code
            in the returncode attribute, and output & stderr attributes if those streams
            were captured.
    
            If timeout is given, and the process takes too long, a TimeoutExpired
            exception will be raised.
    
            There is an optional argument "input", allowing you to
            pass bytes or a string to the subprocess's stdin.  If you use this argument
            you may not also use the Popen constructor's "stdin" argument, as
            it will be used internally.
    
            By default, all communication is in bytes, and therefore any "input" should
            be bytes, and the stdout and stderr will be bytes. If in text mode, any
            "input" should be a string, and stdout and stderr will be strings decoded
            according to locale encoding, or by "encoding" if set. Text mode is
            triggered by setting any of text, encoding, errors or universal_newlines.
    
            The other arguments are the same as for the Popen constructor.
            """
            if input is not None:
                if kwargs.get('stdin') is not None:
                    raise ValueError('stdin and input arguments may not both be used.')
                kwargs['stdin'] = PIPE
    
            if capture_output:
                if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                    raise ValueError('stdout and stderr arguments may not be used '
                                     'with capture_output.')
                kwargs['stdout'] = PIPE
                kwargs['stderr'] = PIPE
    
            with Popen(*popenargs, **kwargs) as process:
                try:
                    stdout, stderr = process.communicate(input, timeout=timeout)
                except TimeoutExpired as exc:
                    process.kill()
                    if _mswindows:
                        # Windows accumulates the output in a single blocking
                        # read() call run on child threads, with the timeout
                        # being done in a join() on those threads.  communicate()
                        # _after_ kill() is required to collect that and add it
                        # to the exception.
                        exc.stdout, exc.stderr = process.communicate()
                    else:
                        # POSIX _communicate already populated the output so
                        # far into the TimeoutExpired exception.
                        process.wait()
                    raise
                except:  # Including KeyboardInterrupt, communicate handled that.
                    process.kill()
                    # We don't call process.wait() as .__exit__ does that for us.
                    raise
                retcode = process.poll()
                if check and retcode:
    >               raise CalledProcessError(retcode, process.args,
                                             output=stdout, stderr=stderr)
    E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero exit status 1.
    
    /usr/lib64/python3.8/subprocess.py:516: CalledProcessError
    ----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1580>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1730>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e18e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1a90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
    ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
    ERROR: No matching distribution found for path.py
    WARNING: There was an error checking the latest version of pip.
    Traceback (most recent call last):
      File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
        __name__ == '__main__' and run()
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
        with deps.load(*deps.not_installed(pip_args)) as home:
      File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
        Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
      File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-xz8c_iz3'), '-v', '--index-url', 'https://devpi.net/root/pypi/+simple/', 'path.py')' returned non-zero exit status 1.
    ================================================================================== short test summary info ==================================================================================
    XFAIL tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
    FAILED tests/test_scripts.py::test_pkg_imported - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit sta...
    FAILED tests/test_scripts.py::test_pkg_loaded_from_alternate_index - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero ...
    =================================================================== 2 failed, 34 passed, 1 xfailed in 1647.11s (0:27:27) ====================================================================
    

    Here is list of installed modules in build env

    Package                       Version
    ----------------------------- -----------------
    alabaster                     0.7.12
    appdirs                       1.4.4
    asn1crypto                    1.5.1
    attrs                         22.2.0
    autocommand                   2.2.1
    Babel                         2.11.0
    bcrypt                        3.2.2
    Brlapi                        0.8.3
    build                         0.9.0
    cffi                          1.15.1
    charset-normalizer            3.0.1
    contourpy                     1.0.6
    cryptography                  38.0.4
    cssselect                     1.1.0
    cycler                        0.11.0
    distro                        1.8.0
    dnspython                     2.2.1
    docutils                      0.19
    exceptiongroup                1.0.0
    extras                        1.0.0
    fastjsonschema                2.16.1
    fixtures                      4.0.0
    fonttools                     4.38.0
    gpg                           1.18.0-unknown
    idna                          3.4
    imagesize                     1.4.1
    importlib-metadata            5.1.0
    importlib-resources           5.10.1
    inflect                       5.0.0
    iniconfig                     1.1.1
    jaraco.classes                3.2.3
    jaraco.context                4.2.0
    jaraco.functools              3.5.2
    jaraco.packaging              9.1.1
    jaraco.path                   3.4.0
    jaraco.text                   3.11.0
    jaraco.tidelift               1.5.0
    jeepney                       0.8.0
    Jinja2                        3.1.2
    jsonschema                    4.17.3
    jupyter_core                  5.1.1
    keyring                       23.11.0
    kiwisolver                    1.4.4
    libcomps                      0.1.19
    louis                         3.24.0
    lxml                          4.9.1
    MarkupSafe                    2.1.1
    matplotlib                    3.6.2
    more-itertools                9.0.0
    nbformat                      5.7.0
    numpy                         1.23.1
    olefile                       0.46
    packaging                     21.3
    path                          16.6.0
    pbr                           5.9.0
    pep517                        0.13.0
    Pillow                        9.3.0
    pip                           22.3.1
    pkgutil_resolve_name          1.3.10
    platformdirs                  2.6.0
    pluggy                        1.0.0
    ply                           3.11
    pyasn1                        0.4.8
    pyasn1-modules                0.2.8
    pycparser                     2.21
    Pygments                      2.13.0
    PyGObject                     3.42.2
    pyparsing                     3.0.9
    pyrsistent                    0.19.2
    pytest                        7.2.0
    python-dateutil               2.8.2
    pytz                          2022.4
    PyYAML                        6.0
    requests                      2.28.1
    requests-toolbelt             0.10.1
    rpm                           4.17.0
    rst.linker                    2.3.1
    scour                         0.38.2
    SecretStorage                 3.3.2
    setuptools                    65.6.3
    setuptools-scm                7.0.5
    six                           1.16.0
    snowballstemmer               2.2.0
    Sphinx                        5.3.0
    sphinxcontrib-applehelp       1.0.2.dev20221204
    sphinxcontrib-devhelp         1.0.2.dev20221204
    sphinxcontrib-htmlhelp        2.0.0
    sphinxcontrib-jsmath          1.0.1.dev20221204
    sphinxcontrib-qthelp          1.0.3.dev20221204
    sphinxcontrib-serializinghtml 1.1.5
    testtools                     2.5.0
    tomli                         2.0.1
    tpm2-pkcs11-tools             1.33.7
    tpm2-pytss                    1.1.0
    traitlets                     5.8.0
    typing_extensions             4.4.0
    urllib3                       1.26.12
    wheel                         0.38.4
    zipp                          3.11.0
    
    opened by kloczek 8
  • Shared redesign of embedded requirements feature

    Shared redesign of embedded requirements feature

    Background

    In pypa/pip#3971, key members of the PyPA have leveled critiques of the embedded requirements feature, where a user can supply requirements and other common installer directives to signal to the tool how (best) to run the script. This feature emerged from the use-case where a script author would like to distribute a single file and have that file be runnable by any number of end users with minimal guidance, allowing the file to be published to a gist or alongside other scripts in a directory without needing additional context for executability.

    Goals

    • the tool should be able to infer install requirements and index URL.
    • the tool must be able to parse these directives without executing the script (as executing the script should be able to rely on the presence of those items).
    • the syntax should be as intuitive as possible. As a corollary, the syntax should aim to re-use syntax familiar to the user (primarily the author, but also the end user).
    • (optional) the tool should be able to infer additional installer directives such as --extra-index-url or --quiet.
    opened by jaraco 8
  • Git URLs no longer supported in __requires__?

    Git URLs no longer supported in __requires__?

    Somewhere between versions 2.15.1 and 3.0, rwt stopped supporting (specifically, it appears to flat-out ignore) package specifiers of the form git+ssh://[email protected]/repo.git in __requires__ lists, though it continues to support them in requirements.txt files. I see no mention of this change in CHANGES.rst; was it intentional? If not, could we have the old behavior back?

    opened by jwodder 8
  • app_paths dependency is a nuisance

    app_paths dependency is a nuisance

    Per https://github.com/jaraco/pip-run/commit/346392685903dfcd1414e438c734ac4263047665#r94301964, @mgorny reports that the app_paths dependency is problemmatic. Let's explore ways to fix that.

    opened by jaraco 7
  • Environment caching

    Environment caching

    I have a number of scripts for my personal use and I found pip-run very useful for managing my dependencies. However it still takes several seconds for pip to install the packages, even if it already uses the download cache of pip.

    I propose adding an option (e.g. --cache) where the directory containing the packages is not deleted after the program exits, and it can be reused as long as the dependencies list does not change. Similar tools (e.g. kotlin-main-kts) also have caching that makes rerunning a script practically free.

    There are several things that need to be considered

    • The cached environments need to be managed, since they take up disk space (e.g. numpy takes up around 70MiB). A simple solution can be keep using /tmp and let the system manages it.
    • Identification of a cached environment. I think using the hash of the dependency list should work.

    I think it is best to keep the functionality simple, and in the worst case just destroy and recreate the environment. But I do expect it to be able to avoid recreation of the environment if I run the same script twice.

    opened by knthmn 7
  • Automatic dependency management

    Automatic dependency management

    Hey @jaraco. I am reading about the project and for some reason I want to believe that rwt is able to figure out and fetch dependencies automatically, but so far I haven't found proof of that - we still need to create a custom "import" section just for rwt. In this respect rwt is still manual dependency management like http://docs.groovy-lang.org/latest/html/documentation/grape.html

    enhancement 
    opened by techtonik 6
  • Change `-m pip-run` in README to `-m pip_run`

    Change `-m pip-run` in README to `-m pip_run`

    Recent versions of Python (not sure when it started; 3.9.0? 3.9.1?) no longer support using the -m option with an invalid module name. This PR thus updates the documentation to invoke pip-run with -m in a way that is guaranteed to work with all Python versions.

    opened by jwodder 5
  • setup.py containing __requires__ fails when run without rwt

    setup.py containing __requires__ fails when run without rwt

    hi @jaraco,

    Here's a setup.py which defines __requires__ for use with rwt tool. It's taken straight from your README.rst for the sake of example.

    #!/usr/bin/env python
    
    __requires__ = ['setuptools', 'setuptools_scm']
    
    from setuptools import setup
    
    setup(
        name='test_rwt',
        use_scm_version=True,
        setup_requires=__requires__,
    )
    

    Now, if I try to run this without also invoking python -m rwt -- setup.py ..., like the usual:

    $ python3 setup.py --help
    

    I get this error from pkg_resources:

    Traceback (most recent call last):
      File "setup_rwt.py", line 5, in <module>
        from setuptools import setup
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
        import setuptools.version
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
        import pkg_resources
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3018, in <module>
        @_call_aside
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3002, in _call_aside
        f(*args, **kwargs)
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3031, in _initialize_master_working_set
        working_set = WorkingSet._build_master()
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in _build_master
        ws.require(__requires__)
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 962, in require
        needed = self.resolve(parse_requirements(requirements))
      File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 848, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application
    

    The error happens when a requirement listed in __requires__ (like setuptools_scm here) is not already installed in the current environment.

    It seems like pkg_resources (imported by setuptools) also attempts to use the __requires__ variable, in WorkinSet._build_master method:

    https://github.com/pypa/setuptools/blob/089cdeb489a0fa94d11b7307b54210ef9aa40511/pkg_resources/init.py#L640-L658

    This behavior is documented in the pkg_resources docs: http://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects

    All distributions available directly on sys.path will be activated automatically when pkg_resources is imported. This behaviour can cause version conflicts for applications which require non-default versions of those distributions. To handle this situation, pkg_resources checks for a requires attribute in the main module when initializing the default working set, and uses this to ensure a suitable version of each affected distribution is activated. For example:

    __requires__ = ["CherryPy < 3"] # Must be set before pkg_resources import
    import pkg_resources
    

    In your REAME.rst, you say that this technique:

    When invoked with rwt, the dependencies will be assured before the script is run, or if run with setuptools, the dependencies will be loaded using the older technique, so the script is backward compatible.

    However, it appears to me that one cannot have such a backward-compatible setup.py which also takes advantage of rwt via the __requires__ variable, because once one uses that, it fails as one tries to run it in the old-fashioned way.

    Am I missing something? Thank you for your help.

    In the meantime I think I'll just use python3 -m rwt -r requirements/setup.txt -- setup.py ... instead of __requires__.

    opened by anthrotype 5
  • Add a console entry point

    Add a console entry point

    I gave this a try, and my immediate thought was to run it as "rwt". I know the docs say to use "python -m rwt", but maybe add a console entry point as a convenience?

    opened by pfmoore 5
  • double-dash is irritating, especially for script invocation

    double-dash is irritating, especially for script invocation

    In https://github.com/pypa/pip/issues/3971#issuecomment-1320927420, @pfmoore has to say:

    I find having to include the -- to separate the command parameters clumsy, and I'm not in love with details like the default being to print all of pip's output. Which all adds up to a general sense of "I never think to use it, because it's sort of alright, but irritating". Sorry, I know that's annoyingly vague, and I do think that the UI is the hardest part of this to get right. What I want is to be able to type pip-run my_script.py arg1 arg2 arg3, and have it run my script with all dependencies available. The other features of pip-run are secondary (to me).

    In this issue, I'll focus on the double-dash behavior.

    The reason for the -- is to clearly separate the parameters to the script from parameters to pip/pip-run. Since there are two concerns to deal with here, there needs to be clear separation. For me, 95% of the usage is on the install parameters side, with the right hand side accepting any argument to Python (sometimes no args (for a repl), sometimes -c, sometimes -m, and sometimes a script). This generalization means the tool has a wide variety of applications.

    I'd like to improve the reported scenario, but I struggle a bit with how to do that without drastically degrading the current experience.

    One possible improvement could be to allow the -- to be optional in the specific case where a script is the first parameter. That is, if the first parameter is an extant file, that's not a valid parameter to pip install, so pip-run could infer a -- before it. If someone wanted to pass arguments to the interpreter or to pip, they would need to bring back the --, but for convenience, a simple script invocation could omit it. WDYT?

    opened by jaraco 4
  • Give pip_run.read-deps an option for newline-terminated output

    Give pip_run.read-deps an option for newline-terminated output

    Feature request: Give python -m pip_run.read-deps a --newline (or whatever you want to call it) option that would cause the individual requirements to be output on separate lines. This would make it easy to create requirements.txt files for __requires__-using scripts, especially when dealing with requirements with markers like importlib-metadata>=3.6; python_version < "3.10", which are currently output with spaces embedded, making it difficult to separate them from adjacent requirements.

    opened by jwodder 4
  • [upstream] Better support for already-installed requirements

    [upstream] Better support for already-installed requirements

    I have a scenario where if I make the following requirements.txt file:

    cryptography<=3.2.1,>2.9.2
    azure-storage-blob==12.8.1
    

    and pass it with -r, everything is fine.

    If I add this to the script i'm running, and remove the -r flag:

    __requires__ = [
        'cryptography<=3.2.1,>2.9.2',
        'azure-storage-blob>12',
    ]
    

    I get an error because my site-packages has jwt 1.1.0 and cryptography 3.2.1 installed:

    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 3.4.7 which is incompatible.
    

    What i'd like:

    • a flag that will give me the -r behavior without having to make another separate file

    What I'd love:

    • a flag that will make it not matter what I have installed in site-packages at all. I just want a one line command that will run my script regardless of what might have been installed with pip on the machine in the past.
    opened by fcrick 4
Releases(v10.0.2)
Owner
Jason R. Coombs
Jason R. Coombs
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.1k Jan 1, 2023
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.2k Jan 5, 2023
An installation and dependency system for Python

Pyflow Simple is better than complex - The Zen of Python Pyflow streamlines working with Python projects and files. It's an easy-to-use CLI app with a

David O'Connor 1.2k Dec 23, 2022
Install and Run Python Applications in Isolated Environments

pipx — Install and Run Python Applications in Isolated Environments Documentation: https://pipxproject.github.io/pipx/ Source Code: https://github.com

null 5.8k Dec 31, 2022
A set of tools to keep your pinned Python dependencies fresh.

pip-tools = pip-compile + pip-sync A set of command line tools to help you keep your pip-based packages fresh, even when you've pinned them. You do pi

Jazzband 6.5k Dec 29, 2022
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/. bandersnatch >=4.0 supports Linux, MacOSX + Wind

Python Packaging Authority 345 Dec 28, 2022
Python PyPi staging server and packaging, testing, release tool

devpi: PyPI server and packaging/testing/release tool This repository contains three packages comprising the core devpi system on the server and clien

null 629 Jan 1, 2023
The Python Package Index

Warehouse Warehouse is the software that powers PyPI. See our development roadmap, documentation, and architectural overview. Getting Started You can

Python Packaging Authority 3.1k Jan 1, 2023
The Python package installer

pip - The Python Package Installer pip is the package installer for Python. You can use pip to install packages from the Python Package Index and othe

Python Packaging Authority 8.4k Dec 30, 2022
:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED Forks: https://github.com/orsinium/forks DepHell -- project management for Python. Why it is better than all other tools: Form

DepHell 1.7k Dec 30, 2022
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 23.5k Jan 6, 2023
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/. bandersnatch >=4.0 supports Linux, MacOSX + Wind

Python Packaging Authority 345 Dec 28, 2022
Simple Library Management made with Python

Installation pip install mysql-connector-python NOTE: You must make a database (library) & and table (books, student) to hold all data. Languange and

SonLyte 10 Oct 21, 2021
PokerFace is a Python package for various poker tools.

PokerFace is a Python package for various poker tools. The following features are present in PokerFace... Types for cards and their componen

Juho Kim 21 Dec 29, 2022
Example for how to package a Python library based on Cython.

Cython sample module This project is an example of a module that can be built using Cython. It is an upgrade from a similar model developed by Arin Kh

Juan José García Ripoll 4 Aug 28, 2022
A software manager for easy development and distribution of Python code

Piper A software manager for easy development and distribution of Python code. The main features that Piper adds to Python are: Support for large-scal

null 13 Nov 22, 2022
🗽 Like yarn outdated/upgrade, but for pip. Upgrade all your pip packages and automate your Python Dependency Management.

pipupgrade The missing command for pip Table of Contents Features Quick Start Usage Basic Usage Docker Environment Variables FAQ License Features Upda

Achilles Rasquinha 529 Dec 31, 2022
Dependency Injector is a dependency injection framework for Python.

What is Dependency Injector? Dependency Injector is a dependency injection framework for Python. It helps implementing the dependency injection princi

ETS Labs 2.6k Jan 4, 2023
Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage and potential attacks.

Dependency Combobulator Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage

Apiiro 84 Dec 23, 2022
DepFine Is a tool to find the unregistered dependency based on dependency confusion valunerablility and lead to RCE

DepFine DepFine Is a tool to find the unregistered dependency based on dependency confusion valunerablility and lead to RCE Installation: You Can inst

Hossam mesbah 14 Nov 11, 2022