Python process launching

Related tags

python devops subprocess
Overview
Logo

Version Downloads Status Python Versions Build Status Coverage Status

sh is a full-fledged subprocess replacement for Python 2.6 - 3.8, PyPy and PyPy3 that allows you to call any program as if it were a function:

from sh import ifconfig
print(ifconfig("eth0"))

sh is not a collection of system commands implemented in Python.

Complete documentation here

Installation

$> pip install sh

Support

Developers

Updating the docs

Check out the gh-pages branch and follow the README.rst there.

Testing

I've included a Docker test suite in the docker_test_suit/ folder. To build the image, cd into that directory and run:

$> ./build.sh

This will install ubuntu 18.04 LTS and all python versions from 2.6-3.8. Once it's done, stay in that directory and run:

$> ./run.sh

This will mount your local code directory into the container and start the test suite, which will take a long time to run. If you wish to run a single test, you may pass that test to ./run.sh:

$> ./run.sh FunctionalTests.test_unicode_arg

To run a single test for a single environment:

$> ./run.sh -e 3.4 FunctionalTests.test_unicode_arg

Coverage

First run all of the tests:

$> python sh.py test

This will aggregate a .coverage. You may then visualize the report with:

$> coverage report

Or generate visual html files with:

$> coverage html

Which will create ./htmlcov/index.html that you may open in a web browser.

Issues
  • Standard file descriptors should work as generators

    Standard file descriptors should work as generators

    I've implemented a test at https://github.com/pcn/pbs in the generator branch. I think it's clumsy, but I'm still feeling my way around.

    Use case:

    Do something useful with vmstat, tcpdump, etc. These commands are most useful when producing output constantly, and pbs should be able to consume these line by line.

    The implementation avoids calling subprocess.communicate and selects directly from the pbs_object.process objects stdout.

    I'm looking for thoughts on how to do this "right" (e.g. should it be a subclass or a separate module that inherits from pbs?). Currently it changes how it's used, but it is convenient because if only the last command in a pipeline has _generator=True set, then everything should work as normal (once I hook stdin back up :) except the last command will yield lines that can be worked with.

    feature 
    opened by pcn 32
  • sh gets zero exit code for nonzero exit code

    sh gets zero exit code for nonzero exit code

    Consider the following bundle of code:

    http://inversethought.com/jordi/wtf.zip

    Run ./wtf and notice how there is no Python exception. Now look at what sh.py is running, and it's ss/bin/scansetup, which does error out. If you edit wtf to remove the sh.cd("..") call, you'll see that now you do get a pretty Python stack trace.

    I have spent some time trying to debug this, and I am utterly baffled. I have managed to reproduce this on a few systems, including Mac OS X. The Python version has been 2.6.x on all systems I've tested. I haven't tried with a different Python version.

    opened by jordigh 28
  • Suggest pbs into the Standard Library

    Suggest pbs into the Standard Library

    who can to write a PEP for putting pbs into the Standard Library ? I know it might be a bit early, and pbs need to be a bit more mature for that...

    But I was banging my head with subprocess, so many times until I got it right. I need to head to documentation, every time I want to use subprocess.

    with pbs it's much much more simple...

    docs 
    opened by fruch 25
  • Does not work when imported from compiled-only modules

    Does not work when imported from compiled-only modules

    Because pbs assumes that the module that first imports comes from a readable .py file, it won't work when imported from modules where there's only a .pyc. For example:

    test_a.py:

    import test_b
    test_b.main()
    

    test_b.py:

    from pbs import echo
    def main():
        echo('Hello world!')
    

    Run python test_a.py once, then delete python test_b.py (leaving only python test_b.pyc). Now, running test_a.py will crash:

    Traceback (most recent call last):
    File "test_a.py", line 2, in <module>
        import test_b
    File "/home/me/pbs/test_b.py", line 2, in <module>
    File "/home/me/pbs/pbs.py", line 419, in <module>
        with open(script, "r") as h: source = h.readlines()
    IOError: [Errno 2] No such file or directory: '/home/me/pbs/test_b.py'
    

    This could come up when a program is packaged somehow (e.g. frozen in a zip file).


    All in all, the magic here is pretty fragile and will undoubtedly fail in other mysterious ways. Maybe disallowing import * and allowing the above instead (and also import from the REPL) would be the lesser evil?

    bug low priority 
    opened by encukou 24
  • Setting default redirections for all commands

    Setting default redirections for all commands

    Hello,

    First, a big THANK YOU for your work on this project. I find it extremly useful, simple and pythonic. And it's going to be extremely useful to convince my coworkers to migrate their shell scripts to Python :)

    Now, would you be open to allowing some kind of global configuration, specifically on the default redirection for stderr ?

    Currently, some "hardcoded" defaults are defined there: https://github.com/amoffat/sh/blob/master/sh.py#L669 And by default the commands stderr is simply discarded.

    Could that default behaviour be somehow configurable ? I would like the commands stderr to be written to the parent Python script stderr by default. And in the same spirit, it could be sometimes handy to forward stdout the same way.

    I'd love to work on a pull request if you are ok with this feature request.

    Regards

    opened by Lucas-C 23
  • Pipeline failure

    Pipeline failure

    Hi,

    Given the code import sh sh.dd(sh.dd('if=/dev/zero', 'bs=1M', "count=1024"), 'of=/dev/null')

    sh dies with a MemoryError as seen in http://pastebin.ca/2306288

    It looks as though sh is trying to buffer the input rather than connecting the stdout of the inner dd process to the stdin of the outer dd process, and is such blowing up.

    To give you an example of the behaviour I was expecting, see the example below.

    import subprocess
    
    s = subprocess.Popen(['dd', 'if=/dev/zero', 'bs=1M', 'count=10240'], stdout=subprocess.PIPE)
    r = subprocess.Popen(['dd', 'of=/dev/null', 'bs=1M'], stdin=s.stdout)
    
    r.wait()
    

    I've taken a look through sh.py but I can't seem to work out what exactly I'd need to do in order to patch this. Would someone mind lending a hand?

    opened by asharp 19
  • Fix: catching IOError errno=35 on mac os

    Fix: catching IOError errno=35 on mac os

    Using sh in a python script that is running under jenkins, we got the following Exception :

    Exception in thread Thread-330: Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run self.__target(_self.__args, *_self.__kwargs) File "/usr/local/lib/python2.7/site-packages/sh.py", line 1484, in output_thread done = stream.read() File "/usr/local/lib/python2.7/site-packages/sh.py", line 1974, in read self.write_chunk(chunk) File "/usr/local/lib/python2.7/site-packages/sh.py", line 1949, in write_chunk self.should_quit = self.process_chunk(chunk) File "/usr/local/lib/python2.7/site-packages/sh.py", line 1850, in process handler.flush() IOError: [Errno 35] Resource temporarily unavailable

    We identified that this is an issue that may happen under OSX only, and the proposed pull request fixed the issue. This is due to the fact that OSX deals with non blocking IOs in a different way, and the non fatal IOError 35 may happen, you then just have to retry the operation.

    cant reproduce 
    opened by madlag 18
  • Better support for windows platfrom

    Better support for windows platfrom

    • reading PATHEXT to search for executable files
    • building internal commands from cmd.exe, and adding them also
    • added support for uppercase internal command
    • added a section to the README.md
    • support for non-ascii windows console (thanks to @stania)
    feature 
    opened by fruch 17
  • long line truncated in stdout ?

    long line truncated in stdout ?

    It looks like long line are truncated, which make a command fails.

    sh.grep(sh.ps('aux'), "-ie", "java.*Cassandra")
    

    while this shell command works well:

     ps aux | grep -ie "java.*Cassandra"
    

    Anything I should know about the line truncated?

    opened by aboudreault 17
  • Overload

    Overload "|" for piping

    It might be cool to overload "|" for piping ops. I tried this at my branch. It would be cool to grow that into a proper feature.

    opened by bebraw 16
  • WIP: asyncio support

    WIP: asyncio support

    Adds the ability to await sh commands, as well as do async for on commands that could be iterated over.

    One test doesn't pass yet (test_async_iter_exc)

    opened by amoffat 0
  • Make sh.which() an internal function

    Make sh.which() an internal function

    Rename sh.which so normal 'from sh import which' hits the system binary.

    Fixes #579

    opened by ecederstrand 0
  • Type Annotations

    Type Annotations

    It'd be nice if this package shipped mypy type annotations.

    breaking change 
    opened by remexre 4
  • sh.which doesn't return an object and doesn't throw exceptions

    sh.which doesn't return an object and doesn't throw exceptions

    Hi, I experience this unexpected behavior: Is this a bug, or am I doing something wrong?

    Python Version:

    $ python --version
    Python 3.9.5
    

    sh Version: 1.14.2

    Code snippet:

     try:
          # i expect behavior as described here:
          # https://amoffat.github.io/sh/sections/exit_codes.html
          output = sh.which('xxxxx')
          # output returns a string or None , not an object
          # print(output.exit_code) fails because neither 'str' nor 'None'  has the attribute 'exit_code' 
     except ErrorReturnCode:
          # this exception is not thrown 
          logger.error("program not found")
    
    bug breaking change 
    opened by StPanning 9
  • WIP: Release/v2.0.0

    WIP: Release/v2.0.0

    opened by amoffat 0
  • _done callback consistent deadlock when accessing RunningCommand arg passed to callback

    _done callback consistent deadlock when accessing RunningCommand arg passed to callback

    This consistently deadlocks:

    from sh import sleep
    
    def done(cmd, success, exit_code):
        print(cmd, success, exit_code)
    
    sleep('1', _done=done)
    

    This is possibly related to #413 and #518. This is on version 1.14.1 and Python 3.7 (but I doubt that matters).

    Now I suppose one might say "Well, don't do that with cmd and the done callback" ... However, if the argument is passed to the callback it should be safe to use, IMO. And none of the test cases cover accessing done callback arg 0 at all.

    The reason it deadlocks is because of a double wait on _wait_lock: _wait_lock is already held whenever self._process_just_ended() is called (either via self.is_alive() or self.wait()), self._process_just_ended() is what calls the done callback, and then the printing of cmd tries to coerce it to a string using its stdout value, which then also calls self.wait(), which tries to acquire _wait_lock again and deadlocks.

    You can see this in the stack trace:

    ^CTraceback (most recent call last):
      File "y.py", line 6, in <module>
        sleep('1', _done=done)
      File ".../site-packages/sh.py", line 1520, in __call__
        return RunningCommand(cmd, call_args, stdin, stdout, stderr)
      File ".../site-packages/sh.py", line 784, in __init__
        self.wait()
      File ".../site-packages/sh.py", line 831, in wait
        exit_code = self.process.wait()
      File ".../site-packages/sh.py", line 2491, in wait
        self._process_just_ended()
      File ".../site-packages/sh.py", line 2442, in _process_just_ended
        done_callback(success, self.exit_code)
      File "y.py", line 4, in done
        print(cmd, success, exit_code)
      File ".../site-packages/sh.py", line 930, in __str__
        return self.__unicode__()
      File ".../site-packages/sh.py", line 937, in __unicode__
        if self.process and self.stdout:
      File ".../site-packages/sh.py", line 869, in stdout
        self.wait()
      File ".../site-packages/sh.py", line 831, in wait
        exit_code = self.process.wait()
      File ".../site-packages/sh.py", line 2457, in wait
        with self._wait_lock:
    KeyboardInterrupt 
    

    I believe this can be fixed simply by changing _wait_lock to a threading.RLock() instead of threading.Lock(). (I tried it, and it seemed to work just fine.)

    However, I do think a fresh look is needed at what _wait_lock is protecting. The comments suggest it is only for self.exit_code race prevention, but the lock is being held for much more than that. I think the acquisition of this lock can be more fine-grained: only when self.exit_code is being written, and since self.exit_code never changes once written you don't need to hold the lock while reading it once you know it's not None.

    You can do something like this, I think:

    def _do_stuff_when_exit_code_is_already_set():
        pass
    
    if self.exit_code is not None:
        _do_stuff_when_exit_code_is_already_set()
        return
    
    #  it appears that wait lock is None, but now we acquire the lock to be sure
    while True:
        with self._wait_lock:
            if self.exit_code is not None:
                break
            # do stuff that will set exit_code
            break
    _do_stuff_when_exit_code_is_already_set()
    

    (Also, thanks for this library; I like the interface.)

    bug 
    opened by 0xkag 1
  • sh 2.0.0 requests

    sh 2.0.0 requests

    I'd like to aggregate some requests for 2.0.0 here. This is a great time to include breaking changes, since we have no obligation to support backwards compatibility with 1.*

    The list so far, checkmarks next to the things that are done:

    • [x] Drop support for Python < 3.6
    • [x] Use tox for all tests, drop custom test harness code https://github.com/amoffat/sh/issues/526
    • [x] If running synchronously, only return a string, otherwise RunningCommand (tangentially related to https://github.com/amoffat/sh/issues/518)
    • [ ] Support asyncio and allow commands to be awaited
    • [x] Change process model https://github.com/amoffat/sh/issues/495
    • [x] Better _tee support https://github.com/amoffat/sh/issues/215
    • [ ] Migrate docs to readthedocs and version them (1.* and 2.*)
    • [x] Type annotations (https://github.com/amoffat/sh/issues/581)
    • [x] Remove b_cd support and encourage pushd (https://github.com/amoffat/sh/pull/584#discussion_r698055681)

    I will keep this issue open for awhile for feedback, and then organize issues and attach them to a milestone. @ecederstrand

    opened by amoffat 4
  • Support for list kwargs

    Support for list kwargs

    For ex:

    s3cmd_options = dict(
        cache_control="public, max-age={}".format(365 * 24 * 60 * 60),
        exclude="*",
        include=[
            "bar/assets/*",
            "foo/assets/*",
        ],
    )
    sh.aws.s3.sync(local_path, dest_path, **s3cmd_options)
    

    Expected: /usr/bin/aws s3 sync local_path dest_path --exclude=* --include=bar/assets/* --include=foo/assets/* --cache-control=public, max-age=31536000

    Actual: /usr/bin/aws s3 sync local_path dest_path --exclude=* --include=['bar/assets/*', 'foo/assets/*'] --cache-control=public, max-age=31536000

    Most of the command line args do not support array arg value (maybe space-separated string). I know this can be worked around by sending array as sh.aws.s3.sync(local_path, dest_path, '--include="bar/assets/*"', '--include="foo/assets/*"') but that kind of defeats the purpose of passing kwargs. Maybe there is a way to do this that I'm missing.

    feature 
    opened by kumarappan-arumugam 1
  • Parallelize docker test suite

    Parallelize docker test suite

    The current docker test suite installs all supported versions of Python (2.6-3.8) and runs the tests across them multiple times (using different pollers and system locales). Right now they run consecutively, which is slow.

    I'd like to change this to be parallel, which would require spinning up multiple containers (instead of just 1), and switching which python executes the tests from the docker run. I also want to continue to support the current behavior of fast failures.

    "But why not tox?" is a question that I've fielded a number of times. Tox support is terrible on older Python versions, and while I am happy to support others using it, I want a test suite that is separate from tox.

    feature low priority 
    opened by amoffat 0
  • Does RunningCommand.__eq__ make sense?

    Does RunningCommand.__eq__ make sense?

    It was the root cause of https://github.com/amoffat/sh/issues/413

    Here is RunningCommand.__eq__. Basically the issue was that attempting to call list.remove(p) on a list of processes was forcing many of those processes to evaluate/finish, due to __eq__ indirectly calling wait().

    The question is, does it even make sense to compare processes like this? You might be able to have __eq__ just compare pids, but pids can also be re-used so it isn't perfect.

    @ecederstrand what do you think?

    breaking change 
    opened by amoffat 4
Owner
Andrew Moffat
Tech Generalist
Andrew Moffat
Supervisor process control system for UNIX

Supervisor Supervisor is a client/server system that allows its users to control a number of processes on UNIX-like operating systems. Supported Platf

Supervisor 7k Oct 17, 2021
Jurigged lets you update your code while it runs.

jurigged Jurigged lets you update your code while it runs. Using it is trivial: python -m jurigged your_script.py Change some function or method with

Olivier Breuleux 515 Oct 17, 2021
A Python module for controlling interactive programs in a pseudo-terminal

Pexpect is a Pure Python Expect-like module Pexpect makes Python a better tool for controlling other applications. Pexpect is a pure Python module for

null 2.1k Oct 16, 2021