Run a subprocess in a pseudo terminal

Overview

Launch a subprocess in a pseudo terminal (pty), and interact with both the process and its pty.

Sometimes, piping stdin and stdout is not enough. There might be a password prompt that doesn't read from stdin, output that changes when it's going to a pipe rather than a terminal, or curses-style interfaces that rely on a terminal. If you need to automate these things, running the process in a pseudo terminal (pty) is the answer.

Interface:

p = PtyProcessUnicode.spawn(['python'])
p.read(20)
p.write('6+6\n')
p.read(20)
Comments
  • Use stdin in child process

    Use stdin in child process

    Is it possible?

    I'd like to spawn a process that reads stdin.

    cat requirements.txt | ./script_that_spawns.py safety --check --stdin
    

    When doing so and trying to read safety output, it blocks, and I have to interrupt it with control-c:

    Traceback (most recent call last):
      File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/bin/failprint", line 8, in <module>
        sys.exit(main())
      File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 125, in main
        return run(
      File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 54, in run
        output.append(process.read())
      File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 818, in read
        b = super(PtyProcessUnicode, self).read(size)
      File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 516, in read
        s = self.fileobj.read1(size)
    KeyboardInterrupt
    

    Here is the actual Python code I'm using:

    process = PtyProcessUnicode.spawn(cmd)
    
    output = []
    
    while True:
        try:
            output.append(process.read())
        except EOFError:
            break
    
    process.close()
    
    opened by pawamoy 12
  • Potential fix for 'exec' failure case

    Potential fix for 'exec' failure case

    • Adding more robust code to handle the case where the exec call within fails. Now, spawn will raise an exception if this happens.
    • Adding a test to ensure this exception is raised when an invalid binary is run
    opened by anwilli5 9
  • Potential performance issue with unbuffered IO and the PtyProcess readline method

    Potential performance issue with unbuffered IO and the PtyProcess readline method

    Calls to self.fileobj.readline() from the PtyProcess readline() method read data one byte at a time (most likely since fileobj is opened with 'buffering=0'.) Thus, this program:

    from ptyprocess import PtyProcess, PtyProcessUnicode
    p = PtyProcess.spawn(['perl',  '-e', '''use 5.010; foreach my $letter ('a'..'z'){ say $letter x 1000; }'''])
    while(1):
        try:
            print p.readline()
        except EOFError:
            break
    p.close()
    

    Has pretty poor performance (output from strace):

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     93.48    0.465020          18     26214         1 read
      2.28    0.011353          23       489       381 open
      0.84    0.004197        4197         1           clone
      0.61    0.003037          19       160       113 stat
    

    Is there a compelling reason to specify that the fileobj should have unbuffered IO?

    The PtyProcess read() method does not experience this behavior because it uses a default buffer size of 1024.

    enhancement help wanted needs-tests 
    opened by anwilli5 7
  • Flit packaging

    Flit packaging

    @jquast flit is my packaging tool to build wheels without involving setuptools. With this branch I can build a wheel by running flit wheel.

    We can use this:

    1. Standalone, by getting rid of setup.py and MANIFEST.in. This means that future releases would only have wheels on PyPI, not sdist tarballs. I'm already doing this for a number of my other projects - pip has been able to install wheels for about 2½ years - but it may surprise some people.
    2. In parallel, using flit to build wheels and setup.py for sdists. There's a risk of duplicated information getting out of date, but the main thing we update is the version number, and flit takes that from __version__, which we need to update anyway.
    3. I have a shim called flituptools so setup.py can use the flit information. But that would limit sdists to use with Python 3.
    opened by takluyver 6
  • Logging of stdout and stderr

    Logging of stdout and stderr

    Hi :)

    I would like to subprocess any given command and log its stdout and stderr separately. This would seem like an easy thing to do, but im having no luck because:

    1. Processes which detect that their stdout/stderr are not ttys will modify their output
    2. Processes which detect that their stdout/stderr are not the same file path will assume theres stdout redirection and modify their output.

    So in jumps ptys and ptyprocess to the resuce. Tie the stdout and stderr to a pty, you get 1 file path like /dev/tty0021, and isatty() returns true for both. Problem is - now we can't distinguish stdout from stderr. Ok, no problem, just make two ptys - one for stdout and one for stderr. But now although both pass isatty(), their file paths will now look like /dev/tty0021 and /dev/tty0022 (for example). The subprocess reacts as if you weren't using a pty in the first place, and you log nothing.

    I have been trying for four months to figure a way around this, and because you are the expert in ptys I thought i might ask you directly - could you think of a way to log stdout and stderr separatly in your program, and still fool a script like this: https://bpaste.net/show/000d6f70ef41

    THANK YOU :D

    opened by JohnLonginotto 6
  • Solaris 11, python2.6 fixes in ptyprocess.py

    Solaris 11, python2.6 fixes in ptyprocess.py

    • termios.tcgetattr(fd)[6][VEOF] becomes value of 1(int), where we expect '\x04' (^D). seems to be a bug with the compilation of python2.6 that Sun provides with Solaris 11. This raises TypeError when ord(1) happens when trying to determine the byte for EOF.

      If _is_solaris is True, and the return value is an integer, just assume its a bork and manually set eof = 4(int)(^D).

    • Fix several "ValueError: zero length field name in format" errors for Python 2.6, which is what ships with Solaris 11. This caused an exception at a critical path in pty.fork(), where the parent would block on os.read of 'exec_err_pipe_read' indefinitely, as the child raised an exception trying to construct the 'tosend' value (though the exception could not be seen). Test runner just blocks indefinitely without such fixes.

    • In spawn(), allow errno.ENXIO when calling inst.setwindowsize(), in some cases, such as spawn(['/bin/true']), the child process may have exited so quickly as to no longer be able to accept any terminal driver calls. Reported by @reynir in https://github.com/pexpect/pexpect/issues/206

    opened by jquast 5
  • Wait() should return normally if the child process has already terminated

    Wait() should return normally if the child process has already terminated

    Calling wait() on a PtyProcess object whose process has already terminated will raise an exception. This behavior is unexpected[1]. I have made a small test to show case the behavior. Using p.isalive() && p.wait() will only make the race condition worse in my tests.

    [1]: subprocess.Popen.wait() returns normally even when the process has already terminated, although it doesn't say so explicitly in the documentation. In java the waitFor() method "returns immediately if the subprocess has already terminated."

    bug 
    opened by reynir 5
  • test failure on FreeBSD

    test failure on FreeBSD

    Running the tests on FreeBSD 11 I see:

    % py.test  
    ============================= test session starts ==============================
    platform freebsd11 -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4
    collected 5 items 
    
    tests/test_invalid_binary.py .
    tests/test_preexec_fn.py ..
    tests/test_spawn.py FF
    
    
    =================================== FAILURES ===================================
    __________________________ PtyTestCase.test_spawn_sh ___________________________
    
    self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>
    
        def test_spawn_sh(self):
            env = os.environ.copy()
            env['FOO'] = 'rebar'
            p = PtyProcess.spawn(['sh'], env=env)
            p.read()
            p.write(b'echo $FOO\n')
            time.sleep(0.1)
            response = p.read()
    >       assert b'rebar' in response
    E       AssertionError: assert 'rebar' in 'echo $FOO\r\n'
    
    tests/test_spawn.py:15: AssertionError
    ______________________ PtyTestCase.test_spawn_unicode_sh _______________________
    
    self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_unicode_sh>
    
        def test_spawn_unicode_sh(self):
            env = os.environ.copy()
            env['FOO'] = 'rebar'
            p = PtyProcessUnicode.spawn(['sh'], env=env)
            p.read()
            p.write(u'echo $FOO\n')
            time.sleep(0.1)
            response = p.read()
    >       assert u'rebar' in response
    \n'     AssertionError: assert 'rebar' in 'echo $FOO
    
    tests/test_spawn.py:31: AssertionError
    ====================== 2 failed, 3 passed in 0.80 seconds ======================
    

    The expected output is produced, it is just not returned by the second p.read() in the test. Inserting a dummy p.read() before response = p.read() gets the tests passing.

    opened by emaste 5
  • Added screen size parameters.

    Added screen size parameters.

    I found myself wanting to set the size of the pty as seen by the subprocess (really for pexpect.spawn, but this is needed first). In some situations it can make parsing/interaction a lot easier.

    Happy to change any stylistic things, eg.

    1. The dimensions=(24, 80) default arg could also be dimensions=None with some if dimensions is not None logic in spawn().
    2. It could be r=24, c=80 instead, but does it make sense to only have one dimension with a default? (Maybe, I don't know.)

    I meant to add a test, but couldn't figure out a simple enough way to test this (eg. if there was a reliable command to check console size it could be like the other spawn tests, but I don't know of one).

    opened by detly 5
  • FreeBSD fails fork_pty: OSError: [Errno 6] Device not configured: '/dev/tty'

    FreeBSD fails fork_pty: OSError: [Errno 6] Device not configured: '/dev/tty'

    Got a FreeBSD (digital ocean droplet, freebsd.pexpect.org) build agent prepared. It raises exception very early in critical codepath causing test runner to fork and eventually the build agent is killed by the kernel due to an OOM condition.

    Error is in method pty_make_controlling_tty at:

            # Verify we now have a controlling tty.
            fd = os.open("/dev/tty", os.O_WRONLY)
    
    [freebsd@freebsd ~]$ sudo -u teamcity -s
    $ cd /opt/TeamCity/work/210ae16cc3f30c30/ptyprocess
    $ . `which virtualenvwrapper.sh`
    $ mkvirtualenv pexpect27 --python=`which python2.7`
    $ pip install -e .
    $ cd ../pexpect
    $ python
    Python 2.7.9 (default, Jan  8 2015, 21:47:19)
    [GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pexpect
    >>> bash = pexpect.spawn('/bin/bash')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "pexpect/pty_spawn.py", line 189, in __init__
        self._spawn(command, args, preexec_fn)
      File "pexpect/pty_spawn.py", line 281, in _spawn
        cwd=self.cwd, **kwargs)
      File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/ptyprocess.py", line 220, in spawn
        pid, fd = _fork_pty.fork_pty()
      File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 30, in fork_pty
        pty_make_controlling_tty(child_fd)
      File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
        fd = os.open("/dev/tty", os.O_WRONLY)
    OSError: [Errno 6] Device not configured: '/dev/tty'
    

    /dev/tty may be opened under normal conditions.

    bug 
    opened by jquast 5
  • Race condition in tests?

    Race condition in tests?

    There seems to be some kind of race condition in the testsuite on both python-3.4 and python-2.7 here on Fedora 21 (it seems to occour more often in python-3.4 though).

    This is the log of the tests: self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>

        def test_spawn_sh(self):
            env = os.environ.copy()
            env['FOO'] = 'rebar'
            p = PtyProcess.spawn(['sh'], env=env)
            p.read()
            p.write(b'echo $FOO\n')
            time.sleep(0.1)
            response = p.read()
            assert b'rebar' in response
    
            p.sendeof()
            p.read()
    
            with self.assertRaises(EOFError):
    >           p.read()
    E           AssertionError: EOFError not raised
    
    tests/test_spawn.py:21: AssertionError
    

    What could cause these random tests? How can I help debugging it?

    opened by tomspur 4
  • Add support for Python 3.10 and 3.11

    Add support for Python 3.10 and 3.11

    Python 3.11 was released on 2022-10-24 🚀

    image


    Also maybe time to drop EOL Python <= 3.6?

    Here's the pip installs for ptyprocess from PyPI for October 2022:

    | category | percent | downloads | |:---------|--------:|-----------:| | 3.7 | 29.33% | 7,840,362 | | 3.8 | 23.53% | 6,289,131 | | 3.9 | 18.12% | 4,843,521 | | 3.10 | 12.51% | 3,344,769 | | 3.6 | 7.83% | 2,092,917 | | null | 6.31% | 1,687,690 | | 2.7 | 1.73% | 462,046 | | 3.11 | 0.42% | 111,185 | | 3.5 | 0.17% | 45,885 | | 3.4 | 0.05% | 13,545 | | 3.12 | 0.00% | 143 | | 3.3 | 0.00% | 69 | | 3.2 | 0.00% | 6 | | Total | | 26,731,269 |

    Source: pip install -U pypistats && pypistats python_minor ptyprocess --last-month

    opened by hugovk 0
  • test_spawn_sh occasionally fails

    test_spawn_sh occasionally fails

    When I started testing ptyprocess with Python 3.11 I found out that test_spawn_sh fails with:

    E       AssertionError: assert b'echo $ENV_KEY; exit 0' in b'\rjkulik@kuldavm-sparc-6717:...process/ptyprocess-0.7.0/tests$ echo $ENV \x08_KEY; exit 0\r\nenv_value\r\n'                                                
    E        +  where b'echo $ENV_KEY; exit 0' = <built-in method strip of bytes object at 0x1fffc6b2a9f8b0>()                                                                                                              
    E        +    where <built-in method strip of bytes object at 0x1fffc6b2a9f8b0> = b'echo $ENV_KEY; exit 0\n'.strip
    

    I did some testing and it seems that this is caused by some kind of a race condition / it's speed dependent. When I run the test suite for the second time, it passes. But then when I delete the __pycache__ directory, it fails again.

    I also found out that when I add a sleep after the spawn, it fails every time in every Python (and not just this test, test_spawn_sh_unicode as well):

    --- ptyprocess-0.7.0/tests/test_spawn.py
    +++ ptyprocess-0.7.0/tests/test_spawn.py
    @@ -21,6 +21,7 @@ class PtyTestCase(unittest.TestCase):
         def _spawn_sh(self, ptyp, cmd, outp, env_value):
             # given,
             p = ptyp.spawn(['sh'], env=self.env)
    +        time.sleep(1)
             p.write(cmd)
     
             # exercise,
    
    opened by kulikjak 0
  • The positions of the two arguments in setwinsize function appear to be reversed

    The positions of the two arguments in setwinsize function appear to be reversed

    Recently I made a Web terminal with ptyprocess, websocket, and XTerm, but eventually found that the Web terminal size could not adapt after the resize event was triggered.

    I even thought my computer was actually a big phone until I switched the positions of two parameters and the page appeared normal.

    Before:

    def _setwinsize(fd, rows, cols):
        # Some very old platforms have a bug that causes the value for
        # termios.TIOCSWINSZ to be truncated. There was a hack here to work
        # around this, but it caused problems with newer platforms so has been
        # removed. For details see https://github.com/pexpect/pexpect/issues/39
        TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
        # Note, assume ws_xpixel and ws_ypixel are zero.
        s = struct.pack('HHHH', rows, cols, 0, 0)
        fcntl.ioctl(fd, TIOCSWINSZ, s)
    

    After using the following modified code, the interface display is normal:

    def _setwinsize(fd, rows, cols):
        # Some very old platforms have a bug that causes the value for
        # termios.TIOCSWINSZ to be truncated. There was a hack here to work
        # around this, but it caused problems with newer platforms so has been
        # removed. For details see https://github.com/pexpect/pexpect/issues/39
        width, height = cols, rows
        TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
        # Note, assume ws_xpixel and ws_ypixel are zero.
        s = struct.pack('HHHH', width, height, 0, 0)
        fcntl.ioctl(fd, TIOCSWINSZ, s)
    
    opened by coderk17 2
  • Add **kwargs to PtyProcess.spawn

    Add **kwargs to PtyProcess.spawn

    The PtyProcessUnicode class accept keywords argument like encoding and codec_errors. However, it seems not easy to set these argument just using PtyProcess.spawn.

    I think we could add a **kwargs to spawn and create the class instance using cls(pid, fd, **kwargs). It will be neat and improve extensibility.

    opened by dong-zeyu 0
  • PtyProcess.read() returns a different value every call

    PtyProcess.read() returns a different value every call

    This is a very severe bug. When calling Ptyprocess.read() the value returned is different almost every time:

    ptyprocess.PtyProcess.spawn(['openssl', "ec", '-noout', '-text', '-in', '/opt/key/s128r1.key']).read()

    The output: image

    And again with the same params: image

    And again: image

    I don't know what is causing this but this is very weird.

    opened by gggal123 1
  • The preexec_fn should be executed before closing the file descriptors.

    The preexec_fn should be executed before closing the file descriptors.

    Currently, the preexec_fn is executed after the file descriptors are closed. This has some unwanted effects:

    • if preexec_fn opens a file descriptor, it will be inherit by the child process
    • if preexec_fn relays on having some file descriptor open, it will crash (see https://github.com/pexpect/pexpect/issues/368)

    The proposal is to move the "close the fds" section below the "execute the preexec_fn" code: https://github.com/pexpect/ptyprocess/blob/master/ptyprocess/ptyprocess.py#L266-L285

    For reference, this is what it does subprocess.Popen: https://github.com/python/cpython/blob/master/Modules/_posixsubprocess.c#L528-L549

    If it is okay, I can do a PR with the fix but I would like to hear your opinions about this.

    opened by eldipa 0
Releases(0.7.0)
Owner
null
git《Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser》(2021) GitHub: [fig5]

Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser Abstract The success of deep denoisers on real-world colo

Yue Cao 51 Nov 22, 2022
Python utility function to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed

iterable-subprocess Python utility function to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be

Department for International Trade 5 Jul 10, 2022
pyshell is a Linux subprocess module

pyshell A Linux subprocess module, An easier way to interact with the Linux shell pyshell should be cross platform but has only been tested with linux

null 4 Mar 2, 2022
POC using subprocess lib in Python 🐍

POC subprocess ☞ POC using the subprocess library with Python. References: https://github.com/GuillaumeFalourd/poc-subprocess https://geekflare.com/le

Guillaume Falourd 2 Nov 28, 2022
The sarge package provides a wrapper for subprocess which provides command pipeline functionality.

Overview The sarge package provides a wrapper for subprocess which provides command pipeline functionality. This package leverages subprocess to provi

Vinay Sajip 14 Dec 18, 2022
A Python module for controlling interactive programs in a pseudo-terminal

Pexpect is a Pure Python Expect-like module Pexpect makes Python a better tool for controlling other applications. Pexpect is a pure Python module for

null 2.3k Dec 26, 2022
Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as a REST API Endpoint.

Jupter Notebook REST API Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as

Invictify 54 Nov 4, 2022
run-js Goal: The Easiest Way to Run JavaScript in Python

run-js Goal: The Easiest Way to Run JavaScript in Python features Stateless Async JS Functions No Intermediary Files Functional Programming CommonJS a

Daniel J. Dufour 9 Aug 16, 2022
A Pygame Hangman Game coded in Python 3. Run Hangman.py in a terminal if you have Python 3

Hangman A Pygame Hangman Game coded in Python 3. Run python3 Hangman.py in a terminal if you have Python 3.

null 1 Dec 24, 2022
This is the Quiz that I made using Python Programming Language. This can only run in the Terminal

This is the Quiz that I made using Python Programming Language. This can only run in the Terminal

YOSHITHA RATHNAYAKE 1 Apr 8, 2022
Gamestonk Terminal is an awesome stock and crypto market terminal

Gamestonk Terminal is an awesome stock and crypto market terminal. A FOSS alternative to Bloomberg Terminal.

Gamestonk Terminal 18.6k Jan 3, 2023
Blackjack-Py is a terminal based game of blackjack within your terminal playing against CPU.

About Blackjack-Py is a terminal based game of blackjack within your terminal playing against CPU. Usage Clone the repo and run it with whatever pytho

Eccentrici 1 Dec 28, 2021
Terminal-Video-Player - A program that can display video in the terminal using ascii characters

Terminal-Video-Player - A program that can display video in the terminal using ascii characters

null 15 Nov 10, 2022
[AAAI 21] Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning

◥ Curriculum Labeling ◣ Revisiting Pseudo-Labeling for Semi-Supervised Learning Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, Vicente Ordonez. In the

UVA Computer Vision 113 Dec 15, 2022
Pseudo-Visual Speech Denoising

Pseudo-Visual Speech Denoising This code is for our paper titled: Visual Speech Enhancement Without A Real Visual Stream published at WACV 2021. Autho

Sindhu 94 Oct 22, 2022
This is an unofficial PyTorch implementation of Meta Pseudo Labels

This is an unofficial PyTorch implementation of Meta Pseudo Labels. The official Tensorflow implementation is here.

Jungdae Kim 320 Jan 8, 2023
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021)

Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021, official Pytorch implementatio

Microsoft 247 Dec 25, 2022
DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

Differentiable Model Compression via Pseudo Quantization Noise DiffQ performs differentiable quantization using pseudo quantization noise. It can auto

Facebook Research 145 Dec 30, 2022
[CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

TorchSemiSeg [CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision by Xiaokang Chen1, Yuhui Yuan2, Gang Zeng1, Jingdong Wang

Chen XiaoKang 387 Jan 8, 2023
Pytorch implementation of the paper SPICE: Semantic Pseudo-labeling for Image Clustering

SPICE: Semantic Pseudo-labeling for Image Clustering By Chuang Niu and Ge Wang This is a Pytorch implementation of the paper. (In updating) SOTA on 5

Chuang Niu 154 Dec 15, 2022