Git Product home page Git Product logo

python-subprocess32's Introduction

subprocess32

PyPI version POSIX Build Status Windows Build Status

This is a backport of the Python 3 subprocess module for use on Python 2. This code has not been tested on Windows or other non-POSIX platforms.

subprocess32 includes many important reliability bug fixes relevant on POSIX platforms. The most important of which is a C extension module used internally to handle the code path between fork() and exec(). This module is reliable when an application is using threads.

Refer to the Python 3.5 subprocess documentation for usage information.

  • Timeout support backported from Python 3.3 is included.
  • The run() API from Python 3.5 was backported in subprocess32 3.5.0.
  • Otherwise features are frozen at the 3.2 level.

Usage

The recommend pattern for cross platform code is to use the following:

if os.name == 'posix' and sys.version_info[0] < 3:
    import subprocess32 as subprocess
else:
    import subprocess

Or if you fully control your POSIX Python 2.7 installation, this can serve as a replacement for its subprocess module. Users will thank you by not filing concurrency bugs.

Got Bugs?

Try to reproduce them on the latest Python 3.x itself and file bug reports on bugs.python.org. Add gregory.p.smith to the Nosy list.

If you have can prove that the issue is specifically with this backport and not a problem in Python 3 itself, feel free to use the github issue tracker for others to collaborate with you on a fix, but do not expect any support.

Python 2 has reached EOL, as has this project.

-- Gregory P. Smith [email protected] [Google LLC]

python-subprocess32's People

Contributors

adamchainz avatar darkfeline avatar gpshead avatar joakim-hove avatar luca-vercelli avatar tlandschoff-scale avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-subprocess32's Issues

Cross-version compatibility for `subprocess.getstatusoutput`

When Windows support was added to subprocess.getstatusoutput we inadvertently made it backwards incompatible with the behaviour of the 2.7 version: https://bugs.python.org/issue22635

Now, backwards compatibility within the 3.x series means we can't readily change it back to make it match the Python 2.7 behaviour (even though the only reason it changed in the first place was due to a gap in our test coverage).

This impacts subprocess32 in two ways:

  1. You likely want to make sure you don't backport that mistake (since it would break migrations from the 2.7 stdlib to subprocess32)
  2. If subprocess32 were to support 3.x in addition to 2.7, it could provide a way for affected applications to readily work around the problem

I'm not sure how difficult the latter would be, though.

build fails on FreeBSD 11.1

One gets

    _posixsubprocess.c:764:22: error: use of undeclared identifier 'O_CLOEXEC'
        res = pipe2(fds, O_CLOEXEC);

while building the extension. The problem is that O_CLOEXEC defined in fcntl.h inside an #if, with the condition that does not hold true:

#if __POSIX_VISIBLE >= 200809
/* Defined by POSIX 1003.1-2008; BSD default, but reserve for future use. */
#define O_TTY_INIT      0x00080000      /* Restore default termios attributes */

#define O_CLOEXEC       0x00100000
#endif

To fix the build I ended up adding -DO_CLOEXEC=0x00100000 to CFLAGS, after trying without success to do this more properly.

AttributeError: 'module' object has no attribute 'Signals'

This happens with subprocess32==3.5.0:

>>> import subprocess32 as s32
>>> exc = s32.CalledProcessError(-1, "eggs")
>>> raise exc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
subprocess32.CalledProcessError: <exception str() failed>
>>> str(exc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jwilk/.local/lib/python2.7/site-packages/subprocess32.py", line 72, in __str__
    self.cmd, signal.Signals(-self.returncode))
AttributeError: 'module' object has no attribute 'Signals'

Aside from bugginess, I wonder why this code is there in the first place.
subprocess uses signal.Signals only since Python 3.6, whereas subprocess32 is supposed to be 3.2 with some 3.3 and 3.5 backports.

subprocess32 doesn't include updates from 3.7

>>> subprocess32.run(["ls", "-l"], capture_output=True)
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/vincent/.local/lib/python2.7/site-packages/subprocess32.py", line 406, in run process = Popen(*popenargs, **kwargs) TypeError: __init__() got an unexpected keyword argument 'capture_output'

This is not surprising because that's what the code does. However, this behavior has been updated in Python 3.7 (https://docs.python.org/3/library/subprocess.html#subprocess.run, "Changed in version 3.7: Added the text parameter, as a more understandable alias of universal_newlines. Added the capture_output parameter."). In the cpython repository, subprocess.py has been updated with additional parameters (see https://github.com/python/cpython/blob/master/Lib/subprocess.py#L433), that are not present in subprocess32.

While subprocess32 only supporting subprocess up to 3.5 is well documented, there's some additional updates which would be valuable to backport. Thanks!

Timeout doesn't work properly

Timeout parameter doesn't really force timeout. It's happening on both Python 3.5 and Python 2.7 + subprocess32. However it works in Python 3.9.5. Not sure if it's worth to fix it, but at least it will be documented here. :)

Tests:

import os
import sys
import unittest
from datetime import datetime, timedelta

if os.name == 'posix' and sys.version_info[0] < 3:
    import subprocess32 as subprocess
else:
    import subprocess


class TestSubprocess(unittest.TestCase):

    def test_long_sleep(self):
        start = datetime.now()
        try:
            subprocess.run(
                "sleep 10",
                timeout=1,
                shell=True,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
        except Exception as exc:
            print("We got error:", type(exc))

        delta = datetime.now() - start
        print("It took", delta.seconds, "seconds")

        self.assertLess(delta, timedelta(seconds=2))


if __name__ == '__main__':
    unittest.main()

Results

Python 3.5.10:

We got error: <class 'subprocess.TimeoutExpired'>
It took 10 seconds
F
======================================================================
FAIL: test_long_sleep (__main__.TestSubprocess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess.py", line 30, in test_long_sleep
    self.assertLess(delta, timedelta(seconds=2))
AssertionError: datetime.timedelta(0, 10, 2203) not less than datetime.timedelta(0, 2)

----------------------------------------------------------------------
Ran 1 test in 10.003s

FAILED (failures=1)

Python 2.7.18

('We got error:', <class 'subprocess32.TimeoutExpired'>)
('It took', 10, 'seconds')
F
======================================================================
FAIL: test_long_sleep (__main__.TestSubprocess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess.py", line 30, in test_long_sleep
    self.assertLess(delta, timedelta(seconds=2))
AssertionError: datetime.timedelta(0, 10, 2108) not less than datetime.timedelta(0, 2)

----------------------------------------------------------------------
Ran 1 test in 10.003s

FAILED (failures=1)

Python 3.9.5

We got error: <class 'subprocess.TimeoutExpired'>
It took 1 seconds
.
----------------------------------------------------------------------
Ran 1 test in 1.002s

OK

Missing local import: warnings

            if pass_fds and not close_fds:
                warnings.warn("pass_fds overriding close_fds.", RuntimeWarning)

Results in NameError: global name 'warnings' is not defined

How to specify dependency to subprocess32?

I have a library (subx) which depends on subprocess32.

My library needs the timeout kwarg.

I need subprocess32 only if the target platform is Python2.x.

How should I define the dependency in my project?

I get this error message, if I define a dependency to subprocess32 via "install_requires" (setup.py) and I am inside a python3 virtualenv:

===> pip install -e git+https://github.com/guettli/subx.git#egg=subx
Obtaining subx from git+https://github.com/guettli/subx.git#egg=subx
  Cloning https://github.com/guettli/subx.git to ./src/subx
Collecting subprocess32 (from subx)
  Using cached subprocess32-3.2.7.tar.gz
    Complete output from command python setup.py egg_info:
    This backport is for Python 2.x only.
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-lju3nl1y/subprocess32/

Same question here: https://stackoverflow.com/questions/47098185/dependency-management-subprocess32-needed-for-python2-7

Release a 3.5.x build.

There have been some important changes since release 3.2.7, in particular PR #13 . Are there any plans to release an up-to-date version soon? Thanks

_args_from_interpreter_flags function not present causes an import error while importing multiprocessing module

I did a drop in replacement of python system library subprocess with subprocess32 & renamed the subprocess32 to subprocess in the standard install location. When importing multiprocessing I get an import error

>>> import multiprocessing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vsnag/Python-2.7.12/Lib/multiprocessing/__init__.py", line 65, in
<module>
    from multiprocessing.util import SUBDEBUG, SUBWARNING
  File "/home/vsnag/Python-2.7.12/Lib/multiprocessing/util.py", line 41, in <mod
ule>
    from subprocess import _args_from_interpreter_flags
ImportError: cannot import name _args_from_interpreter_flags

Adding this https://github.com/python/cpython/blob/master/Lib/subprocess.py#L536 could be a possible fix.

build fails on archlinux

(coma) ➜  coma git:(master) ✗ pip install subprocess32
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting subprocess32
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/32/c8/564be4d12629b912ea431f1a50eb8b3b9d00f1a0b1ceff17f266be190007/subprocess32-3.5.4.tar.gz
Building wheels for collected packages: subprocess32
  Building wheel for subprocess32 (setup.py) ... error
  ERROR: Complete output from command ${HOME}/.conda/envs/coma/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-6Mmw2Y/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-hJ231z --python-tag cp27:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  copying subprocess32.py -> build/lib.linux-x86_64-2.7
  running build_ext
  running build_configure
  checking for gcc... gcc
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C compiler... yes
  checking whether gcc accepts -g... yes
  checking for gcc option to accept ISO C89... none needed
  checking how to run the C preprocessor... gcc -E
  checking for grep that handles long lines and -e... /usr/bin/grep
  checking for egrep... /usr/bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking for unistd.h... (cached) yes
  checking fcntl.h usability... yes
  checking fcntl.h presence... yes
  checking for fcntl.h... yes
  checking signal.h usability... yes
  checking signal.h presence... yes
  checking for signal.h... yes
  checking sys/cdefs.h usability... yes
  checking sys/cdefs.h presence... yes
  checking for sys/cdefs.h... yes
  checking for sys/types.h... (cached) yes
  checking for sys/stat.h... (cached) yes
  checking sys/syscall.h usability... yes
  checking sys/syscall.h presence... yes
  checking for sys/syscall.h... yes
  checking for dirent.h that defines DIR... yes
  checking for library containing opendir... none required
  checking for pipe2... yes
  checking for setsid... yes
  checking whether dirfd is declared... yes
  configure: creating ./config.status
  config.status: creating _posixsubprocess_config.h
  building '_posixsubprocess32' extension
  creating build/temp.linux-x86_64-2.7
  gcc -pthread -B ${HOME}/.conda/envs/coma/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I${HOME}/.conda/envs/coma/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
  In file included from ${HOME}/.conda/envs/coma/include/python2.7/Python.h:8,
                   from _posixsubprocess.c:16:
  ${HOME}/.conda/envs/coma/include/python2.7/pyconfig.h:1220: warning: "_POSIX_C_SOURCE" redefined
   1220 | #define _POSIX_C_SOURCE 200112L
        |
  In file included from /usr/include/sys/cdefs.h:23,
                   from _posixsubprocess.c:12:
  /usr/include/features.h:266: note: this is the location of the previous definition
    266 | # define _POSIX_C_SOURCE 200809L
        |
  In file included from ${HOME}/.conda/envs/coma/include/python2.7/Python.h:8,
                   from _posixsubprocess.c:16:
  ${HOME}/.conda/envs/coma/include/python2.7/pyconfig.h:1242: warning: "_XOPEN_SOURCE" redefined
   1242 | #define _XOPEN_SOURCE 600
        |
  In file included from /usr/include/sys/cdefs.h:23,
                   from _posixsubprocess.c:12:
  /usr/include/features.h:203: note: this is the location of the previous definition
    203 | # define _XOPEN_SOURCE 700
        |
  gcc -pthread -shared -B ${HOME}/.conda/envs/coma/compiler_compat -L${HOME}/.conda/envs/coma/lib -Wl,-rpath=${HOME}/.conda/envs/coma/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/_posixsubprocess.o -L${HOME}/.conda/envs/coma/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/_posixsubprocess32.so
  ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
  ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
  ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
  ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
  build/temp.linux-x86_64-2.7/_posixsubprocess.o: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for subprocess32
  Running setup.py clean for subprocess32
Failed to build subprocess32
Installing collected packages: subprocess32
  Running setup.py install for subprocess32 ... error
    ERROR: Complete output from command ${HOME}/.conda/envs/coma/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-6Mmw2Y/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-Zv3qW2/install-record.txt --single-version-externally-managed --compile:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying subprocess32.py -> build/lib.linux-x86_64-2.7
    running build_ext
    running build_configure
     _posixsubprocess_config.h is already up to date.
    building '_posixsubprocess32' extension
    creating build/temp.linux-x86_64-2.7
    gcc -pthread -B ${HOME}/.conda/envs/coma/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I${HOME}/.conda/envs/coma/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
    In file included from ${HOME}/.conda/envs/coma/include/python2.7/Python.h:8,
                     from _posixsubprocess.c:16:
    ${HOME}/.conda/envs/coma/include/python2.7/pyconfig.h:1220: warning: "_POSIX_C_SOURCE" redefined
     1220 | #define _POSIX_C_SOURCE 200112L
          |
    In file included from /usr/include/sys/cdefs.h:23,
                     from _posixsubprocess.c:12:
    /usr/include/features.h:266: note: this is the location of the previous definition
      266 | # define _POSIX_C_SOURCE 200809L
          |
    In file included from ${HOME}/.conda/envs/coma/include/python2.7/Python.h:8,
                     from _posixsubprocess.c:16:
    ${HOME}/.conda/envs/coma/include/python2.7/pyconfig.h:1242: warning: "_XOPEN_SOURCE" redefined
     1242 | #define _XOPEN_SOURCE 600
          |
    In file included from /usr/include/sys/cdefs.h:23,
                     from _posixsubprocess.c:12:
    /usr/include/features.h:203: note: this is the location of the previous definition
      203 | # define _XOPEN_SOURCE 700
          |
    gcc -pthread -shared -B ${HOME}/.conda/envs/coma/compiler_compat -L${HOME}/.conda/envs/coma/lib -Wl,-rpath=${HOME}/.conda/envs/coma/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/_posixsubprocess.o -L${HOME}/.conda/envs/coma/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/_posixsubprocess32.so
    ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
    ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
    ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
    ${HOME}/.conda/envs/coma/compiler_compat/ld: build/temp.linux-x86_64-2.7/_posixsubprocess.o: unable to initialize decompress status for section .debug_info
    build/temp.linux-x86_64-2.7/_posixsubprocess.o: file not recognized: file format not recognized
    collect2: error: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "${HOME}/.conda/envs/coma/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-6Mmw2Y/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-Zv3qW2/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-6Mmw2Y/subprocess32/

Hi, I encountered build error on archlinux, possibly because of my gcc and ld version.

(coma) ➜  coma git:(master) ✗ gcc --version
gcc (GCC) 9.1.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(coma) ➜  coma git:(master) ✗ ld --version
GNU ld (GNU Binutils) 2.32
Copyright (C) 2019 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

Different behaviour when running subprocess in python 2.7.12 and python 3.6.9

Hi,

I am trying to execute a process from python, specifically CASA NRAO 5.7 which is a Python 2.7.12 environment plus some radio astronomical functions. The thing is that when I run my process using subprocess whether in just Python 2.7.12 or CASA I get a std::bad_alloc error, and that does not happen in Python 3.6.9. I have installed subprocess32 to see if this behaviour continued in Python 2.7 and yes, it continues. I don't know what change could be done to the subprocess library that make my program work in Python 3.6.9.

I am basically running this code:


import numpy as np
import os
if os.name == 'posix' and sys.version_info[0] < 3:
    import subprocess32 as subprocess
else:
    import subprocess
import shlex

args = "/home/miguel/Documents/gpuvmem/bin/gpuvmem -X 16 -Y 16 -V 256 -i A1314.ms.selfcal -o selfcal_data/output_residuals.ms -z 0.001,3.5 -Z 0.008,0.0,0.0 -G 1 -m output_ph3.image.fits -O selfcal_data/output.fits -I input_example.dat -R 2.0 -t 100000 -g 20 --verbose --savemodel-input"

args = shlex.split(args)
p2 = subprocess.Popen(args, bufsize=-1, shell=False)
p2.wait()

Best regards

Heap Overflow in _PySequence_BytesToCharpArray

_PySequence_BytesToCharpArray contains a heap overflow after a integer wraparound occurs when large sequences are passed to subprocess_fork_exec.

This can be reproduced on a 32-bit system. It probably can be reproduced on 64bit systems with a few exabytes of RAM. (I don't have anything like that to test with unfortunately...)

import subprocess32 as subprocess
subprocess._posixsubprocess.fork_exec("a",'a'*0x3FFFFFFF,"a",set(),"a","a",1,1,1,1,1,1,1,3,1,1,"a")
*** glibc detected *** /usr/bin/python: double free or corruption (!prev): 0x09464b38 ***
Segmentation fault (core dumped)

It can be seen in _posixsubprocess_helpers.c on line 119:

    argc = PySequence_Size(self);
    if (argc == -1)
        return NULL;

array = malloc((argc + 1) * sizeof(char *));

argc is the size of the sequence passed to the C library. Thus, if we can pass a value such that size + 1 * 4 > 0xFFFFFFFF (0x3fffffff), malloc should return a small value, and invalid heap writes will occur in the loop below.

I was unable to trigger this through the actual unhidden API interface, so this flaw seems low priority, but it's always possible someone else could figure out a better way to trigger the overflow. Trying to generate a valid args list that large doesn't seem possible on my system. Guessing the list overhead is too much to make this work. Passing string sequences work, but the API converts them to a list before sending them to the C library I believe.

Consider uploading wheels for OSX / Linux?

Over at matplotlib/matplotlib#6945 a user found that they needed to install XCode only for subprocess32, in an install of the matplotlib plotting package. All the other dependencies are either pure Python, or have wheels for OSX / Linux (e.g. numpy).

Would you consider uploading wheels for subprocess32 ? That would be very helpful.

As proof of concept, I made a wheel-building repo at:

https://github.com/MacPython/subprocess32-wheels

This already builds a functional OSX wheel for 3.2.7:

https://travis-ci.org/MacPython/subprocess32-wheels/jobs/152523378

Manylinux wheels get broken by #12 - does eae9e95 fix that problem for manylinux wheels? Tests seem to pass on Ubuntu 14.04 and a wheel from current master:

https://travis-ci.org/MacPython/subprocess32-wheels/jobs/152529549

I'm very happy to add y'all as owners of the build repo.

Use of subprocess32 on MS Windows

I'm trying to use the latest version of subprocess32 (3.2.5) on a cross 
platform project and have encountered some problems using the subprocess32 
module on Windows.

OS = Windows 7
Python = 2.7.6

Issue 1: Using the setup.py intsaller fails on Windows because of the posix 
extension modules. Removing the line 
ext_modules=[ext],
from setup.py allows the installation to proceed for windows systems.

Issue 2: The subprocess32 code assumes that _subprocess.c has an attribute 
WAIT_TIMEOUT which isn't the case for Python 2.7 (line 1137 of subprocess32.py)



Nick




Original issue reported on code.google.com by [email protected] on 24 Mar 2014 at 3:40

error: package directory 'python3_redirect' does not exist

When testing https://github.com/google/python-subprocess32/blob/master/setup.py#L20 promised ability to install under Python3.x

$ pip install subprocess32==3.5.0rc1
Collecting subprocess32==3.5.0rc1
  Downloading https://files.pythonhosted.org/packages/28/91/d1283618eba07c4e8e18c58b3fd8b5ff3a8992fb652a3720535ddf2f2916/subprocess32-3.5.0rc1.tar.gz (54kB)
    100% |████████████████████████████████| 61kB 843kB/s 
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/subprocess32.egg-info
    writing top-level names to pip-egg-info/subprocess32.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/subprocess32.egg-info/dependency_links.txt
    writing pip-egg-info/subprocess32.egg-info/PKG-INFO
    writing manifest file 'pip-egg-info/subprocess32.egg-info/SOURCES.txt'
    /home/michael/cwltool/env3/lib/python3.5/site-packages/setuptools/version.py:1: UserWarning: Module _posixsubprocess was already imported from None, but /tmp/pip-install-gk2ke0g4/subprocess32 is being added to sys.path
      import pkg_resources
    subprocess32 == subprocess on Python 3.
    error: package directory 'python3_redirect' does not exist
    
    ----------------------------------------

Not a full replacement for subprocess of Python 2.7

Hey there,

thanks for making subprocess32 available, it really helped us fighting random lock-ups in our application. I would not have dared to backport subprocess from Python 3 and resorted to workarounds.

After issues with mixups between subprocess32 and subprocess (some libraries still use subprocess and know nothing about subprocess32) we decided to heed this advice:

Or if you fully control your POSIX Python 2.7 installation, this can serve as a replacement for its subprocess module. Users will thank you by not filing concurrency bugs.

So far this looked good. Out of curiousity I ran the Python 2.7 stdlib test suite against subprocess32 and was surprised that it failed.

I offer this Dockerfile to reproduce this:

$ cat Dockerfile 
FROM debian:9

RUN apt-get update
RUN apt-get install -y python2.7 libpython2.7-testsuite python-pip

# Install subprocess32, replace subprocess
RUN pip install subprocess32
RUN cp /usr/local/lib/python2.7/dist-packages/subprocess32.py /usr/lib/python2.7/subprocess.py
RUN cp /usr/local/lib/python2.7/dist-packages/_posixsubprocess32.so /usr/lib/python2.7/lib-dynload/_posixsubprocess32.so

# Run regrtest
RUN python -m test.regrtest --verbose test_subprocess

This results into these failures:

$ sudo docker build .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM debian:9
 ---> de8b49d4b0b3
Step 2/7 : RUN apt-get update
 ---> Using cache
 ---> 21f3d4930519
[...]
Step 7/7 : RUN python -m test.regrtest --verbose test_subprocess
 ---> Running in 58064776ae73
== CPython 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]
==   Linux-4.15.0-47-generic-x86_64-with-debian-9.6 little-endian
==   /tmp/test_python_7
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
[1/1] test_subprocess
test_call_kwargs (test.test_subprocess.ProcessTestCase) ... ok
[...]
test test_subprocess failed -- multiple errors occurred
skipped 'mswindows only'

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

======================================================================
ERROR: test_exceptions (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 803, in test_exceptions
    self.assertIn("os.chdir", c.exception.child_traceback)
AttributeError: 'exceptions.OSError' object has no attribute 'child_traceback'

======================================================================
ERROR: test_preexec_errpipe_does_not_double_close_pipes (test.test_subprocess.POSIXProcessTestCase)
Issue16140: Don't double close pipes on preexec error.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 868, in test_preexec_errpipe_does_not_double_close_pipes
    stderr=subprocess.PIPE, preexec_fn=raise_it)
  File "/usr/lib/python2.7/test/test_subprocess.py", line 827, in __init__
    subprocess.Popen.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 614, in __init__
    restore_signals, start_new_session)
TypeError: _execute_child() takes exactly 18 arguments (20 given)

======================================================================
ERROR: test_communicate_epipe (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 706, in test_communicate_epipe
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 724, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1538, in _communicate
    orig_timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1684, in _communicate_with_select
    bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 146 tests in 16.424s

Some tests are bound to fail given that they access the internals of the Python 2.7 implementation. But these look relevant:

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 706, in test_communicate_epipe
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 724, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1538, in _communicate
    orig_timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1684, in _communicate_with_select
    bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

I intent to find out the cause and fix this, but for reference I'd like to have this issue public in the mean time.

BTW: There is no mention of epipe in the test_subprocess32.py tests:

~/workspace/python-subprocess32$ grep -i epipe test_subprocess32.py 

Greetings, Torsten

test_empty_env fails w/ Gentoo sandbox

Gentoo sandbox enforces itself into child processes, and therefore the environment is not empty:

======================================================================
FAIL: test_empty_env (__main__.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess32.py", line 545, in test_empty_env
    self.assertEqual(stdout.strip(), "[]")
AssertionError: "['LD_PRELOAD', 'SANDBOX_READ', 'SANDBOX_MESSAGE_P@TH', 'SANDBOX_ON', 'SANDBOX_DENY', 'SANDBOX_VERBOSE', 'SANDBOX_LOG', 'SANDBOX_PREDICT', 'SANDBOX_DEBUG_LOG', 'SANDBOX_ACTIVE', 'SANDBOX_WRITE']" != '[]'

======================================================================
FAIL: test_empty_env (__main__.ProcessTestCasePOSIXPurePython)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess32.py", line 545, in test_empty_env
    self.assertEqual(stdout.strip(), "[]")
AssertionError: "['LD_PRELOAD', 'SANDBOX_READ', 'SANDBOX_MESSAGE_P@TH', 'SANDBOX_ON', 'SANDBOX_DENY', 'SANDBOX_VERBOSE', 'SANDBOX_LOG', 'SANDBOX_PREDICT', 'SANDBOX_DEBUG_LOG', 'SANDBOX_ACTIVE', 'SANDBOX_WRITE']" != '[]'

======================================================================
FAIL: test_empty_env (__main__.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess32.py", line 545, in test_empty_env
    self.assertEqual(stdout.strip(), "[]")
AssertionError: "['LD_PRELOAD', 'SANDBOX_READ', 'SANDBOX_MESSAGE_P@TH', 'SANDBOX_ON', 'SANDBOX_DENY', 'SANDBOX_VERBOSE', 'SANDBOX_LOG', 'SANDBOX_PREDICT', 'SANDBOX_DEBUG_LOG', 'SANDBOX_ACTIVE', 'SANDBOX_WRITE']" != '[]'

----------------------------------------------------------------------
Ran 264 tests in 197.918s

FAILED (failures=3)

on Windows 10, TimeoutExpired exception gets TypeError

Run into an issue, see below, when using check_output and timeout. This is on Windows 10, and understand that "This code has not been tested on Windows or other non-POSIX platforms.", but thought you might want to know.

C:\Users\djea>python -V
Python 2.7.10

C:\Users\djea>pip show subprocess32

---
Metadata-Version: 1.1
Name: subprocess32
Version: 3.2.7
Summary: A backport of the subprocess module from Python 3.2/3.3 for use on 2.x.
Home-page: https://github.com/google/python-subprocess32
Author: Gregory P. Smith
Author-email: [email protected]
License: PSF license
Location: c:\python27\lib\site-packages
Requires:

When running this code,

import subprocess32
try:
    p = subprocess32.check_output("timeout 5",shell=True,timeout=3)
    print p
except subprocess32.TimeoutExpired:
    print "expired"

Will get this Error

Traceback (most recent call last):
  File "C:\Users\djea\timeout.py", line 5, in <module>
    p = subprocess32.check_output("timeout 5",shell=True,timeout=3)
  File "C:\Python27\lib\site-packages\subprocess32.py", line 631, in check_output
    output, unused_err = process.communicate(timeout=timeout)
  File "C:\Python27\lib\site-packages\subprocess32.py", line 927, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Python27\lib\site-packages\subprocess32.py", line 1190, in _communicate
    raise TimeoutExpired(self.args)
TypeError: __init__() takes at least 3 arguments (2 given)

Same code (except using 'sleep' instead of 'timeout') has no problem on Linux.

File descriptors given to pass_fds should have close-on-exec flag unset

Here's a script to reproduce the issue:

import os
import sys

if os.name == 'posix' and sys.version_info[0] < 3:
    import subprocess32 as subprocess
else:
    import subprocess

p1 = subprocess.Popen(['echo', 'hi'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['echo', 'bye'], stdout=subprocess.PIPE)

# uncommenting these lines will produce expected result in python2.7
# subprocess._set_cloexec(p1.stdout.fileno(), False)
# subprocess._set_cloexec(p2.stdout.fileno(), False)

subs = [p1, p2]

fds = [i.stdout.fileno() for i in subs]
args = ['/dev/fd/{}'.format(fd) for fd in fds]

p2 = subprocess.Popen(
    ['cat'] + args,
    pass_fds=fds
)

python 3.5 output:

hi
bye

python 2.7 output:

cat: /dev/fd/3: No such file or directory
cat: /dev/fd/4: No such file or directory

My C isn't very good, but I traced through the _posixsubprocess.c and fileutils.c source on the 2.7 tag of cpython sources on GitHub, and it looks like it should be calling ioctl(fd, FIONCLEX, NULL) for each fd in pass_fds, so I haven't been able to figure out why this isn't working in python 2.7.

Can the setup routine please fail silently?

Otherwise continuous integration with this thing included needs an unnecessary complex way of discerning 2.7 & 3.x dependencies - i.e. I don't want to have a seperate requirements.txt for 2.x.. Or is there something i missed?

universal_newlines=True behaves differently on 3.5 and 2.7

$ pip install -U subprocess32
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /home/graingert/.cache/pip/wheels/68/39/1a/5e402bdfdf004af1786c8b853fd92f8c4a04f22aad179654d1/subprocess32-3.5.4-cp27-cp27mu-linux_x86_64.whl
Installing collected packages: subprocess32
Successfully installed subprocess32-3.5.4
$ python2
Python 2.7.16 (default, Oct  7 2019, 17:36:04) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess32 as subprocess
>>> isinstance(subprocess.run("ls", universal_newlines=True, stdout=subprocess.PIPE).stdout, bytes)
True
$ python3.5
Python 3.5.8 (default, Oct 31 2019, 00:57:08) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> isinstance(subprocess.run("ls", universal_newlines=True, stdout=subprocess.PIPE).stdout, bytes)
False

build fails on Solaris 11

The dirfd() workaround for older versions does not work here (as it's d_fd, rather than dd_fd), and is in fact not needed, as Solaris 11 provides dirfd().

    building '_posixsubprocess' extension
    creating build/temp.solaris-2.11-sun4v.64bit-2.7
    gcc -DNDEBUG -g -O3 -Wall -D_XPG6 -fPIC -I/datapool/dima/Sage/sagetrac-mirror/local/include/python2.7 -c _posixsubprocess.c -o build/temp.solaris-2.11-sun4v.64bit-2.7/_posixsubprocess.o
    _posixsubprocess.c: In function ‘_close_open_fds_maybe_unsafe’:
    _posixsubprocess.c:334:50: error: ‘DIR {aka struct <anonymous>}’ has no member named ‘dd_fd’; did you mean ‘d_fd’?
             int fd_used_by_opendir = dirfd(proc_fd_dir);
                                                      ^~~
                                                      d_fd
    error: command 'gcc' failed with exit status 1

If I export CFLAGS with CFLAGS="$CFLAGS -DHAVE_DIRFD" then the build works.

I am not sure what the correct way to fix this is. HAVE_DIRFD needs to be set somehow, I guess.

Error during pip install on ubuntu

Subprocess32 is a dependency for the fbprophet library.

While pip installing the fbprophet module I received the following error:

Command "/home/ubuntu/anaconda3/envs/resfuture/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-DWKA4l/subprocess32/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-zcj5OE-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-DWKA4l/subprocess32/

When I tried to pip install just subprocess I encountered the same error:
Failed building wheel for subprocess32
Command "/home/ubuntu/anaconda3/envs/resfuture/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-paHlZS/subprocess32/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mEbekx-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-paHlZS/subprocess32/

Does anyone know how I can resolve this issue?

Issue with _posixsubprocess when importing subprocess32

I posted this issue here but as Gregory pointed out it is more suited to here, if at all. https://bugs.python.org/issue31383

I am trying to install subprocess32 with my python 2.7 installation via buildroot. It appeared to install correctly but when I import it on the embedded system I get an error:

>>> import subprocess32
/usr/lib/python2.7/site-packages/subprocess32.py:472: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads. 
"program uses threads.", RuntimeWarning)

Following this path I tried to import _posixsubprocess

import _posixsubprocess
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define init function (init_posixsubprocess)

I can use subprocess32 otherwise, tested out basic functionality like subprocess32.Popen(["ls"]).communicate()

As was pointed out on bugs.python.org init_posixsubprocess does exist in _posixsubprocess.c

This may be an issue with buildroot handling it, however I will post a solution if I find it here to help others.

What I see when I build it seems innocuous:

^[[7m>>> python-subprocess32 3.2.7 Extracting^[[27m
^[[7m>>> python-subprocess32 3.2.7 Patching package/python-subprocess32^[[27m
^[[7m>>> python-subprocess32 3.2.7 Configuring^[[27m
^[[7m>>> python-subprocess32 3.2.7 Building^[[27m
(cd /media/vmpart/5.1.0_x86/buildroot/output/build/python-subprocess32-3.2.7; /media/vmpart/5.1.0_x86/buildroot/output/host/usr/bin/python setup.py build)
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying subprocess32.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_posixsubprocess' extension
creating build/temp.linux-x86_64-2.7
copying subprocess32.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_posixsubprocess' extension
creating build/temp.linux-x86_64-2.7
/media/vmpart/5.1.0_x86/buildroot/output/host/usr/bin/ccache /usr/bin/gcc -pthread -fno-strict-aliasing -O2 -I/media/vmpart/5.1.0_x86/buildroot/output/host/include -I/media/vmpart/5.1.0_x86/buildroot/output/host/usr/include -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/media/vmpart/5.1.0_x86/buildroot/output/host/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
/media/vmpart/5.1.0_x86/buildroot/output/host/usr/bin/ccache /usr/bin/gcc -pthread -shared -L/media/vmpart/5.1.0_x86/buildroot/output/host/lib -L/media/vmpart/5.1.0_x86/buildroot/output/host/usr/lib -Wl,-rpath,/media/vmpart/5.1.0_x86/buildroot/output/host/usr/lib build/temp.linux-x86_64-2.7/_posixsubprocess.o -L/media/vmpart/5.1.0_x86/buildroot/output/host/usr/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/_posixsubprocess.so
^[[7m>>> python-subprocess32 3.2.7 Installing to target^[[27m
(cd /media/vmpart/5.1.0_x86/buildroot/output/build/python-subprocess32-3.2.7; /media/vmpart/5.1.0_x86/buildroot/output/host/usr/bin/python setup.py install --prefix=/media/vmpart/5.1.0_x86/buildroot/output/target/usr)
running install
running build
running build_py
running build_ext
running install_lib
copying build/lib.linux-x86_64-2.7/_posixsubprocess.so -> /media/vmpart/5.1.0_x86/buildroot/output/target/usr/lib/python2.7/site-packages
running install_egg_info
Removing /media/vmpart/5.1.0_x86/buildroot/output/target/usr/lib/python2.7/site-packages/subprocess32-3.2.7-py2.7.egg-info
Writing /media/vmpart/5.1.0_x86/buildroot/output/target/usr/lib/python2.7/site-packages/subprocess32-3.2.7-py2.7.egg-info


Make file (buildroot version does not have python-package):

#############################################################
#
# Subprocess32 module for python
#
#############################################################

PYTHON_SUBPROCESS32_VERSION = 3.2.7
PYTHON_SUBPROCESS32_SOURCE = subprocess32-$(PYTHON_SUBPROCESS32_VERSION).tar.gz
PYTHON_SUBPROCESS32_SITE = https://pypi.python.org/pypi/subprocess32
PYTHON_SUBPROCESS32_DEPENDENCIES = python

define PYTHON_SUBPROCESS32_BUILD_CMDS
        (cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
endef

define PYTHON_SUBPROCESS32_INSTALL_TARGET_CMDS
        (cd $(@D); $(HOST_DIR)/usr/bin/python setup.py install --prefix=$(TARGET_DIR)/usr)
endef

$(eval $(call GENTARGETS,package,python-subprocess32))

Does not build on Windows with GCC on Cygwin

I try to build it, but the compiler hang on this screen:

python setup.py --verbose build
running build
running build_py
not copying subprocess32.py (output up-to-date)
running build_ext
building '_posixsubprocess' extension
gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python2-2.7.13-1.x86_64/build=/usr/src/debug/python2-2.7.13-1 -fdebug-prefix-map=/usr/src/ports/python/python2-2.7.13-1.x86_64/src/Python-2.7.13=/usr/src/debug/python2-2.7.13-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.cygwin-2.8.0-x86_64-2.7/_posixsubprocess.o
$ gcc --version
gcc (GCC) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ python --version
Python 2.7.13

It also hangs only using:

$ gcc -c _posixsubprocess.c -o build/temp.cygwin-2.8.0-x86_64-2.7/_posixsubprocess.o

subprocess32 fails to compile on ancient RHEL/CentOS 5 & manylinux1 wheels - O_CLOEXEC and HAVE_PIPE2 related?

OS:RHEL 5.11
PYTHON:2.7.11/12

When i use pip or python install to install this extension:
pip install subprocess32-3.2.7.tar.gz

The error messages:

Processing ./subprocess32-3.2.7.tar.gz
Installing collected packages: subprocess32
Running setup.py install for subprocess32 ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-ekkMER-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-EebZzY-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying subprocess32.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_posixsubprocess' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/python-2.7/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
_posixsubprocess.c: In function 'subprocess_cloexec_pipe':
_posixsubprocess.c:764: warning: implicit declaration of function 'pipe2'
_posixsubprocess.c:764: error: 'O_CLOEXEC' undeclared (first use in this function)
_posixsubprocess.c:764: error: (Each undeclared identifier is reported only once
_posixsubprocess.c:764: error: for each function it appears in.)
error: command 'gcc' failed with exit status 1

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-ekkMER-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-EebZzY-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-ekkMER-build/

subprocess.run is not backported to python 3.4

Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess32
>>> subprocess32.run
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'run'
>>> subprocess32.__file__
'C:\\Program Files (x86)\\Python34\\lib\\subprocess.py'

C:\Users\Owner\AppData\Roaming\Python\Python34\site-packages\subprocess32_init_.py

"""When installed as subprocess32, this sets up a redirect to subprocess."""

import subprocess
import sys
if sys.version_info[:2] < (3,3):
    raise ImportError('Ancient Python 3 versions are not supported.')
# Doing this could crash some older Python interpreters due to the module
# reference going away before the import is complete?
sys.modules['subprocess32'] = subprocess

Python 2 works fine. I guess something needs to be fixed in the build script?

else: # PY3
# Install a redirect that makes subprocess32 == subprocess on import.
packages.append('subprocess32')
package_dir['subprocess32'] = 'python3_redirect'
sys.stderr.write('subprocess32 == subprocess on Python 3.\n')

TimeoutExpired is not pickleable.

In Python 2.x, subclasses of Exception must either call super(...).__init__(*args) or set self.args in order to be successfully pickled/unpickled.

Example:

>>> import subprocess32, pickle
>>> t = subprocess32.TimeoutExpired(['foo', 'bar'], 60)
>>> pickle.loads(pickle.dumps(t))
TypeError                                 Traceback (most recent call last)
<ipython-input-6-c105e6dd98fb> in <module>()
----> 1 pickle.loads(pickle.dumps(t))

/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in loads(str)
   1386 def loads(str):
   1387     file = StringIO(str)
-> 1388     return Unpickler(file).load()
   1389
   1390 # Doctest

/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in load(self)
    862             while 1:
    863                 key = read(1)
--> 864                 dispatch[key](self)
    865         except _Stop, stopinst:
    866             return stopinst.value

/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in load_reduce(self)
   1137         args = stack.pop()
   1138         func = stack[-1]
-> 1139         value = func(*args)
   1140         stack[-1] = value
   1141     dispatch[REDUCE] = load_reduce

TypeError: __init__() takes at least 3 arguments (1 given)

Python bug: https://bugs.python.org/issue1692335
Explanation: https://stackoverflow.com/questions/41808912/cannot-unpickle-exception-subclass

Example Solution:

>>> import subprocess32, pickle
>>> t = subprocess32.TimeoutExpired(['foo', 'bar'], 60)
>>> pickle.loads(pickle.dumps(t))
>>> t.args = (t.cmd, t.timeout, t.stdout, t.stderr)  # Magic!
>>> pickle.loads(pickle.dumps(t))
# No exception now! 

run() command

Hi,

First, thank you for this port.
I use subprocess32 quite alot, and recently discovered that Python 3.5 actually has a new function named run() which is somewhat more advanced. Would it be possible to add it to subprocess32?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.