Six - a Python 2 and 3 compatibility library

Some time in the last 12 hours, our build system using gsutil started failing and it appears to be related to six 1.11.0, which was just released.

$ gsutil
...
 File "myproject/venv/lib/python2.7/site-packages/apitools/base/protorpclite/messages.py", line 1168, in <module>
    class Field(six.with_metaclass(_FieldMeta, object)):
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

gsutil declares a dependency on six>=1.9.0 (ugh) so it pulls in the just released 1.11.0. Rolling back to six 1.10.0 and gsutil runs alright. I'm not really involved in Python or gsutil, so maybe gsutil needs to fix something, no idea, but it seems like a breakage that's not called out in the release notes or maybe something to fix.

cc @jdemeyer @benjaminp Possibly introduced in https://github.com/benjaminp/six/pull/191

We don't yet know if 3.10 or 4.0 will follow Python 3.9, but it will probably happen in 2020 when Python 3.9 reaches beta and as six is vendored and used in many codebases, it would be good for six to be ready for both.

There's several places in the codebase which essentially do:

if sys.version_info[0] == 3:
   pass  # Python 3+ stuff
else:
   pass  # Python 2 stuff

When run on Python 4, this will run the Python 2 code!

This PR flips it around:

if sys.version_info[0] == 2:
   pass  # Python 2 stuff
else:
   pass  # Python 3+ stuff

Thanks to @asottile for flake8-2020.

As an aside, there's a slight inconsistency here:

# Useful for very coarse version differentiation.
PY2 = sys.version_info[0] == 2
PY3 = sys.version_info[0] == 3
PY34 = sys.version_info[0:2] >= (3, 4)

PY34 is true on Python 4, but PY3 is not. Would it be too big a break to change it?

-PY3 = sys.version_info[0] == 3
+PY3 = sys.version_info[0] >= 3

Originally reported by: Andrej Antonov (Bitbucket: polymorphm, GitHub: polymorphm)

good day!

variable six.PY3 -- is time bomb :)

file six.py now has code:

# True if we are running on Python 3.
PY3 = sys.version_info[0] == 3

...but must be somewhere like:

# True if we are running on Python version at least 3.
PY3 = sys.version_info[0] >= 3

thanks in advance! (and sorry for my bad english :))

• Bitbucket: https://bitbucket.org/gutworth/six/issue/22

• Summary add ensure_binary, ensure_str, ensure_text helper functions to solve the encoding compatibility between py2 and py3 more easily. -- ensure_binary, input --> six.binary_type -- ensure_str, input -> six.string_type -- ensure_text, input -> six.text_type This is mentioned in Lisa Guo's keynote in Pycon 2017 (from 31:21 in https://www.youtube.com/watch?v=66XoCk79kjM), which helped Instagram to upgrade to Python 3 and I hope this can help other developers, too

• Test Plan add new unit tests to cover added code

 195 passed, 2 skipped in 0.48 seconds 

Originally reported by: Joshua Harlow (Bitbucket: harlowja, GitHub: harlowja)

When running a piece of threaded code that directly uses six.moves.$XYZ:

For example:

https://github.com/openstack/taskflow/blob/master/taskflow/examples/jobboard_produce_consume_colors.py

I am periodically able to trigger an unusually occurrence with-in six (it doesn't occur on every run, but once every ~20 runs of the above example).

#!python

Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "taskflow/examples/jobboard_produce_consume_colors.py", line 140, in producer
    for i in six.moves.xrange(0, PRODUCER_UNITS):
  File "/home/josh/Dev/taskflow/.tox/py27/lib/python2.7/site-packages/six.py", line 90, in __get__
    delattr(obj.__class__, self.name)
AttributeError: xrange

Is it possible for the six.moves.xrange attribute to disappear somehow when running from daemon threads (although those daemon threads are joined on by the parent). Perhaps there is some underlying issue with the above six code?

This is with six==1.7.3 (if that matters).

Full environment @ http://paste.ubuntu.com/8495305/

• Bitbucket: https://bitbucket.org/gutworth/six/issue/98

Originally reported by: Anselm Kruis (Bitbucket: akruis, GitHub: akruis)

The python2.7 library module pickle contains the function whichmodule(func, funcname). This function eventually inspects every value in sys.modules, triggering imports, that might fail.

This way importing six in one module can break pickling in a completely unrelated module.

An example trace-back on Linux:

  File "/scr/fg2ef/kruis/slp276/lib/python2.7/pickle.py", line 854, in whichmodule
    if name != '__main__' and getattr(module, funcname, None) is func:
  File "/scr/fg2ef/kruis/slp276_six/lib/python2.7/site-packages/six.py", line 116, in __getattr__
    _module = self._resolve()
  File "/scr/fg2ef/kruis/slp276_six/lib/python2.7/site-packages/six.py", line 105, in _resolve
    return _import_module(self.mod)
  File "/scr/fg2ef/kruis/slp276_six/lib/python2.7/site-packages/six.py", line 76, in _import_module
    __import__(name)
ImportError: No module named _winreg

Code of function whichmodule:

def whichmodule(func, funcname):
    """Figure out the module in which a function occurs.

    Search sys.modules for the module.
    Cache in classmap.
    Return a module name.
    If the function cannot be found, return "__main__".
    """
    # Python functions should always get an __module__ from their globals.
    mod = getattr(func, "__module__", None)
    if mod is not None:
        return mod
    if func in classmap:
        return classmap[func]

    for name, module in sys.modules.items():
        if module is None:
            continue # skip dummy package entries
        if name != '__main__' and getattr(module, funcname, None) is func:
            break
    else:
        name = '__main__'
    classmap[func] = name
    return name

Similar code exists in the C-function whichmodule of the extension module Modules/cPickle.c. Therefore it is probably impossible to fix this problem by monkey-patching the Python library.

Idea for a fix: don't populate sys.modules during the import of six itself, but use a sys.meta_path finder (see PEP302) to add six-modules on demand.

• Bitbucket: https://bitbucket.org/gutworth/six/issue/63

Originally reported by: Claude Paroz (Bitbucket: claudep, GitHub: claudep)

Hi,

We're planning to embed six in Django as the blessed Python 3 compatibility layer. One thing we're missing is a function that returns a byte string on Python 2 and Unicode in Python 3. This is typically useful when providing arguments for strftime() or type(). Our current implementation is:

• Python 2: n = lambda n: n.encode('utf-8')
• Python 3 (noop): n = lambda n: n

We could of course provide it in some of our own modules, but it would be nice to have it in six, unless you suggest a better replacement.

• Bitbucket: https://bitbucket.org/gutworth/six/issue/12

It seems as though it would be helpful to modify the pickle.dump(s) methods to use protocol=HIGHEST_PROTOCOL, as this would be consistent with Python3 and should be functional in Python2.6+ (stated functionality of six).

The default Python2 protocol is 0, which require __getstate__ and __setstate__ methods; however Python 3 uses protocol=HIGEST_PROTCOL (equivalent to 4 since Python 3.4).

Manipulating the dump(s) methods to use the highest protocol would allow someone to write the most "slim" code that will work in both Python2 and Python3.

Originally reported by: Eklavya Sharma (Bitbucket: ekujupr, GitHub: Unknown)

I'm fed up of doing

try:
    import collections.abc as collections_abc # only works on python 3.3+
except ImportError:
    import collections as collections_abc

A six.moves.collections.abc would be very convenient.

• Bitbucket: https://bitbucket.org/gutworth/six/issue/155

Originally reported by: Barry Warsaw (Bitbucket: warsaw, GitHub: warsaw)

six hg head is not compatible with Python 3.5 (We're building against Python 3.5 in Ubuntu). ast module API breakage.

GLOB sdist-make: /home/barry/projects/debian/six/upstream/setup.py
py35 create: /home/barry/projects/debian/six/upstream/.tox/py35
py35 installdeps: pytest
py35 inst: /home/barry/projects/debian/six/upstream/.tox/dist/six-1.9.0.zip
py35 installed: py==1.4.30,pytest==2.7.2,six==1.9.0
py35 runtests: PYTHONHASHSEED='3069104211'
py35 runtests: commands[0] | py.test -rfsxX
============================= test session starts ==============================
platform linux -- Python 3.5.0 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/barry/projects/debian/six/upstream, inifile: setup.cfg
collected 0 items / 1 errors 

==================================== ERRORS ====================================
_________________________ ERROR collecting test_six.py _________________________
.tox/py35/lib/python3.5/site-packages/py/_path/local.py:650: in pyimport
    __import__(modname)
<frozen importlib._bootstrap>:969: in _find_and_load
    ???
<frozen importlib._bootstrap>:954: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:892: in _find_spec
    ???
<frozen importlib._bootstrap>:873: in _find_spec_legacy
    ???
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:137: in find_module
    source_stat, co = _rewrite_test(state, fn_pypath)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:278: in _rewrite_test
    rewrite_asserts(tree)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:336: in rewrite_asserts
    AssertionRewriter().run(mod)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:557: in run
    new.extend(self.visit(child))
/usr/lib/python3.5/ast.py:245: in visit
    return visitor(node)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:665: in visit_Assert
    top_condition, explanation = self.visit(assert_.test)
/usr/lib/python3.5/ast.py:245: in visit
    return visitor(node)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:796: in visit_Compare
    left_res, left_expl = self.visit(comp.left)
/usr/lib/python3.5/ast.py:245: in visit
    return visitor(node)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:757: in visit_Call
    new_func, func_expl = self.visit(call.func)
/usr/lib/python3.5/ast.py:245: in visit
    return visitor(node)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:757: in visit_Call
    new_func, func_expl = self.visit(call.func)
/usr/lib/python3.5/ast.py:245: in visit
    return visitor(node)
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:700: in visit_Name
    locs = ast.Call(self.builtin("locals"), [], [], None, None)
E   TypeError: Call constructor takes either 0 or 3 positional arguments
=========================== 1 error in 0.22 seconds ============================
ERROR: InvocationError: '/home/barry/projects/debian/six/upstream/.tox/py35/bin/py.test -rfsxX'
___________________________________ summary ___________________________________
ERROR:   py35: commands failed

• Bitbucket: https://bitbucket.org/gutworth/six/issue/131

Originally reported by: Bernhard Gschaider (Bitbucket: bgschaid, GitHub: bgschaid)

In the documentation it says

#!python

six.exec_(code, globals=None, locals=None)

but the implementation is

#!python

def exec_(code, globs=None, locs=None):
        """Execute code in a namespace."""

for the Python 2.x-branch

• Bitbucket: https://bitbucket.org/gutworth/six/issue/16

Hi! I'm writing a python program where I need to hook up the __import__ function in certain modules‘ __builtins__ dict to do some stuff, so that I can manage the imports in a way meta path hooks cannot provide. In python3+ it works just fine, but in python2.7 the interpreter I got this error

  File "six.py", line 625, in <module>
    get_unbound_function, """Get the function out of a possibly unbound function"""
  File "six.py", line 82, in _add_doc
    func.__doc__ = doc
RuntimeError: restricted attribute

It turns out the python 2.7 interpreter does not allow changing function object's __doc__ attribute in a restricted frame, which is defined by the frame's __builtins__ reference is not pointing to the internal one.

here is a repro script

#! /usr/bin/python2.7

import imp
import sys
import os

name = 'six'
dir_name = os.getcwd()  # where six.py lives

def my_import(name, globals, locals, fromlist, level):
    print(name, fromlist, level)
    return __import__(name, globals, locals, fromlist, level)

spec = imp.find_module(name, [dir_name])
module = imp.new_module(name)
my_builtins = dict(__builtins__.__dict__)
my_builtins['__import__'] = my_import
module.__builtins__ = my_builtins
sys.modules[name] = module
imp.load_module(name, *spec)

Thanks!

Python version: 3.10
Django version: 3.2.15
six: 1.16.0

If there is any import fails then this warning message comes in the console: <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

python manage.py shell
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

I debug it and found where and when exactly the message is appearing: shell.py

available_shells=['ipython', 'bpython', 'python'] so, if when ipython and bpython is not there in the system then it throws the above warning message. For the last value python there is no message, bcos its able to find it and there is no import error.

These multiple same messages look very ugly, is there any way to remove them?

The Python3 json module raises JSONDecodeError where the Python2 json module raises ValueError. Added six.JSONDecodeError that can be used to catch the exception regardless of version.

Unittests mock module was added in python 3, it's called unittest.mock in python 3 but it was made available as a backport for python 2 via pypi as mock.

The information is listed on the python 3 documentation https://docs.python.org/3/library/unittest.mock.html

unittest.mock — mock object library New in version 3.3. Source code: Lib/unittest/mock.py ... There is a backport of unittest.mock for earlier versions of Python, available as mock on PyPI.

Mock on pypi: https://pypi.org/project/mock/

This lets a user write six.moves.unittest_mock which is handy for projects like mine that use mock on python 2 and 3.

Hope this PR is alright. I also updated the changes file and version number etc, as it seemed the right thing to do.