Enable ++x and --x expressions in Python

Overview

plusplus

PyPI version CI status Codecov

Enable ++x and --x expressions in Python

What's this?

By default, Python supports neither pre-increments (like ++x) nor post-increments (like x++). However, the first ones are syntactically correct since Python parses them as two subsequent +x operations, where + is the unary plus operator (same with --x and the unary minus). They both have no effect, since in practice -(-x) == +(+x) == x.

This module turns the ++x-like expressions into x += 1 at the bytecode level. Increments and decrements of collection items and object attributes are supported as well, for example:

dictionary = {'key': 42}
++dictionary['key']
assert dictionary['key'] == 43

Unlike x += 1, ++x is still an expression, so the increments work fine inside other expressions, if/while conditions, lambda functions, and list/dict comprehensions:

array[++index] = new_value

if --connection.num_users == 0:
    connection.close()

button.add_click_callback(lambda: ++counter)

index = 0
indexed_cells = {++index: cell for row in table for cell in row}

See tests for more sophisticated examples.

[How it works] [Installation]

Why?

This module is made for fun, as a demonstration of Python flexibility and bytecode manipulation techniques. Note that enabling increments in real projects may be risky: such code may confuse new developers and behave differently if copied to environments without the plusplus module. Also, this feature gives more opportunities to write unreadable code in general.

Nevertheless, there are situations where increments (if used with care) may allow to avoid repetitions or make code more readable. Some of them are listed here with the examples from the source code of the Python standard library.

Also, having the increment expressions seems consistent with PEP 572 "Assignment Expressions" that introduced the x := value expressions in Python 3.8+. They can be used inside if/while conditions and lambda functions as well.

How it works?

Patching bytecode

Python compiles all source code to a low-level bytecode executed on the Python's stack-based virtual machine. Each bytecode instruction consumes a few items from the stack, does something with them, and pushes the results back to the stack.

The ++x expressions are compiled into two consecutive UNARY_POSITIVE instructions that do not save the intermediate result in between (same with --x and two UNARY_NEGATIVE instructions). No other expressions produce a similar bytecode pattern.

plusplus replaces these patterns with the bytecode for x += 1, then adds the bytecode for storing the resulting value to the place where the initial value was taken.

This is what happens for the y = ++x line:

A similar but more complex transformation happens for the code with subscription expressions like value = ++dictionary['key']. We need the instructions from the yellow boxes to save the initial location and recall it when the increment is done (see the explanation below):

This bytecode is similar to what the string dictionary['key'] += 1 compiles to. The only difference is that it keeps an extra copy of the incremented value, so we can return it from the expression and assign it to the value variable.

Arguably, the least clear part here is the second yellow box. Actually, it is only needed to reorder the top 4 items of the stack. If we need to reorder the top 2 or 3 items of the stack, we can just use the ROT_TWO and ROT_THREE instructions (they do a circular shift of the specified number of items of the stack). If we had a ROT_FOUR instruction, we would be able to just replace the second yellow box with two ROT_FOURs to achieve the desired order.

However, ROT_FOUR was removed in Python 3.2 (since it was rarely used by the compiler) and recovered back only in Python 3.8. If we want to support Python 3.3 - 3.7, we need to use a workaround, e.g. the BUILD_TUPLE and UNPACK_SEQUENCE instructions. The first one replaces the top N items of the stack with a tuple made of these N items. The second unpacks the tuple putting the values on the stack right-to-left, i.e. in reverse order. We use them to reverse the top 4 items, then swap the top two to achieve the desired order.

[Source code]

The @enable_increments decorator

The first way to enable the increments is to use a decorator that would patch the bytecode of a given function.

The decorator disassembles the bytecode, patches the patterns described above, and recursively calls itself for any nested bytecode objects (this way, the nested function and class definitions are also patched).

The bytecode is disassembled and assembled back using the MatthieuDartiailh/bytecode library.

[Source code]

Enabling increments in the whole package

The Python import system allows loading modules not only from files but from any reasonable place (e.g. there was a module that enables importing code from Stack Overflow answers). The only thing you need is to provide module contents, including its bytecode.

We can leverage this to implement a wrapping loader that imports the module as usual but patching its bytecode as described above. To do this, we can create a new MetaPathFinder and install it to sys.meta_path.

[Source code]

Why not just override unary plus operator?

  • This way, it would be impossible to distinguish applying two unary operators consequently (like ++x) from applying them in separate places of a program (like in the snippet below). It is important to not change behavior in the latter case.

    x = -value
    y = -x
  • Overriding operators via magic methods (such as __pos__() and __neg__()) do not work for built-in Python types like int, float, etc. In contrast, plusplus works with all built-in and user-defined types.

Caveats

  • pytest does its own bytecode modifications in tests, adding the code to save intermediate expression results to the assert statements. This is necessary to show these results if the test fails (see pytest docs).

    By default, this breaks the plusplus patcher because the two UNARY_POSITIVE instructions become separated by the code saving the result of the first UNARY_POSITIVE.

    We fix that by removing the code saving some of the intermediate results, which does not break the pytest introspection.

    [Source code]

How to use it?

You can install this module with pip:

pip install plusplus

For a particular function or method

Add a decorator:

from plusplus import enable_increments

@enable_increments
def increment_and_return(x):
    return ++x

This enables increments for all code inside the function, including nested function and class definitions.

For all code in your package

In package/__init__.py, make this call before you import submodules:

from plusplus import enable_increments

enable_increments(__name__)

# Import submodules here
...

This enables increments in the submodules, but not in the package/__init__.py code itself.

Other ideas

The same approach could be used to implement the assignment expressions for the Python versions that don't support them. For example, we could replace the x <-- value expressions (two unary minuses + one comparison) with actual assignments (setting x to value).

See also

  • cpmoptimize — a module that optimizes a Python code calculating linear recurrences, reducing the time complexity from O(n) to O(log n).
  • dontasq — a module that adds functional-style methods (such as .where(), .group_by(), .order_by()) to built-in Python collections.

Authors

Copyright © 2021 Alexander Borzunov

You might also like...
Python @deprecat decorator to deprecate old python classes, functions or methods.

deprecat Decorator Python @deprecat decorator to deprecate old python classes, functions or methods. Installation pip install deprecat Usage To use th

Find dependent python scripts of a python script in a project directory.

Find dependent python scripts of a python script in a project directory.

SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

An OData v4 query parser and transpiler for Python

odata-query is a library that parses OData v4 filter strings, and can convert them to other forms such as Django Queries, SQLAlchemy Queries, or just plain SQL.

Python program to do with percentages and chances, random generation.

Chances and Percentages Python program to do with percentages and chances, random generation. What is this? This small program will generate a list wi

A Python library for reading, writing and visualizing the OMEGA Format
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

A simple and easy to use Spam Bot made in Python!

This is a simple spam bot made in python. You can use to to spam anyone with anything on any platform.

RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.

pydsinternals - Directory Services Internals Library A Python native library containing necessary classes, functions and structures to interact with W

Owner
Alexander Borzunov
Building hivemind for @learning-at-home // ex⁠-⁠research engineer at Yandex Self-Driving, ex⁠-⁠intern at Facebook
Alexander Borzunov
This is discord nitro code generator and checker made with python. This will generate nitro codes and checks if the code is valid or not. If code is valid then it will print the code leaving 2 lines and if not then it will print '*'.

Discord Nitro Generator And Checker ⚙️ Rᴜɴ Oɴ Rᴇᴘʟɪᴛ ??️ Lᴀɴɢᴜᴀɢᴇs Aɴᴅ Tᴏᴏʟs If you are taking code from this repository without a fork, then atleast

Vɪɴᴀʏᴀᴋ Pᴀɴᴅᴇʏ 37 Jan 7, 2023
A python package containing all the basic functions and classes for python. From simple addition to advanced file encryption.

A python package containing all the basic functions and classes for python. From simple addition to advanced file encryption.

PyBash 11 May 22, 2022
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Boltons boltons should be builtins. Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously mis

Mahmoud Hashemi 6k Jan 4, 2023
A Python utility belt containing simple tools, a stdlib like feel, and extra batteries. Hashing, Caching, Timing, Progress, and more made easy!

Ubelt is a small library of robust, tested, documented, and simple functions that extend the Python standard library. It has a flat API that all behav

Jon Crall 638 Dec 13, 2022
isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.

isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type. It provides a command line utility, Python library and plugins for various editors to quickly sort all your imports.

Python Code Quality Authority 5.5k Jan 8, 2023
A python lib for generate random string and digits and special characters or A combination of them

A python lib for generate random string and digits and special characters or A combination of them

Torham 4 Nov 15, 2022
Cleaning-utils - a collection of small Python functions and classes which make cleaning pipelines shorter and easier

cleaning-utils [] [] [] cleaning-utils is a collection of small Python functions

null 4 Aug 31, 2022
✨ Voici un code en Python par moi, et en français qui permet d'exécuter du Javascript en Python.

JavaScript In Python ❗ Voici un code en Python par moi, et en français qui permet d'exécuter du Javascript en Python. ?? Une vidéo pour vous expliquer

MrGabin 4 Mar 28, 2022
Simple python module to get the information regarding battery in python.

Battery Stats A python3 module created for easily reading the current parameters of Battery in realtime. It reads battery stats from /sys/class/power_

Shreyas Ashtamkar 5 Oct 21, 2022
ticktock is a minimalist library to view Python time performance of Python code.

ticktock is a minimalist library to view Python time performance of Python code.

Victor Benichoux 30 Sep 28, 2022