Hi All,

I'm trying to reduce the packaged size of the scientific computing ecosystem to make it friendlier for distributed deployments using tools like docker, kubernetes, etc.. An issue for this topic generally is here: https://github.com/ContinuumIO/anaconda-issues/issues/8242

One important issue that came up from @mingwandroid relates to Cython, see https://github.com/ContinuumIO/anaconda-issues/issues/8242#issuecomment-359524128

It seems the shared libraries are chock full of instruction code

PyInit_cython_special is huge; more than 1MB.

why all of these movq instructions are not being coalesced into a memset.

In a separate conversation he said the following:

Now ideally you'd like your compiler to be smart enough to coalesce all those clears over the whole object into 1 memset, and then even better, put them all together so that a single memset could do it. the package size on disk is swamped by the instructions to do these clears! instead of a call to a function to memset it's a huge amount of instructions, like tens of thousands. and that takes space on the extension module text segment.

To be honest, I don't have much expertise here, but I thought I'd raise an issue here in case this sort of problem can be resolved within Cython, or, if not, if anyone here has any additional helpful thoughts.

Thank you all for your time.

Allow including snippets of C code in a cdef extern from * block using docstring syntax:

cdef extern from *:
    """
    static long square(long x)
    {
        return x * x;
    }
    """
    long square(long)

This is an alternative to adding an external .h file containing the C code. This would be useful especially for simple things, like a one-line macro.

I still have to write documentation, but I'll wait for feedback first.

Edit: This is currently a syntax error and thus cannot conflict with existing code.

I added tests for all versions of python to windows and solved some problems to let them work, but some don't: 2.7, 3.4 - since no suitable compiler version is installed 3.9 - inline, details: #3450, #4379 3.5 - one cpp-only test fails, output

I also uncommented the coverage tests and added some code to the .sh to make them work. They work for a decent amount of time, which is the opposite of what was said in the comments.

If anyone wants to try to fix the problems and run 2.7 (and maybe 3.4), then you will need to use this, probably relying on both versions of this answer.

Used cython.dataclasses.dataclass and cython.dataclasses.field to mark dataclasses and their fields.

Tries to match the interface provided by a regular dataclass as much as possible. This means taking the types from the dataclasses module if available (so they match exactly) or a fallback Python version that just implements the core parts (obtained with PyRun_SimpleString in the C source).

Use of placeholders in generated __init__ code means the code in the C file isn't hugely readable. Probably not a huge issue, but don't really see a way round that.

As part of this I've also also implemented a Cython version of typing.ClassVar. Although really designed for use with dataclasses it behaves sensibly when used in types in a normal cdef class. Potentially this might be worth documenting more thoroughly?

Status

• [x] Both annotated variables and cdef attributes included in dataclass - done (but assignment syntax for cdef attributes is a bit clunky because it needs to be on a separate line)
• [x] visibility of the attributes decided to be visible by default for annotations, invisible for cdef attributes
• [x] non-public attributes omitted from __dataclass_fields__
• [x] moving "directives" into cython.dataclasses and cython.typing submodules
  • [ ] I'd quite like these cython.dataclasses submodules and their attributes to be available at runtime and just forwarded to their standard library modules if available. This may be fiddly
• [x] "frozen" option of dataclasses works. (Obviously hard to enforce at C level)

Old commentary on design decisions (can now mostly be ignored)

When finished closes https://github.com/cython/cython/issues/2903 - however, some design decisions pending before it's finished:

What attributes should be included in the dataclass? Definitely annotated variables. Maybe regular cdef variables?

What should the visibility of the attributes be? There's a few options:

1. Default to invisible, like for a standard cdef class. This is obviously consistent. The issue with this is that there's a few functions in the dataclasses module like asdict which assume that every attribute declared in __dataclass_fields__ is readable. If they aren't then the classes won't be compatible with those interfaces.

   If so, should non-public attributes be omitted from __dataclass_fields__? This seems inconsistent since they do appear in the destructor, the repr, and affect the comparisons.

2. Default to visible. This is inconsistent with the standard cdef class behaviour, but would make them as compatible as possible with standard dataclasses. It would also make sense for most use-cases I think. One problem is that the syntax doesn't really exist of override that (only public and readonly are defined).

3. Annotated variables default to visible, cdef variables to invisible? It kind of makes sense and gives a way to control visibility, but it'd be a little inconsistent with everything. (I'm leaning towards this as the answer)

The likely implementation deviates from the normal cdef class behaviour where

cdef class C:
   a: `int` = 0

makes a class-variable instead of an instance variable. I think this is unavoidable and makes sense in this case, but comments on this welcome too?

It dumps a number of names in the cython scope (dataclass, field, InitVar, ClassVar). Would some sort of subscoping be better? Especially given that it isn't 100% obvious that any of these but dataclass related to dataclasses?

This is the first step in adding Pure Python code versions to all Cython documentation!

Note, since the current tests for .py files do not add cython to namespace, some tests are expected to fail due to the convention not to use import cython in most pure python files.

To be able to try this, you will need to install doc-requirements.txt (it has been updated).

Lots of process details and standardization are found here: #4187

Small preview:

Fixes https://github.com/cython/cython/issues/2636

With the exception of the the parser changes I don't think this is too bad so I'm marking it as ready.

It implements the assignment expression largely as a composite of existing nodes, which I think is a reasonable approach.

https://github.com/cython/cython/issues/1894#issuecomment-339966952.

I'm getting an AttributeError due to a missing __reduce_cython__ attribute in an embedded environment when I build lxml with Cython 0.26 or 0.27. 0.25(.2) works fine. The issue seems to be triggered by reinitializing the environment but unfortunately I was not able to find a minimal sample that replicates it yet.

I did a git-bisect and found https://github.com/cython/cython/commit/f8b3405e926d2ba9bc2ee24d79848235875ee12e to be the first broken commit.

Will try to find a simple test case, but I'm not sure how soon I'll have a result.

EDIT: This was resolved in Cython 0.28. See https://github.com/cython/cython/issues/1953#issuecomment-398128940.

This pull request adds support for a late #include. My proposed syntax is to add a leading pipe to the filename:

cdef extern from "|spam.h":
    ...

For this cdef extern block, Cython will generate the #include statement after its own variable declarations. So it can be used if the C code in spam.h needs to refer to Cython variables.

cysignals has a use-case which is currently "solved" by some ugly hack involving .pxi files (for which I need #483). If this pull request is accepted, cysignals would no longer need to rely on .pxi files: https://github.com/sagemath/cysignals/pull/49

Of course, there are plenty of bikeshedding opportunities for the syntax, but I care about the feature, not the syntax. I chose the leading pipe because it is unlikely to occur in actual filenames. (Also: think of the pipe as "piping" the Cython variables into the header file).

Is your feature request related to a problem? Please describe. C functions that do not return Python objects cannot currently propagate exceptions by default but require an explicit except clause. In Python code, declared return types default to safe exception propagation instead. Both should behave the same.

cdef int func():
    raise TypeError  # is not propagated

@cython.cfunc
def pyfunc() -> cython.int:
    raise TypeError  # is propagated

cdef int no_exc_func():
    return 0  # no exception raised

cdef extern from *:
    int no_exc_cfunc()  # no exception expected

Describe the solution you'd like C functions should also propagate their exceptions by default.

cdef int func():
    raise TypeError  # CHANGE: should get propagated as with `except? -1`, the default exception value.

@cython.exceptval(check=False)
cdef int func():
    raise TypeError  # is not propagated

cdef int no_exc_func():
    return 0  # CHANGE: no exception raised, but tested by caller

cdef int explicit_no_exc_func() noexcept:    # <- CHANGE
    return 0  # no exception raised, and not tested by caller

cdef extern from *:
    int no_exc_cfunc()  # unclear - should exceptions also be expected for `extern` functions?

Questions:

• non-extern cdef functions declared in .pxd files should probably be affected, too, since they should match the implementation in the corresponding .pyx / .py file.
• C functions defined in cdef extern blocks are unlikely to benefit from this change. Can we live with keeping them different?

Also see https://github.com/cython/cython/issues/3580 https://github.com/cython/cython/issues/933

I've been working for quite some time on the usage of Pythran as a backend for the Numpy operations that Cython can generate. The associated PR on github can be found here: . This work has been sponsored by the OpenDreamKit project (https://github.com/OpenDreamKit/OpenDreamKit/).

First of all, the Pythran project (https://github.com/serge-sans-paille/pythran) is a (subset of) Python to C++ compiler, that aims at optimizing "scientific" Python code. It also provides a full C++ implementation of a major set of the Numpy API. Some of the advantage of this implementation is that it supports expression templates and SIMD instructions (partially thanks to Boost.SIMD [1]).

One of the limitation of the current Numpy support of Cython is that it relies on the original Numpy Python module for a lot of computations. The overall idea is to replace these calls by the Numpy implementation provided within the Pythran project.

I'll discuss in this mail the various choices that have been made, why and some implementation details. Then we'll also show some benchmark to see the potential improvements, which is the point of all this in the end :)

Pythran limitations

The Pythran Numpy implementation has some limitations:

• array "views" are not supported. That means that arrays must be stored in contiguous memory. Fortran and C-style format are supported.
• the endianness of the integers must be the same that the one of the targeted architecture (note that Cython has the same limitation)

That's why we did two things:

• the usage of the Pythran backend needs to be explicitly asked by the user by providing the --np-pythran flag to the Cython compiler, or by using the "np_pythran" flag to the cythonize call (for distutils)
• in function arguments, Numpy buffers are replaced by fused types to be able to fall back in case of unsupported buffers. More on this below.

Implementation choices and details within Cython

a) PythranExpr

We defined a new type in PyrexTypes.py, which defines a Pythran buffer or expression. A Pythran expression is associated to a Pythran expression template, whose C++ type can be something like "decltype(a+b)". We thus compose every expression/function call like this, which allows us to use Pythran's expression template mechanism.

We also choose to let the C++ compiler deduced the final type of every expression, and emit errors if something goes wrong. This choice allows not to have to rewrite in Python all the (potentially implicit) conversion rules that can apply in a C/C++ program, which could be error prone. The disadvantage is that it may generate not that trivial error messages for the end-user.

b) Fused types for function arguments

As Pythran has some limitations about the Numpy buffers it can support, we chose to replace Numpy buffer arguments by a fused type that can be either a Pythran buffer or the original Numpy buffer. The decision is made to use one type or another according to the limitations described above.

This allows a fallback to the original Cython implementation in case of an unsupported buffer type.

Tests

A flag has been added to the runtests.py script. If provided with a path to a Pythran installation, it will run the C++ tests in "Pythran" mode. This allows to reuse the whole test suite of Cython.

Benchmark

The whole idea of this is to get better performances.

Here is a simple benchmark of what this mode can achieve, using this cython code:

def sqrt_sum(numpy.ndarray[numpy.float_t, ndim=1] a, numpy.ndarray[numpy.float_t, ndim=1] b):
    return numpy.sqrt(numpy.sqrt(a*a+b*b))

On my computer (Core i7-6700HQ), this gives there results, with an array of 100000000 32-bit floats as input:

• for the classical Cython version: 960ms
• for the Cython version using the Pythran backend: 761ms
• for the Cython version using the Pythran backend using SIMD instructions: 243ms

which makes a speedup of ~3.9x using SIMD instructions.

Documentation

I put an example of how to use this with distutils in the documentation. It could be put elsewhere if needed, or formatted differently.

As alternative to #412, add support for a hook function create_extension() allowing complete customization of creating the Extension object after Cython has processed the list of sources and # distutils declarations.

Describe the bug

Cython emits TypeError when TypeAlias contains custom Generic type, while mypy passes it.

Code to reproduce the behaviour:

from typing import Generic, TypeVar, TypeAlias

T_co = TypeVar("T_co", covariant=True)


class ExprProxy(Generic[T_co]):
    def __init__(self, value) -> None:
        self.value: T_co = value


class ConstExpr:
    pass


Evaluable: TypeAlias = ConstExpr | int | ExprProxy[ConstExpr]
# TypeError: 'type' object is not subscriptable

Expected behaviour

Compile successfully without TypeError

Environment

OS: Windows Python version : 3.10.9 Cython version : 0.29.32 and 3.0.0a11 c97b77a

Additional context

Full source: https://github.com/armoha/eudplib/blob/f933c6b8da5d1b9bde9327c26719ff066380f405/eudplib/core/allocator/constexpr.pyx#L274

Wanted to see if we could modernize this a bit - looks like a lot of the documentation still wants users to use Python2.7 alongside this.

Not sure everything here is 100% correct but I think still represents progress over what is there today. You can get a working Cython debugger image and mini instructions from this Docker repo:

https://hub.docker.com/repository/docker/willayd/cython-debug

Describe the bug

If you use the cimport keyword in pure Python it doesn't work (correctly). However, the error message suggests that you should use the cimport keyword

Code to reproduce the behaviour:

from cython.operator cimport dereference

Output is:

from cython.operator cimport dereference
                     ^
------------------------------------------------------------

somefilename.py:1:21: Expected 'import' or 'cimport'

Expected behaviour

A useful error message. This isn't quite as simple as it looks because the cython module wants to be imported, while most other thing want from cython.cimports.... Although given that this is at the parsing stage, it'd probably be sufficient to just write expected "import"

Environment

Current master

Additional context

No response

I've written a parallelization tutorial. It doesn't cover that much new ground from the main parallelization documents, but hopefully it present a separate example of each use case, and is a little less of "here are the parameters, good luck" than the main parallelization reference

Kind of closes https://github.com/cython/cython/issues/5125

Describe the bug

I found it in #5177.

def main():
    cdef int arr[4] = [1, 2, 3, 4]

This code doesn't compile:

Error compiling Cython file:
------------------------------------------------------------
...
def main():
    cdef int arr[4] = [1, 2, 3, 4]
                    ^
------------------------------------------------------------

test.pyx:2:20: Syntax error in C variable declaration

But these two do:

def main():
    cdef int arr1[4]
    arr1 = [1, 2, 3, 4]

    cdef int[4] arr2 = [1, 2, 3, 4]

I think it is already known, because it has been mentioned in the documentation:

• https://github.com/cython/cython/blob/b2cad768499cf5763251fe757491e9fc24a1f583/docs/src/userguide/language_basics.rst#L124-L129

but I couldn't find the related issue.

Anyway, if it can't be fixed easily, it will be good to document it more clearly.

Code to reproduce the behaviour:

No response

Expected behaviour

No response

Environment

Cython version: both 0.29.32 and 3.0.0a11

Additional context

No response