The most widely used Python to C compiler

Overview

Welcome to Cython!

Cython is a language that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports more cutting edge functionality and optimizations.

The Cython language is very close to the Python language, but Cython additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code.

This makes Cython the ideal language for wrapping external C libraries, and for fast C modules that speed up the execution of Python code.

You can support the Cython project via Github Sponsors or Tidelift.

Installation:

If you already have a C compiler, just run following command:

pip install Cython

otherwise, see the installation page.

License:

The original Pyrex program was licensed "free of restrictions" (see below). Cython itself is licensed under the permissive Apache License.

See LICENSE.txt.

Contributing:

Want to contribute to the Cython project? Here is some help to get you started.

We are currently building the next great Cython edition: Cython 3.0. You can help us make the life of Python 3.x users easier.

Get the full source history:

Note that Cython used to ship the full version control repository in its source distribution, but no longer does so due to space constraints. To get the full source history from a downloaded source archive, make sure you have git installed, then step into the base directory of the Cython source distribution and type:

make repo

The following is from Pyrex:

This is a development version of Pyrex, a language for writing Python extension modules.

For more info, take a look at:

  • Doc/About.html for a description of the language
  • INSTALL.txt for installation instructions
  • USAGE.txt for usage instructions
  • Demos for usage examples

Comments, suggestions, bug reports, etc. are most welcome!

Copyright stuff: Pyrex is free of restrictions. You may use, redistribute, modify and distribute modified versions.

The latest version of Pyrex can be found here.

Greg Ewing, Computer Science Dept
University of Canterbury
Christchurch, New Zealand
A citizen of NewZealandCorp, a wholly-owned subsidiary of USA Inc.
Comments
  • Large output size of compiled code

    Large output size of compiled code

    Hi All,

    I'm trying to reduce the packaged size of the scientific computing ecosystem to make it friendlier for distributed deployments using tools like docker, kubernetes, etc.. An issue for this topic generally is here: https://github.com/ContinuumIO/anaconda-issues/issues/8242

    One important issue that came up from @mingwandroid relates to Cython, see https://github.com/ContinuumIO/anaconda-issues/issues/8242#issuecomment-359524128

    It seems the shared libraries are chock full of instruction code

    PyInit_cython_special is huge; more than 1MB.

    why all of these movq instructions are not being coalesced into a memset.

    In a separate conversation he said the following:

    Now ideally you'd like your compiler to be smart enough to coalesce all those clears over the whole object into 1 memset, and then even better, put them all together so that a single memset could do it. the package size on disk is swamped by the instructions to do these clears! instead of a call to a function to memset it's a huge amount of instructions, like tens of thousands. and that takes space on the extension module text segment.

    To be honest, I don't have much expertise here, but I thought I'd raise an issue here in case this sort of problem can be resolved within Cython, or, if not, if anyone here has any additional helpful thoughts.

    Thank you all for your time.

    enhancement Code Generation Build System 
    opened by mrocklin 52
  • Verbatim C code using docstring syntax.

    Verbatim C code using docstring syntax.

    Allow including snippets of C code in a cdef extern from * block using docstring syntax:

    cdef extern from *:
        """
        static long square(long x)
        {
            return x * x;
        }
        """
        long square(long)
    

    This is an alternative to adding an external .h file containing the C code. This would be useful especially for simple things, like a one-line macro.

    I still have to write documentation, but I'll wait for feedback first.

    Edit: This is currently a syntax error and thus cannot conflict with existing code.

    enhancement 
    opened by jdemeyer 50
  • Make GitHub Actions work on Windows

    Make GitHub Actions work on Windows

    I added tests for all versions of python to windows and solved some problems to let them work, but some don't: 2.7, 3.4 - since no suitable compiler version is installed 3.9 - inline, details: #3450, #4379 3.5 - one cpp-only test fails, output

    I also uncommented the coverage tests and added some code to the .sh to make them work. They work for a decent amount of time, which is the opposite of what was said in the comments.

    If anyone wants to try to fix the problems and run 2.7 (and maybe 3.4), then you will need to use this, probably relying on both versions of this answer.

    Testing 
    opened by 0dminnimda 39
  • cdef dataclasses

    cdef dataclasses

    Used cython.dataclasses.dataclass and cython.dataclasses.field to mark dataclasses and their fields.

    Tries to match the interface provided by a regular dataclass as much as possible. This means taking the types from the dataclasses module if available (so they match exactly) or a fallback Python version that just implements the core parts (obtained with PyRun_SimpleString in the C source).

    Use of placeholders in generated __init__ code means the code in the C file isn't hugely readable. Probably not a huge issue, but don't really see a way round that.

    As part of this I've also also implemented a Cython version of typing.ClassVar. Although really designed for use with dataclasses it behaves sensibly when used in types in a normal cdef class. Potentially this might be worth documenting more thoroughly?

    Status

    • [x] Both annotated variables and cdef attributes included in dataclass - done (but assignment syntax for cdef attributes is a bit clunky because it needs to be on a separate line)
    • [x] visibility of the attributes decided to be visible by default for annotations, invisible for cdef attributes
    • [x] non-public attributes omitted from __dataclass_fields__
    • [x] moving "directives" into cython.dataclasses and cython.typing submodules
      • [ ] I'd quite like these cython.dataclasses submodules and their attributes to be available at runtime and just forwarded to their standard library modules if available. This may be fiddly
    • [x] "frozen" option of dataclasses works. (Obviously hard to enforce at C level)

    Old commentary on design decisions (can now mostly be ignored)

    When finished closes https://github.com/cython/cython/issues/2903 - however, some design decisions pending before it's finished:

    What attributes should be included in the dataclass? Definitely annotated variables. Maybe regular cdef variables?

    What should the visibility of the attributes be? There's a few options:

    1. Default to invisible, like for a standard cdef class. This is obviously consistent. The issue with this is that there's a few functions in the dataclasses module like asdict which assume that every attribute declared in __dataclass_fields__ is readable. If they aren't then the classes won't be compatible with those interfaces.

      If so, should non-public attributes be omitted from __dataclass_fields__? This seems inconsistent since they do appear in the destructor, the repr, and affect the comparisons.

    2. Default to visible. This is inconsistent with the standard cdef class behaviour, but would make them as compatible as possible with standard dataclasses. It would also make sense for most use-cases I think. One problem is that the syntax doesn't really exist of override that (only public and readonly are defined).

    3. Annotated variables default to visible, cdef variables to invisible? It kind of makes sense and gives a way to control visibility, but it'd be a little inconsistent with everything. (I'm leaning towards this as the answer)

    The likely implementation deviates from the normal cdef class behaviour where

    cdef class C:
       a: `int` = 0
    

    makes a class-variable instead of an instance variable. I think this is unavoidable and makes sense in this case, but comments on this welcome too?

    It dumps a number of names in the cython scope (dataclass, field, InitVar, ClassVar). Would some sort of subscoping be better? Especially given that it isn't 100% obvious that any of these but dataclass related to dataclasses?

    feature Cython Language Feature 
    opened by da-woods 39
  • Pythonise the documentation according to #4187: Basic Tutorial (cython_tutorial.rst)

    Pythonise the documentation according to #4187: Basic Tutorial (cython_tutorial.rst)

    This is the first step in adding Pure Python code versions to all Cython documentation!

    Note, since the current tests for .py files do not add cython to namespace, some tests are expected to fail due to the convention not to use import cython in most pure python files.

    To be able to try this, you will need to install doc-requirements.txt (it has been updated).

    Lots of process details and standardization are found here: #4187

    Small preview: Small preview

    Documentation 
    opened by 0dminnimda 36
  • walrus operator/named expressions

    walrus operator/named expressions

    Fixes https://github.com/cython/cython/issues/2636

    With the exception of the the parser changes I don't think this is too bad so I'm marking it as ready.

    It implements the assignment expression largely as a composite of existing nodes, which I think is a reasonable approach.

    feature Python Semantics 
    opened by da-woods 36
  • no attribute __reduce_cython__

    no attribute __reduce_cython__

    https://github.com/cython/cython/issues/1894#issuecomment-339966952.

    I'm getting an AttributeError due to a missing __reduce_cython__ attribute in an embedded environment when I build lxml with Cython 0.26 or 0.27. 0.25(.2) works fine. The issue seems to be triggered by reinitializing the environment but unfortunately I was not able to find a minimal sample that replicates it yet.

    I did a git-bisect and found https://github.com/cython/cython/commit/f8b3405e926d2ba9bc2ee24d79848235875ee12e to be the first broken commit.

    Will try to find a simple test case, but I'm not sure how soon I'll have a result.

    EDIT: This was resolved in Cython 0.28. See https://github.com/cython/cython/issues/1953#issuecomment-398128940.

    opened by cschramm 36
  • Late includes

    Late includes

    This pull request adds support for a late #include. My proposed syntax is to add a leading pipe to the filename:

    cdef extern from "|spam.h":
        ...
    

    For this cdef extern block, Cython will generate the #include statement after its own variable declarations. So it can be used if the C code in spam.h needs to refer to Cython variables.

    cysignals has a use-case which is currently "solved" by some ugly hack involving .pxi files (for which I need #483). If this pull request is accepted, cysignals would no longer need to rely on .pxi files: https://github.com/sagemath/cysignals/pull/49

    Of course, there are plenty of bikeshedding opportunities for the syntax, but I care about the feature, not the syntax. I chose the leading pipe because it is unlikely to occur in actual filenames. (Also: think of the pipe as "piping" the Cython variables into the header file).

    opened by jdemeyer 36
  • [ENH] C functions should propagate exceptions by default

    [ENH] C functions should propagate exceptions by default

    Is your feature request related to a problem? Please describe. C functions that do not return Python objects cannot currently propagate exceptions by default but require an explicit except clause. In Python code, declared return types default to safe exception propagation instead. Both should behave the same.

    cdef int func():
        raise TypeError  # is not propagated
    
    @cython.cfunc
    def pyfunc() -> cython.int:
        raise TypeError  # is propagated
    
    cdef int no_exc_func():
        return 0  # no exception raised
    
    cdef extern from *:
        int no_exc_cfunc()  # no exception expected
    

    Describe the solution you'd like C functions should also propagate their exceptions by default.

    cdef int func():
        raise TypeError  # CHANGE: should get propagated as with `except? -1`, the default exception value.
    
    @cython.exceptval(check=False)
    cdef int func():
        raise TypeError  # is not propagated
    
    cdef int no_exc_func():
        return 0  # CHANGE: no exception raised, but tested by caller
    
    cdef int explicit_no_exc_func() noexcept:    # <- CHANGE
        return 0  # no exception raised, and not tested by caller
    
    cdef extern from *:
        int no_exc_cfunc()  # unclear - should exceptions also be expected for `extern` functions?
    

    Questions:

    • non-extern cdef functions declared in .pxd files should probably be affected, too, since they should match the implementation in the corresponding .pyx / .py file.
    • C functions defined in cdef extern blocks are unlikely to benefit from this change. Can we live with keeping them different?

    Also see https://github.com/cython/cython/issues/3580 https://github.com/cython/cython/issues/933

    enhancement Python Semantics P: blocker 
    opened by scoder 35
  • Add a Pythran backend for Numpy operation

    Add a Pythran backend for Numpy operation

    I've been working for quite some time on the usage of Pythran as a backend for the Numpy operations that Cython can generate. The associated PR on github can be found here: . This work has been sponsored by the OpenDreamKit project (https://github.com/OpenDreamKit/OpenDreamKit/).

    First of all, the Pythran project (https://github.com/serge-sans-paille/pythran) is a (subset of) Python to C++ compiler, that aims at optimizing "scientific" Python code. It also provides a full C++ implementation of a major set of the Numpy API. Some of the advantage of this implementation is that it supports expression templates and SIMD instructions (partially thanks to Boost.SIMD [1]).

    One of the limitation of the current Numpy support of Cython is that it relies on the original Numpy Python module for a lot of computations. The overall idea is to replace these calls by the Numpy implementation provided within the Pythran project.

    I'll discuss in this mail the various choices that have been made, why and some implementation details. Then we'll also show some benchmark to see the potential improvements, which is the point of all this in the end :)

    Pythran limitations

    The Pythran Numpy implementation has some limitations:

    • array "views" are not supported. That means that arrays must be stored in contiguous memory. Fortran and C-style format are supported.
    • the endianness of the integers must be the same that the one of the targeted architecture (note that Cython has the same limitation)

    That's why we did two things:

    • the usage of the Pythran backend needs to be explicitly asked by the user by providing the --np-pythran flag to the Cython compiler, or by using the "np_pythran" flag to the cythonize call (for distutils)
    • in function arguments, Numpy buffers are replaced by fused types to be able to fall back in case of unsupported buffers. More on this below.

    Implementation choices and details within Cython

    a) PythranExpr

    We defined a new type in PyrexTypes.py, which defines a Pythran buffer or expression. A Pythran expression is associated to a Pythran expression template, whose C++ type can be something like "decltype(a+b)". We thus compose every expression/function call like this, which allows us to use Pythran's expression template mechanism.

    We also choose to let the C++ compiler deduced the final type of every expression, and emit errors if something goes wrong. This choice allows not to have to rewrite in Python all the (potentially implicit) conversion rules that can apply in a C/C++ program, which could be error prone. The disadvantage is that it may generate not that trivial error messages for the end-user.

    b) Fused types for function arguments

    As Pythran has some limitations about the Numpy buffers it can support, we chose to replace Numpy buffer arguments by a fused type that can be either a Pythran buffer or the original Numpy buffer. The decision is made to use one type or another according to the limitations described above.

    This allows a fallback to the original Cython implementation in case of an unsupported buffer type.

    Tests

    A flag has been added to the runtests.py script. If provided with a path to a Pythran installation, it will run the C++ tests in "Pythran" mode. This allows to reuse the whole test suite of Cython.

    Benchmark

    The whole idea of this is to get better performances.

    Here is a simple benchmark of what this mode can achieve, using this cython code:

    def sqrt_sum(numpy.ndarray[numpy.float_t, ndim=1] a, numpy.ndarray[numpy.float_t, ndim=1] b):
        return numpy.sqrt(numpy.sqrt(a*a+b*b))
    

    On my computer (Core i7-6700HQ), this gives there results, with an array of 100000000 32-bit floats as input:

    • for the classical Cython version: 960ms
    • for the Cython version using the Pythran backend: 761ms
    • for the Cython version using the Pythran backend using SIMD instructions: 243ms

    which makes a speedup of ~3.9x using SIMD instructions.

    Documentation

    I put an example of how to use this with distutils in the documentation. It could be put elsewhere if needed, or formatted differently.

    opened by aguinet 34
  • Add create_extension() hook

    Add create_extension() hook

    As alternative to #412, add support for a hook function create_extension() allowing complete customization of creating the Extension object after Cython has processed the list of sources and # distutils declarations.

    opened by jdemeyer 34
  • [BUG] Using custom generic type in type alias is not supported

    [BUG] Using custom generic type in type alias is not supported

    Describe the bug

    Cython emits TypeError when TypeAlias contains custom Generic type, while mypy passes it.

    Code to reproduce the behaviour:

    from typing import Generic, TypeVar, TypeAlias
    
    T_co = TypeVar("T_co", covariant=True)
    
    
    class ExprProxy(Generic[T_co]):
        def __init__(self, value) -> None:
            self.value: T_co = value
    
    
    class ConstExpr:
        pass
    
    
    Evaluable: TypeAlias = ConstExpr | int | ExprProxy[ConstExpr]
    # TypeError: 'type' object is not subscriptable
    

    Expected behaviour

    Compile successfully without TypeError

    Environment

    OS: Windows Python version : 3.10.9 Cython version : 0.29.32 and 3.0.0a11 c97b77a

    Additional context

    Full source: https://github.com/armoha/eudplib/blob/f933c6b8da5d1b9bde9327c26719ff066380f405/eudplib/core/allocator/constexpr.pyx#L274

    opened by armoha 1
  • Update Cython Debugger

    Update Cython Debugger

    Wanted to see if we could modernize this a bit - looks like a lot of the documentation still wants users to use Python2.7 alongside this.

    Not sure everything here is 100% correct but I think still represents progress over what is there today. You can get a working Cython debugger image and mini instructions from this Docker repo:

    https://hub.docker.com/repository/docker/willayd/cython-debug

    opened by WillAyd 0
  • [BUG] Misleading error message when trying to cimport in pure Python

    [BUG] Misleading error message when trying to cimport in pure Python

    Describe the bug

    If you use the cimport keyword in pure Python it doesn't work (correctly). However, the error message suggests that you should use the cimport keyword

    Code to reproduce the behaviour:

    from cython.operator cimport dereference
    

    Output is:

    from cython.operator cimport dereference
                         ^
    ------------------------------------------------------------
    
    somefilename.py:1:21: Expected 'import' or 'cimport'
    

    Expected behaviour

    A useful error message. This isn't quite as simple as it looks because the cython module wants to be imported, while most other thing want from cython.cimports.... Although given that this is at the parsing stage, it'd probably be sufficient to just write expected "import"

    Environment

    Current master

    Additional context

    No response

    Error Reporting 
    opened by da-woods 0
  • [docs] Parallelization tutorial

    [docs] Parallelization tutorial

    I've written a parallelization tutorial. It doesn't cover that much new ground from the main parallelization documents, but hopefully it present a separate example of each use case, and is a little less of "here are the parameters, good luck" than the main parallelization reference

    Kind of closes https://github.com/cython/cython/issues/5125

    Documentation 
    opened by da-woods 3
  • [BUG] Syntax error in C array declaration and initialization

    [BUG] Syntax error in C array declaration and initialization

    Describe the bug

    I found it in #5177.

    def main():
        cdef int arr[4] = [1, 2, 3, 4]
    

    This code doesn't compile:

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    def main():
        cdef int arr[4] = [1, 2, 3, 4]
                        ^
    ------------------------------------------------------------
    
    test.pyx:2:20: Syntax error in C variable declaration
    

    But these two do:

    def main():
        cdef int arr1[4]
        arr1 = [1, 2, 3, 4]
    
        cdef int[4] arr2 = [1, 2, 3, 4]
    

    I think it is already known, because it has been mentioned in the documentation:

    • https://github.com/cython/cython/blob/b2cad768499cf5763251fe757491e9fc24a1f583/docs/src/userguide/language_basics.rst#L124-L129

    but I couldn't find the related issue.

    Anyway, if it can't be fixed easily, it will be good to document it more clearly.

    Code to reproduce the behaviour:

    No response

    Expected behaviour

    No response

    Environment

    Cython version: both 0.29.32 and 3.0.0a11

    Additional context

    No response

    opened by GalaxySnail 0
Releases(3.0.0a11)
A small C compiler written in Python for learning purposes

A small C compiler written in Python. Generates x64 Intel-format assembly, which is then assembled and linked by nasm and ld.

Scattered Thoughts 3 Oct 22, 2021
a simple functional programming language compiler written in python

Functional Programming Language A compiler for my small functional language. Written in python with SLY lexer/parser generator library. Requirements p

Ashkan Laei 3 Nov 5, 2021
A simple BrainF**k compiler written in Python

bf-comp A simple BrainF**k compiler written in Python. What else were you looking for?

null 1 Jan 9, 2022
🎉 🎉 PyComp - Java Code compiler written in python.

?? ?? PyComp Java Code compiler written in python. This is yet another compiler meant for babcock students project which was created using pure python

Alumona Benaiah 5 Nov 30, 2022
Pulse sequence builder and compiler for q1asm

q1pulse Pulse sequence builder and compiler for q1asm. q1pulse is a simple library to compile pulse sequence to q1asm, the assembly language of Qblox

Sander de Snoo 3 Dec 14, 2022
MatroSka Mod Compiler for ts4scripts

MMC Current Version: 0.2 MatroSka Mod Compiler for .ts4script files Requirements Have Python 3.7 installed and set as default. Running from Source pip

MatroSka 1 Dec 13, 2021
This is a small compiler to demonstrate how compilers work.

This is a small compiler to demonstrate how compilers work. It compiles our own dialect to C, while being written in Python.

Md. Tonoy Akando 2 Jul 19, 2022
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.

██████╗ ██╗██████╗ ███████╗██╗ ██╗███╗ ██╗███████╗ ██████╗ ██╔══██╗██║██╔══██╗██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝ ██████╔╝██║██████╔╝█

Julian Kemmerer 391 Jan 1, 2023
Hotpile: High Order Turing Machine Language Compiler

Hotpile: High Order Turing Machine Language Compiler Build and Run Requirements: Python 3.6+, bison, flex, and GCC installed. Needs to be run under UN

Jiang Weihao 4 Dec 29, 2021
Compiler Final Project - Lisp Interpreter

Compiler Final Project - Lisp Interpreter

null 2 Jan 23, 2022
Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars.

Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars. Practically useless for me now, but can be used as an example of tkinter.

BoggoTV 1 Dec 11, 2021
Python most simple|stupid programming language (MSPL)

Most Simple|Stupid Programming language. (MSPL) Stack - Based programming language "written in Python" Features: Interpretate code (Run). Generate gra

Kirill Zhosul 14 Nov 3, 2022
CBLang is a programming language aiming to fix most of my problems with Python

CBLang A bad programming language made in Python. CBLang is a programming language aiming to fix most of my problems with Python (this means that you

Chadderbox 43 Dec 22, 2022
This program goes thru reddit, finds the most mentioned tickers and uses Vader SentimentIntensityAnalyzer to calculate the ticker compound value.

This program goes thru reddit, finds the most mentioned tickers and uses Vader SentimentIntensityAnalyzer to calculate the ticker compound value.

null 195 Dec 13, 2022
A step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

This project has a step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

Steve Macenski 49 Dec 22, 2022
Wrappers around the most common maya.cmds and maya.api use cases

Maya FunctionSet (maya_fn) A package that decompose core maya.cmds and maya.api features to a set of simple functions. Tests The recommended approach

Ryan Porter 9 Mar 12, 2022
The most hackable keyboard in all the land

MiRage Modular Keyboard © 2021 Zack Freedman of Voidstar Lab Licensed Creative Commons 4.0 Attribution Noncommercial Share-Alike The MiRage is a 60% o

Zack Freedman 558 Dec 30, 2022
The goal of this program was to find the most common color in my living room.

The goal of this program was to find the most common color in my living room. I found a dataset online with colors names and their corr

null 1 Nov 9, 2021
Supply Chain will be a SAAS platfom to provide e-logistic facilites with most optimal

Shipp It Welcome To Supply Chain App [ Shipp It ] In "Shipp It" we are creating a full solution[web+app] for a entire supply chain from receiving orde

SAIKAT_CLAW 25 Dec 26, 2022