Grumpy is a Python to Go source code transcompiler and runtime.

Overview

Grumpy: Go running Python

Build Status Join the chat at https://gitter.im/grumpy-devel/Lobby

Overview

Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop-in replacement for CPython 2.7. The key difference is that it compiles Python source code to Go source code which is then compiled to native code, rather than to bytecode. This means that Grumpy has no VM. The compiled Go source code is a series of calls to the Grumpy runtime, a Go library serving a similar purpose to the Python C API (although the API is incompatible with CPython's).

Limitations

Things that will probably never be supported by Grumpy

  1. exec, eval and compile: These dynamic features of CPython are not supported by Grumpy because Grumpy modules consist of statically-compiled Go code. Supporting dynamic execution would require bundling Grumpy programs with the compilation toolchain, which would be unwieldy and impractically slow.

  2. C extension modules: Grumpy has a different API and object layout than CPython and so supporting C extensions would be difficult. In principle it's possible to support them via an API bridge layer like the one that JyNI provides for Jython, but it would be hard to maintain and would add significant overhead when calling into and out of extension modules.

Things that Grumpy will support but doesn't yet

There are three basic categories of incomplete functionality:

  1. Language features: Most language features are implemented with the notable exception of old-style classes. There are also a handful of operators that aren't yet supported.

  2. Builtin functions and types: There are a number of missing functions and types in __builtins__ that have not yet been implemented. There are also a lot of methods on builtin types that are missing.

  3. Standard library: The Python standard library is very large and much of it is pure Python, so as the language features and builtins get filled out, many modules will just work. But there are also a number of libraries in CPython that are C extension modules which will need to be rewritten.

  4. C locale support: Go doesn't support locales in the same way that C does. As such, some functionality that is locale-dependent may not currently work the same as in CPython.

Running Grumpy Programs

Method 1: make run:

The simplest way to execute a Grumpy program is to use make run, which wraps a shell script called grumprun that takes Python code on stdin and builds and runs the code under Grumpy. All of the commands below are assumed to be run from the root directory of the Grumpy source code distribution:

echo "print 'hello, world'" | make run

Method 2: grumpc and grumprun:

For more complicated programs, you'll want to compile your Python source code to Go using grumpc (the Grumpy compiler) and then build the Go code using go build. Since Grumpy programs are statically linked, all the modules in a program must be findable by the Grumpy toolchain on the GOPATH. Grumpy looks for Go packages corresponding to Python modules in the __python__ subdirectory of the GOPATH. By convention, this subdirectory is also used for staging Python source code, making it similar to the PYTHONPATH.

The first step is to set up the shell so that the Grumpy toolchain and libraries can be found. From the root directory of the Grumpy source distribution run:

make
export PATH=$PWD/build/bin:$PATH
export GOPATH=$PWD/build
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages

You will know things are working if you see the expected output from this command:

echo 'import sys; print sys.version' | grumprun

Next, we will write our simple Python module into the __python__ directory:

echo 'def hello(): print "hello, world"' > $GOPATH/src/__python__/hello.py

To build a Go package from our Python script, run the following:

mkdir -p $GOPATH/src/__python__/hello
grumpc -modname=hello $GOPATH/src/__python__/hello.py > \
    $GOPATH/src/__python__/hello/module.go

You should now be able to build a Go program that imports the package "__python__/hello". We can also import this module into Python programs that are built using grumprun:

echo 'from hello import hello; hello()' | grumprun

grumprun is doing a few things under the hood here:

  1. Compiles the given Python code to a dummy Go package, the same way we produced __python__/hello/module.go above
  2. Produces a main Go package that imports the Go package from step 1. and executes it as our __main__ Python package
  3. Executes go run on the main package generated in step 2.

Developing Grumpy

There are three main components and depending on what kind of feature you're writing, you may need to change one or more of these.

grumpc

Grumpy converts Python programs into Go programs and grumpc is the tool responsible for parsing Python code and generating Go code from it. grumpc is written in Python and uses the pythonparser module to accomplish parsing.

The grumpc script itself lives at tools/grumpc. It is supported by a number of Python modules in the compiler subdir.

Grumpy Runtime

The Go code generated by grumpc performs operations on data structures that represent Python objects in running Grumpy programs. These data structures and operations are defined in the grumpy Go library (source is in the runtime subdir of the source distribution). This runtime is analogous to the Python C API and many of the structures and operations defined by grumpy have counterparts in CPython.

Grumpy Standard Library

Much of the Python standard library is written in Python and thus "just works" in Grumpy. These parts of the standard library are copied from CPython 2.7 (possibly with light modifications). For licensing reasons, these files are kept in the third_party subdir.

The parts of the standard library that cannot be written in pure Python, e.g. file and directory operations, are kept in the lib subdir. In CPython these kinds of modules are written as C extensions. In Grumpy they are written in Python but they use native Go extensions to access facilities not otherwise available in Python.

Source Code Overview

  • compiler: Python package implementating Python -> Go transcompilation logic.
  • lib: Grumpy-specific Python standard library implementation.
  • runtime: Go source code for the Grumpy runtime library.
  • third_party/ouroboros: Pure Python standard libraries copied from the Ouroboros project.
  • third_party/pypy: Pure Python standard libraries copied from PyPy.
  • third_party/stdlib: Pure Python standard libraries copied from CPython.
  • tools: Transcompilation and utility binaries.

Contact

Questions? Comments? Drop us a line at [email protected] or join our Gitter channel

Issues
  • Build gets into strange state when first invocation uses wrong version of Python

    Build gets into strange state when first invocation uses wrong version of Python

    [@localhost grumpy]$ make build/src/grumpy/lib/itertools/module.go:5: can't find import: "grumpy/lib/sys" make: *** [build/pkg/linux_amd64/grumpy/lib/itertools.a] Error 1

    duplicate 
    opened by yyh-graphsql 43
  • Tag `attr_mode:

    Tag `attr_mode:"rw"` to read-write native attributes

    Use tag attr_mode:"rw" to allow native attributes to be changed by Python code. The attribute should be public on Golang (start with a Capital).

    • makeStructFieldDescriptor needs a last boolean argument. Please provide false for private fields.
    • When the new argument is true and the struct field is public, it can be changed (set) via Python
    • Will not be locked on changes. If locking is needed, do it in Python or via a custom Property setter
    opened by alanjds 32
  • Replace use of builtin ast module with pythonparser.

    Replace use of builtin ast module with pythonparser.

    pythonparser is a high quality Python parser written in Python that produces an AST very similar to the one produced by the standard ast module. There are three very appealing qualities to switching Grumpy to pythonparser:

    1. It's pure Python so there's a reasonable chance of getting it to build under Grumpy in the near future, whereas the ast module, being written in C, is unlikely to be work anytime soon.

    2. The AST produced by pythonparser contains detailed span location data. The standard ast module provides line numbers only and not for all nodes. pythonparser provides column data, sub-node locations, locations for comments and more.

    3. Experimenting with custom syntax (e.g. to properly support importing Go packages with arbitrary names, like from __go__ import "github.com/labstack/echo") will become much simpler.

    I suspect pythonparser is quite a bit slower than the ast module but it doesn't seem to matter since Go compilation dominates. Timings for make test were 9m35.968s under the ast module and 10m7.250s under pythonparser.

    There are a few minor outstanding issues with the library I've filed. These seem relatively straightforward to fix and the maintainer is open to PRs. Overall I'm very satisfied with the quality and completeness of the library.

    opened by trotterdylan 12
  • Implementation of raw_input()

    Implementation of raw_input()

    I implemented a simplified version of raw_input().

    Please review on focusing these. First, It might be not 100% compatible with CPython and pypy's raw_input(). and also need more implementation to handle error cases.

    Second, I need some guide for writing unit test codes to check it is well working for raw_input(). I don't know how to test user's input from keyboard.

    Third, prompt printing is slower when running with go run *.go so if you have solutions then please give me a feedback.

    Related PR= #247

    opened by corona10 11
  • STD lib roadmap

    STD lib roadmap

    Very nice project.

    I am a golang programmer and it would be nice to have many of the machine learning python code available to golang programmers.

    Is there a CI build system to build and test the pythons libs yet ?

    opened by joeblew99 11
  • Add support for the `callable` builtin

    Add support for the `callable` builtin

    Implement the callable as defined here:

    • https://docs.python.org/2.7/library/functions.html#callable

    make precommit passed.

    Signed-off-by: Cholerae Hu [email protected]

    opened by choleraehyq 11
  • Added math module

    Added math module

    This request contains a shim package that wraps Go functions from the math package in functions that are identical to those in the Python math module. There are three comments that begin with "NOTE", that I think require input from others and probably some corrections.

    opened by lalaithion 10
  • Create Standard Library checklist.md

    Create Standard Library checklist.md

    #!/usr/bin/env python3
    import bs4, requests
    url = 'https://docs.python.org/2.7/py-modindex.html'
    soup = bs4.BeautifulSoup(requests.get(url).content, 'html.parser')
    for module in soup.find_all('code', class_='xref'):
        if '.' not in module.string:
            print('- [ ] ' + module.string.replace('__', '\_\_', 1))
    
    opened by cclauss 9
  • Add _struct, _md5 to third_party/pypy

    Add _struct, _md5 to third_party/pypy

    Added _struct, _md5, md5(wrapper to _md5) to third_party/pypy.

    Hard-coded byteorder = 'little' on sys module for now, which is needed by _struct Let me know if I better hardcode inside _struct module.

    Note that I cannot add wrapper struct.py because struct is a keyword in go as you know.

    opened by S-YOU 9
  • Homebrew does not have a formula for Grumpy

    Homebrew does not have a formula for Grumpy

    https://github.com/Homebrew/homebrew-core/issues/9213

    Does anyone know how the homebrew installer for Mac OSX works? I spent an few hours trying to build a formula to install Grumpy based on these examples but I failed.

    opened by cclauss 8
  • [WIP] Add more test cases from stdlib

    [WIP] Add more test cases from stdlib

    list, dict, set, str, unicode, int, slice

    ./run third_party/stdlib/test/test_list.py
    test_basic (__main__.ListTest) ... ok
    test_identity (__main__.ListTest) ... ok
    test_len (__main__.ListTest) ... ok
    test_overflow (__main__.ListTest) ... ok
    test_truth (__main__.ListTest) ... ok
    
    ----------------------------------------------------------------------
    Ran 5 tests in 0.001133s
    
    OK
    
    opened by S-YOU 8
  • make: *** No rule to make target 'run'.  Stop.

    make: *** No rule to make target 'run'. Stop.

    I installed grumpy using snap (sudo snap install grumpy). I would like to know which exactly is the root directory of the Grumpy source code distribution.

    I'm asking this because I tried to do echo "print 'hello, world'" | make run into /snap/grumpy/ and get error:

    $ echo "print 'hello, world'" | make run
    make: *** No rule to make target 'run'.  Stop.
    

    here my content from directory

    $ ls /snap/grumpy/
    15 current
    
    opened by DiegoBneiNoah 0
  • Rebuild tool for automated refactoring

    Rebuild tool for automated refactoring

    opened by XVilka 0
  • Failed to import some python std libs(e.g. json) in python code and then called in go code

    Failed to import some python std libs(e.g. json) in python code and then called in go code

    My python code is as below

    def feature1(inputJsonString):
        print inputJsonString
        import json
        import sys
        print json.dumps(1)
        return sys.version
    

    and the go main code is as below

    package main
    
    import ... // I have imported all the package under the "$GOPATH/src/__python__" and thus "json" related libs should be imported
    
    func main() {
        f := grumpy.NewRootFrame()
        mods, _ := grumpy.ImportModule(f, "feature1")
        fmt.Printf("mods: %v\n", mods)
        feature1, _ := grumpy.GetAttr(f, mods[0], grumpy.NewStr("feature1"), nil)
        args := f.MakeArgs(1)
        args[0] = grumpy.NewStr(`{"a": 1, "b": "I am string"}`).ToObject()
        result, err := feature1.Call(f, args, nil)
        fmt.Printf("result type: %v\n", reflect.TypeOf(result))
        fmt.Printf("result: %v\n", result.String())
        fmt.Printf("err: %v\n", err)
    }
    

    When I tried to run the go main code, and I got result like this

    result: nil
    err: TypeError('bad operand type for unary -: \'tuple\'',)
    

    I found this error is raised from grumpy.ImportModule function in generated go code feature1.go but I am not clear how this error is raised. Also when I try to import some other python std libs like "sys", and it works fine.

    My Golang version is go1.10.3 darwin/amd64

    opened by always-less 0
Owner
Google
Google ❤️ Open Source
Google
DO NOT USE. Implementation of Python 3.x for .NET Framework that is built on top of the Dynamic Language Runtime.

IronPython 3 IronPython3 is NOT ready for use yet. There is still much that needs to be done to support Python 3.x. We are working on it, albeit slowl

IronLanguages 1.8k May 25, 2022
x2 - a miniminalistic, open-source language created by iiPython

x2 is a miniminalistic, open-source language created by iiPython, inspired by x86 assembly and batch. It is a high-level programming language with low-level, easy-to-remember syntaxes, similar to x86 assembly.

Benjamin 3 Mar 20, 2022
A faster and highly-compatible implementation of the Python programming language. The code here is out of date, please follow our blog

Pyston is a faster and highly-compatible implementation of the Python programming language. Version 2 is currently closed source, but you can find the

null 5k May 20, 2022
MicroPython - a lean and efficient Python implementation for microcontrollers and constrained systems

The MicroPython project This is the MicroPython project, which aims to put an implementation of Python 3.x on microcontrollers and small embedded syst

MicroPython 14.2k May 26, 2022
A faster and highly-compatible implementation of the Python programming language.

Pyston Pyston is a fork of CPython 3.8.8 with additional optimizations for performance. It is targeted at large real-world applications such as web se

null 2k May 17, 2022
Rust syntax and lexical analyzer implemented in Python.

Rust Scanner Rust syntax and lexical analyzer implemented in Python. This project was made for the Programming Languages class at ESPOL (SOFG1009). Me

Joangie Marquez 1 Dec 20, 2021
An implementation of Python in Common Lisp

CLPython - an implementation of Python in Common Lisp CLPython is an open-source implementation of Python written in Common Lisp. With CLPython you ca

Willem Broekema 330 May 3, 2022
The Python programming language

This is Python version 3.10.0 alpha 5 Copyright (c) 2001-2021 Python Software Foundation. All rights reserved. See the end of this file for further co

Python 45k May 22, 2022
x86-64 assembler embedded in Python

Portable Efficient Assembly Code-generator in Higher-level Python (PeachPy) PeachPy is a Python framework for writing high-performance assembly kernel

Marat Dukhan 1.6k May 19, 2022
Pyjion - A JIT for Python based upon CoreCLR

Pyjion Designing a JIT API for CPython A note on development Development has moved to https://github.com/tonybaloney/Pyjion FAQ What are the goals of

Microsoft 1.6k May 21, 2022
The Stackless Python programming language

This is Python version 3.7.0 alpha 4+ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 20

Stackless Python 825 May 26, 2022
A mini implementation of python library.

minipy author = RQDYSGN date = 2021.10.11 version = 0.2 1. 简介 基于python3.7环境,通过py原生库和leetcode上的一些习题构建的超小型py lib。 2. 环境 Python 3.7 2. 结构 ${project_name}

RQDYGSN 2 Oct 26, 2021
Core Python libraries ported to MicroPython

This is a repository of libraries designed to be useful for writing MicroPython applications.

MicroPython 1.6k May 22, 2022
null 203 May 18, 2022
ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS. It currently supports four examples for you to quickly experience the power of ONNX Runtime Web.

Microsoft 41 May 6, 2022
Runtime analysis of code with plotting

Runtime analysis of code with plotting A quick comparison among Python, Cython, and the C languages A Programming Assignment regarding the Programming

Cena Ashoori 2 Dec 24, 2021
Python for .NET is a package that gives Python programmers nearly seamless integration with the .NET Common Language Runtime (CLR) and provides a powerful application scripting tool for .NET developers.

pythonnet - Python.NET Python.NET is a package that gives Python programmers nearly seamless integration with the .NET Common Language Runtime (CLR) a

null 3.1k May 25, 2022
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 3.8k May 18, 2022
Unbearably fast O(1) runtime type-checking in pure Python.

Look for the bare necessities, the simple bare necessities. Forget about your worries and your strife. — The Jungle Book.

null 1.1k May 24, 2022
DO NOT USE. Implementation of Python 3.x for .NET Framework that is built on top of the Dynamic Language Runtime.

IronPython 3 IronPython3 is NOT ready for use yet. There is still much that needs to be done to support Python 3.x. We are working on it, albeit slowl

IronLanguages 1.8k May 25, 2022
Unbearably fast O(1) runtime type-checking in pure Python.

Look for the bare necessities, the simple bare necessities. Forget about your worries and your strife. — The Jungle Book.

beartype 1.2k May 26, 2022
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 3.8k May 20, 2022
Robyn is an async Python backend server with a runtime written in Rust, btw.

Robyn is an async Python backend server with a runtime written in Rust, btw. Python server running on top of of Rust Async RunTime. Installation

Sanskar Jethi 1.4k May 27, 2022
Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph, ICSE 2022

PyCRE Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph, ICSE 2022 Dependencies This project is developed

Websoft@NJU 7 May 6, 2022
Python classes with types validation at runtime.

typedclasses Python classes with types validation at runtime. (Experimental & Under Development) Installation You can install this library using Pytho

Izhar Ahmad 8 Feb 6, 2022
a wrapper around pytest for executing tests to look for test flakiness and runtime regression

bubblewrap a wrapper around pytest for assessing flakiness and runtime regressions a cs implementations practice project How to Run: First, install de

Anna Nagy 1 Aug 5, 2021
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

ONNX Runtime is a cross-platform inference and training machine-learning accelerator. ONNX Runtime inference can enable faster customer experiences an

Microsoft 6.7k May 17, 2022
An executor that loads ONNX models and embeds documents using the ONNX runtime.

ONNXEncoder An executor that loads ONNX models and embeds documents using the ONNX runtime. Usage via Docker image (recommended) from jina import Flow

Jina AI 2 Mar 15, 2022
Lowest memory consumption and second shortest runtime in NTIRE 2022 challenge on Efficient Super-Resolution

FMEN Lowest memory consumption and second shortest runtime in NTIRE 2022 on Efficient Super-Resolution. Our paper: Fast and Memory-Efficient Network T

null 20 May 16, 2022