Previously, due to requirements of inspect.getsource and getting the bytecode first line number correct, CPython had to patch the line number of decorated AST nodes. This changes that so for:

@decorate(
        "multi",
        "line"
        )
def foo(
        a: int,
        b: str,
        ):

        pass

The line number for the foo FunctionDef is 5, not 1. Based on my work in https://github.com/python/cpython/pull/6460

Fixes https://github.com/python/typed_ast/issues/50 and fixes https://github.com/python/mypy/issues/3871

This examples comes from the test suite of black

from typed_ast import ast27

ast27.parse('''#!/usr/bin/env python2.7

def function((_globals, _locals)):
    print _globals, _locals
''')

Results in:

Traceback (most recent call last):

  File "/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-3-46b135e3f4de>", line 1, in <module>
    ast27.parse('''#!/usr/bin/env python2.7

  File "/usr/lib/python3.8/site-packages/typed_ast/ast27.py", line 50, in parse
    return _ast27.parse(source, filename, mode)

  File "<unknown>", line 3
SyntaxError: can't assign to ()

The same code parses fine on python 2.7.18.

This is with typed-ast 1.4.1. It only happens on s390x on Alpine Linux edge.

Let me know if you need more information.

If typeshed would start using Python 3.8 only features, such as positional-only arguments, tools such as mypy that use typed-ast when running under 3.7 or earlier would sometimes be unable to parse those stubs, since typed-ast won't get any post-3.7 features according to current policies. This seems to rule out using 3.8 features in typeshed. Similarly, in case Python 3.9 introduces a new syntax for optional types, for example, it looks like we wouldn't be able to use that syntax in typeshed (at least for several years).

I see a few options that could help with the issue:

1. Work towards establishing a typeshed policy that stubs should only use syntax that is supported by the oldest maintained Python 3+ release (currently that would be 3.5). We actually already use 3.6 syntax in typeshed, but we could make it an exception.
2. Revert to the original policy of backporting all new Python syntax to typed-ast.

Option 1 could become a problem in a few scenarios that don't seem totally implausible:

• An existing keyword stops being a keyword, so older Python parsers won't be able to represent identifiers with that name. This could mean that we can't represent some stdlib module stubs precisely.
• Some new type related syntax gets introduced for which we can't design a good fallback syntax for older Python releases.

@gvanrossum @ilevkivskyi @msullivan What do you think about this?

Not sure if this will works, 3.9 is not yet listed on the indicated page.

Might fix #146 and then also will fix https://github.com/python/mypy/issues/9488

Attempting to import the module as installed from PyPI (x86_64 architecture) on Python 3.9.8 produces the following exception:

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/bin/black", line 5, in <module>
    from black import patched_main
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/black/__init__.py", line 52, in <module>
    from typed_ast import ast3, ast27
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/typed_ast/ast3.py", line 40, in <module>
    from typed_ast import _ast3
ImportError: /home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/typed_ast/_ast3.cpython-39-x86_64-linux-gnu.so: undefined symbol: _PyUnicode_DecodeUnicodeEscape

This appears to have broken between Python 3.9.7 and Python 3.9.8.

@ddfisher @gvanrossum @JukkaL

Here is an implementation of the PEP 526 syntax with minimal changes in typed_ast API. The only externally visible change is:

Assign(expr* targets, expr? value, expr? type_comment)

This requires a tiny change to mypy/fastparse.py, there is an accompanying PR to mypy with this change.

I suggest we officially end of life typed-ast:

• Officially stop support after Python 3.10 (i.e. close #167).
• Add recommendation to use ast on Python 3.8+ in the package's long_description and more prominently in the README.
• Add < 3.11 to python_requires.

We can keep maintaining it until 3.7 (or 3.10) reaches end of life, and improved ast is generally available.

Pretty sure this is the same issue as #126 ~but its really annoying that this occurs for every new python release, especially since the first 3.9 testing release has been out for almost a year and final is supposed to be out in a week.~. Sorry didn't see readme explicitly states that python 3.8+ are not supported.

After commit bd75088ea9b03df1618f1e0b8423a496412ae663 (I guess, because it changes _PyObject_FastCall related code) build fails with this message:

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -Iast3/Include -I/usr/local/include/python3.8 -c ast3/Python/ast.c -o build/temp.linux-x86_64-3.8/ast3/Python/ast.o
    ast3/Python/ast.c:22:1: error: conflicting types for '_PyObject_FastCall'
     _PyObject_FastCall(PyObject *func, PyObject *const *args, int nargs)
     ^~~~~~~~~~~~~~~~~~
    In file included from /usr/local/include/python3.8/abstract.h:837,
                     from /usr/local/include/python3.8/Python.h:147,
                     from ast3/Python/ast.c:6:
    /usr/local/include/python3.8/cpython/abstract.h:142:1: note: previous definition of '_PyObject_FastCall' was here
     _PyObject_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs)
     ^~~~~~~~~~~~~~~~~~

Environment: Docker Alpine 3.9 Python 3.8 (3.8-rc-alpine3.9) Related bug report in Red Hat tracker

Header names conflict with python itself, causing cross compile issues. This change forces use of in-package header files.

Not sure if this solution is a desirable fix for #134, or if there is a better solution available.

This change allows programs and other packages to do:

  import typed_ast
  if typed_ast.version_info < (1, 1):
      print(typed_ast.__version__)

currently the only way to get that version number is by doing

  import pkg_resources
  pkg_resources.require('typed-ast')[0].version)

(pr on mypy, rejected at least partly, because of the difficulty of extracting the typed_ast version number)

typed_ast does not support parsing syntax introduced in Python 3.8 and newer, and never will. Instead, it is recommended to use the stdlib ast module in Python 3.8 and newer, which has been augmented to support extracting type comments and has some support for parsing older versions of Python 3.

This means the only valid use cases for typed_ast are a) parsing when run with Python 3.7 and older, or b) parsing Python 2.

typed_ast will continue to be maintained through Python 3.7 EOL (27 June 2023), but will likely become unmaintained soon thereafter.

Given this, we recommend:

• Switch your project over to using ast when run on Python 3.8 or newer. Avoid installing typed_ast when running on Python 3.8 or newer by using a PEP 508 requirement, see for example.
• If your project supports parsing Python 2, make this support an optional extra, so users do not have to install typed_ast unless they need that support, see for example. Also, y'know, consider dropping Python 2 support.

(To avoid user confusion, please discuss in #170)