Closes #45

That part of the code seems not to be tested, so I did some manual testing but nothing automated

$ tail {a,b}.py
==> a.py <==
def a(i):
    # type: (int) -> None
    pass

==> b.py <==
def b(i):
    # type: (int) -> None
    pass


$ com2ann a.py b.py
File: a.py
Comments translated for statements on lines: 1
File: b.py
Comments translated for statements on lines: 1


$ tail {a,b}.py
==> a.py <==
def a(i: int) -> None:
    pass

==> b.py <==
def b(i: int) -> None:
    pass

Because Travis CI seems to be on (permanent?) vacation.

• https://travis-ci.org/github/ilevkivskyi/com2ann/builds

Results: https://github.com/cclauss/com2ann/actions

Given...

from typing import Optional
from typing import List

class Pizza:
    def __init__(self, ingredients=None):
        # type: (Optional[List[str]]) -> None
        if ingredients is None:
            self.ingredients = []
        else:
            self.ingredients = ingredients

    def __repr__(self):
        # type: () -> str
        return "This is a Pizza with %s on it" % " ".join(self.ingredients)

    @classmethod
    def pizza_salami(cls):
        # type: () -> Pizza
        return cls(ingredients=["Salami", "Cheese", "Onions"])

... using com2ann factory.py, I receive the following code ...

from typing import Optional
from typing import List

class Pizza:
    def __init__(self, ingredients: Optional[List[str]] = None) -> None:
        if ingredients is None:
            self.ingredients = []
        else:
            self.ingredients = ingredients

    def __repr__(self) -> str:
        return "This is a Pizza with %s on it" % " ".join(self.ingredients)

    @classmethod
    def pizza_salami(cls) -> Pizza:
        return cls(ingredients=["Salami", "Cheese", "Onions"])

... which is no valid Python code any more as Pizza is not yet defined.

Here, the fix is to add quotes around the Pizza return type in the factory method.

As I am brand new to Python typing, maybe there are some other ways to fix this problem.

Maybe this discussion is any help: https://github.com/python/typing/issues/58

I expect that the copy of this utility in the core Python repo is very soon going to be out of date and forgotten. I propose to delete it from there so we don't have to deal with well-meaning core devs making uncoordinated changes there.

Hi @ilevkivskyi - I was wondering if you could release a new version on PyPI? It would be lovely to have the python-requires metadata reflect actual compatibility, and the annotated-type-ignore support would also be great.

I propose that you handle all type comments in the target program. That means with and for statements and all degrees of complexity in assignments. Comments that are ill-formed should be reported and left alone. Otherwise the comments will be stripped and the appropriate annotations inserted to replace them. I have a package scopetools which can provide lots of help. It is still under development. I'd like to work with you in enhancing scopetools and integrating it into com2ann, so please let me know if you are interested. It can find all the type comments and their containing scopes, and the owning scopes for global and nonlocal names. It can split complex target assignments, or type comments, into simpler items (names, attributes, and subscripts).

Here's the basic strategy:

• Function defs and args are fine. However, if there is an annotation for one of the arguments, the type from the function comment is added to the arg, resulting in two annotations, a SyntaxError. Other than that...

Every statement has a target (or multiple targets for an assignment) and a type. With multiple targets, the same type applies to each target. The type is the type comment, parsed as an expression. Annotating the target with the type is the same process as assigning the type to the target. That is, if the target is a packing (a tuple or list of subtargets), the subtargets are annotated with subtypes derived from iterating on the type, taking into account a possible starred subtarget. If the number of subtypes is wrong, this is a malformed type comment, just like an assignment statement. This is done recursively. The result is that you have a set of (sub)targets and (sub)types, where each target is an ast.Name, ast.Subscript, or ast.Attribute. Attribute and Subscript annotations can just be inserted before the original statement. Name annotations are more complicated:

• If the name is declared nonlocal or global (but not at the module level), then a bare annotation (that is, without an assigned value) is placed in the owning scope (the module or some enclosing function) just before the scope definition which contains the original statement. This way, if the scope definition is unreachable, then a type checker will ignore the new annotation.
• If the statement is a simple assignment with one target and no subtargets, then the annotation is inserted after the name.
• Otherwise, a bare annotation is inserted before the statement.

The target for a statement is as follows:

• Assignment: target [ = target ... ] = value # type: typeexpr Each target is a target and annotated with the entiretypeexpr. Examples: w, (x.a, (y[0], z)) = value # type: t1, (t2, (t3, t4)). This annotates w, x.a, y[0], and z with t1, t2, t3, and t4, resp. t = w, (x.a, (y[0], z)) = value # type: t1, (t2, (t3, t4)). Same and annotates t with (t1, (t2, (t3, t4))).

• For: for target in iterable: # type: typeexpr target is the target.

• With: with context [ as target ] [ , context [as target] ... ]: # type: typeexpr

  • With no as target clause, this statement is ignored.
  • With exactly one as target clause, target is the target.
  • With more than one, then target is (target, target [ , target ... ]). The tuple contains only the targets from all the as target clauses that are present.

  Examples: with c1 as t1, c2, c3 as t3.x: # type: type1, type3:. This annotates t1 with type1 and t3.x with type3. with c1 as t1, c2, c3: # type: type1:. This annotates t1 with type1. with c1 as t1: # type: type1:. This annotates t1 with type1. with c1 as t1: # type: type1, type2:. This annotates t1 with (type1, type2). The tuple is not unpacked.

I could be wrong, but following the code, whatever is put in --python-minor-version, as an int, is passed to ast.parse(feature_version=<>).

The doc of ast.parse says:

Also, setting feature_version to a tuple (major, minor) will attempt to parse using that Python version’s grammar. Currently major must equal to 3. For example, setting feature_version=(3, 4) will allow the use of async and await as variable names. The lowest supported version is (3, 4); the highest is sys.version_info[0:2].

I think we should be doing ast.parse(feature_version=(3, python_minor_version)).

Can you confirm ? Thank you !

Thanks for this tool, it's really useful :)

If you have code like this:

    foo = object()

    bar = (
        # Comment which explains why this ignored
        foo.quox   # type: ignore[attribute]
    )  # type: Mapping[str, Distribution]

Then after running com2ann you end up with:

    foo = object()

    bar: Mapping[str, int] = (
        # Comment which explains why this ignored
        foo.quox   # type: ignore[attribute]
    )  # type: Mapping[str, int]

which mypy then complains about due to the double signature.

The intermediate (explanatory) comment doesn't seem to be related, though the type: ignore comment is.

Without the python_requires metadata, the version check does nothing at all when installed from a bdist from e.g. PyPI. After shipping this new version, you may want to "yank" older releases so that naive install commands on older Pythons get an explicit error rather than an incompatible package.

Prompted by my debugging this exact issue when using com2ann on Hypothesis, then planning to build it into shed --refactor and realising that I couldn't rely on importable == usable.

pip install -v  com2amm                        
Using pip 20.2.1 from /home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-yrrlga4n
Created temporary directory: /tmp/pip-req-tracker-p88wxv0i
Initialized build tracking at /tmp/pip-req-tracker-p88wxv0i
Created build tracker: /tmp/pip-req-tracker-p88wxv0i
Entered build tracker: /tmp/pip-req-tracker-p88wxv0i
Created temporary directory: /tmp/pip-install-2skuyzet
1 location(s) to search for versions of com2amm:
* https://pypi.org/simple/com2amm/
Fetching project page and analyzing links: https://pypi.org/simple/com2amm/
Getting page https://pypi.org/simple/com2amm/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/com2amm/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/com2amm/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/com2amm/: 404 Client Error: Not Found for url: https://pypi.org/simple/com2amm/ - skipping
Given no hashes to check 0 links for project 'com2amm': discarding no candidates
ERROR: Could not find a version that satisfies the requirement com2amm (from versions: none)
ERROR: No matching distribution found for com2amm
Exception information:
Traceback (most recent call last):
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 324, in run
    requirement_set = resolver.resolve(
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
    self._populate_link(req)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
    req.link = self._find_requirement_link(req)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
    best_candidate = self.finder.find_requirement(req, upgrade)
  File "/home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 926, in find_requirement
    raise DistributionNotFound(
pip._internal.exceptions.DistributionNotFound: No matching distribution found for com2amm
Removed build tracker: '/tmp/pip-req-tracker-p88wxv0i'

$ pip --version
pip 20.2.1 from /home/sebastian/Repos/ceph/src/pybind/mgr/venv38/lib/python3.8/site-packages/pip (python 3.8)

Looks like pip needs some new mandatory information? Might be related to https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

typeshed already carries type annotations for many "legacy" Python modules. As they evolve it becomes harder and harder to keep them synchronized. This becomes a non-issue if the type annotation is already included with the upstream source code and thus the typeshed is not needed at all. It would be cool if com2ann had a mode to merge the annotations from typeshed into the source code of the module itself.

Bug Report:

Example program:

def f(x: bool
	) -> foo:
	# type: (int) -> str
	pass

Result of com2ann:

def f(x: bool: int
        ) -> str:
        pass

The x: bool: int is a SyntaxError. In this situation, there are various ways to handle it:

• Reject the function type comment. Preferably, output an explanation for the rejection.
• If the function comment and the parameter annotation are the same, leave the parameter as is. In any event, you will have to examine the syntax tree for the function def and find the parameter annotations. If the function comment uses an ellipsis, then this does not apply; only the return type annotation is added.

Bug Report:

When assigning to a nonlocal or global variable, the translation contains syntax errors, because these variables cannot be annotated. Example

class C:
	global x
	x = 2 # type: int

This translates to

class C:
        global x
        x: int = 2

Running this results in

  File "t.py", line 3
    x: int = 2
    ^
SyntaxError: annotated name 'x' can't be global

Similar result if x is nonlocal rather than global.

What to do?

First of all, you need to find the global or nonlocal statement for the variable in its enclosing scope. Then you will know that the variable cannot be annotated at that point. Next, decide how you want to handle it, and then implement it. Some possibilities are:

• Output an error message, with the line number, variable name, and whether it is nonlocal or global. And/or just put it in the "comments skipped" message.
• Remove the type comment and annotate x in the scope which owns x.
  For a global, this is at the module level. For a nonlocal, you have to search enclosing scopes to find the one where x is bound.
  In either case,
  • if x is assigned in the scope, just add the annotation to it.
  • Otherwise, add a bare annotation somewhere in the scope. It should probably be in the same control flow block, so that if the code is unreachable, then a type checker won't use the annotation either.

Some code that might be useful.

I am writing a package which performs analysis tasks related to scopes and namespaces in a python program. It is still a work in progress. However, there is already enough functionality that you could find useful, not only in fixing the present bug but for other tasks as well. You can find it at mrolle45/scopetools. Please let me know if you find it interesting, and I'd like to work with you in incorporating my code into your code. scopetools will create a tree of Scope objects representing the scopes found in the ast.Module tree. The Scope will give you the status of a variable name including which Scope actually owns it. It also points to the ast object for that scope, so you can traverse it. I plan to provide a method to traverse the ast omitting any nested scopes. That way, you can discover all the type comments in the module and the scopes that contain them.

We should support translating type comments like this:

x, y = ...  # type: Tuple[int, int]

Note this may be not totally trivial in r.h.s. is a single expression (not a tuple). In this case we may need to add annotations before assignment, like this

x: int
y: int
x, y = v

When a function in source file contains both type annotation comment and docstring, but in wrong order, com2ann fails with SyntaxError.

Expected behaviour:

• When the type comment is placed after docstring, it should be considered to be a basic inline comment and skipped.
• Com2ann should skip the function and process the rest of the file.
• The comment should be left in place so users can fix it themselves
• Ideally there should an warning in the output informing user there was a function that was skipped due to misplaced type comment.

How to reproduce: Create file test.py:

class Klass:
    def function(self, parameter):
        """Comment"""
        # type: (str) -> str
        return parameter

Call com2ann: com2ann test.py

Output:

File: test.py
SyntaxError in test.py

Potentially, we can translate this:

for i, j in foo(bar):  # type: (int, int)
    ...

to something like this

i: int
j: int
for i, j in foo(bar):
    ...