Re-apply type annotations from .pyi stubs to your codebase.

Overview

retype

Latest version on PyPi Supported Python versions check Code style: black Downloads

Re-apply type annotations from .pyi stubs to your codebase.

Usage

Usage: retype [OPTIONS] [SRC]...

  Re-apply type annotations from .pyi stubs to your codebase.

Options:
  -p, --pyi-dir DIRECTORY     Where to find .pyi stubs.  [default: types]
  -t, --target-dir DIRECTORY  Where to write annotated sources.  [default:
                              typed-src]
  -i, --incremental           Allow for missing type annotations in both stubs
                              and the source.
  -q, --quiet                 Don't emit warnings, just errors.
  -a, --replace-any           Allow replacing Any annotations.
  --hg                        Post-process files to preserve implicit byte
                              literals.
  --traceback                 Show a Python traceback on error.
  --version                   Show the version and exit.
  --help                      Show this message and exit.

When you run retype, it goes through all files you passed as SRC, finds the corresponding .pyi files in the types/ directory, and re-applies typing annotations from .pyi to the sources, using the Python 3 function and variable annotation syntax. The resulting combined sources are saved in typed-src/.

You can also pass directories as sources, in which case retype will look for .py files in them recursively.

It's smart enough to do the following:

  • reapply typing imports
  • reapply function argument annotations
  • reapply function return value annotations
  • reapply method argument and return value annotations
  • reapply function-level variable annotations
  • reapply module-level name annotations
  • reapply module-level type aliases
  • reapply class-level field annotations
  • reapply instance-level field annotations
  • validate existing source annotations against the .pyi file
  • validate source function signatures against the .pyi file
  • read function signature type comments in .pyi files
  • read variable type comments in .pyi files
  • consider existing source type comments as annotations
  • remove duplicate type comments from source when annotations are applied
  • normalize remaining type comments in the source to annotations; this is done even if the corresponding .pyi file is missing

List of things to be done

  • add a --backward option to output type comments instead of annotations
  • handle if sys.version_info and sys.platform checks in stubs

Design principles

  • it's okay for a given .pyi file to be incomplete (gradual typing, baby!)
  • it's okay for functions and classes to be out of order in .pyi files and the source
  • it's an error for a function or class to be missing in the source
  • it's an error for a function's signature to be incompatible between the .pyi file and the source
  • it's an error for an annotation in the source to be incompatible with the .pyi file

Known limitations

  • Line numbers in the annotated source will no longer match original source code; this is because re-application of types requires copying typing imports and alias definitions from the .pyi file.
  • While formatting of the original source will be preserved, formatting of the applied annotations might differ from the formatting in .pyi files.
  • The source where type annotations get re-applied cannot use the legacy print statement; that wouldn't work at runtime.
  • Class attribute annotations in __init__() methods are moved verbatim to the respective __init__() method in the implementation. They are never translated into class-level attribute annotations, so if that method is missing, the translation will fail. Similarly, class-level attribute annotations are never applied to __init__() methods.
  • Forward references in .pyi files will only be properly resolved for type aliases and type vars (by inserting them right before they're used in the source). Other forms of forward references will not work in the source code due to out-of-order class and function definitions. Modify your .pyi files to use strings. retype will not automatically discover failing forward references and stringify them.
  • Local variable annotations present in the .pyi file are transferred to the body level of the given function in the source. In other words, if the source defines a variable within a loop or a conditional statement branch, retype will create an value-less variable annotation at the beginning of the function. Use a broad type and constrain types in relevant code paths using assert isinstance() checks.
  • Because of the above, existing source variable annotations and type comments buried in conditionals and loops will not be deduplicated (and mypy will complain that a name was already defined).
  • An async function in the stub will match a regular function of the same name in the same scope and vice versa. This is to enable annotating async functions spelled with @asyncio.coroutine.

Tests

Just run:

tox

OMG, this is Python 3 only!

Relax, you can run retype as a tool perfectly fine under Python 3.6+ even if you want to analyze Python 2 code. This way you'll be able to parse all of the new syntax supported on Python 3 but also effectively all the Python 2 syntax at the same time.

By making the code exclusively Python 3.6+, I'm able to focus on the quality of the checks and re-use all the nice features of the new releases (check out pathlib or f-strings) instead of wasting cycles on Unicode compatibility, etc.

Note: to retype modules using f-strings you need to run on Python 3.6.2+ due to bpo-23894.

License

MIT

Change Log

20.10.0

  • Mark python3.8 and python3.9 compatible

19.9.0

  • add a module entry-point, now you can call it via python -m retype
  • automatically all files excluded by .gitignore on merge of folders
  • support for ast3.num
  • fix a bug that meant the merge was not recursive in paths
  • use setup.cfg based packaging configuration
  • add PEP-517/8 declaration via pyproject.toml
  • include license in both wheel and sdist
  • this projects code base is now formatted with black, import ordered via isort, and uses Azure Pipelines instead of Travis (also testing on Windows and macOs)

17.12.0

  • support --replace-any to allow replacing pre-existing Any annotations without raising errors

  • bugfix: don't re-apply # type: ignore as an annotation if followed by another comment. Original patch by Shannon Zhu.

17.6.3

  • bugfix: don't try to re-apply # type: ignore as a function annotation

  • bugfix: support arbitrary source file encodings, patch by Michael Overmeyer.

  • bugfix: support missing newlines at the end of the file, patch by Michael Overmeyer.

  • bugfix: in --incremental, format default values according to PEP 8 (no spaces around the = sign if the type is missing)

17.6.2

  • bugfix: --incremental didn't work with multiple arguments before

17.6.1

  • support --incremental stub application (i.e. allow for both stubs and the source to be missing annotations for some arguments and/or return value)

17.6.0

  • support async functions

  • support --traceback for getting more information about internal errors

17.4.0

  • first published version

  • date-versioned

Authors

Glued together by Łukasz Langa. Multiple improvements by Michael Overmeyer and Bernat Gabor.

Comments
  • fix missing node normalization to match pytree.convert

    fix missing node normalization to match pytree.convert

    I was going to write this as a bug/question, but then I realized what was going on (I had already stepped through the lib2to3 code in PDB). It looks like this issue is the same as #23 and I added it's example and the test case I was going off of.

    I will probably look at #25 since that looks like it's probably related to convert_annotation(Attribute), but I'm not sure if it would be best to merge them together or split them out separately since they will likely be touching much of the same code.

    fixes: #23, fixes #25

    opened by terencehonles 7
  • Fix for #1

    Fix for #1

    • Added handling for cases where the ParseError refers to lines that don't exist in the source.
    • Specifically handling the missing newline case, because it is especially difficult to diagnose.

    Fixes #1

    opened by movermeyer 5
  • Doesn't support property setters

    Doesn't support property setters

    related to #20 example.py:

    class Testclass:
        value = None
    
        @property
        def someprop(self):
            return self.value
    
        @someprop.setter
        def someprop(self, value):
            self.value = value
    

    example.pyi (generated with stubgen and added -> Any)

    from typing import Any
    
    
    class Testclass:
        value: Any = ...
    
        @property
        def someprop(self) -> Any: ...
    
        @someprop.setter
        def someprop(self, value: Any) -> None: ...
    

    output:

    retype -p . .
    error: /path/example.py: Annotation problem in function 'someprop': 5:1: missing regular argument 'value' in source
    
    opened by Findus23 4
  • Don't error on empty files

    Don't error on empty files

    Empty files can't have any type comments to rewrite, so skip processing them.

    This avoids spurious errors like:

    error: tests/__init__.py: string index out of range
    
    opened by aneeshusa 4
  • Class not found in source with @type_check_only

    Class not found in source with @type_check_only

    Hello,

    I think that classes decorated with @type_check_only should be allowed even if they are not present in the source file (since they are not available at runtime). Using them currently results in a Class not found in source error:

    foo.pyi:

    from typing import type_check_only
    from typing_extensions import Protocol
    
    @type_check_only
    class A(Protocol):
        def __call__(self, u: str) -> int: ...
    
    def f(a: A) -> int: ...
    

    foo.py:

    def f(a):
        return a(u='abc')
    

    retype outputs error: /path/to/foo.py: Class 'A' not found in source.

    I was expecting an output similar to this:

    
    from typing import TYPE_CHECKING
    from typing_extensions import Protocol
    
    if TYPE_CHECKING:
        class A(Protocol):
            def __call__(self, u: str) -> int: ...
    
    def f(a: 'A') -> int:
        return a(u='abc')
    
    opened by ilai-deutel 3
  • quote forward references

    quote forward references

    Unquoted forward references are valid in stub files, but not at runtime; when applying stubs to a module, retype should quote them when necessary to avoid creating a module that fails to run.

    In light of PEP 563, this won't be necessary in Python 3.7, which probably reduces the motivation to implement it. But in any case, this issue should exist to document the issue and whatever resolution is decided on.

    opened by carljm 3
  • Don't accept type-ignore with trailing info

    Don't accept type-ignore with trailing info

    Previously, source code that looks like x = await Foo.get(user_id) # type: ignore # more comment

    Will be annotated to read: x: ignore = await Foo.get(user_id)

    The look-behind was matching the string following "#type: " with "ignore" exactly, and so appending anything after "ignore" will still match to the named capture group.

    This change uses a lookahead to refuse to match "#type: ignore" even if followed by further characters. "#type: ignoreSomething" is still accepted.

    Play with this regex change: https://regex101.com/r/e2trZh/2/

    opened by shannonzhu 3
  • Include LICENSE in MANIFEST.in

    Include LICENSE in MANIFEST.in

    Thanks for the work! Please consider including the LICENSE file in the distribution!

    Background: It would be swell to be able to make this available as a conda package with the conda-forge workflow. Part of the process would include ticking some boxes like clear licensing!

    opened by bollwyvl 2
  • Information: is there any public Python API?

    Information: is there any public Python API?

    Hello, thank you very much for the project, it is a very useful tool!

    I was wondering if there is any public Python API that allows using retype from a Python script itself (instead of relying on shell scripting).

    For example, can I consider the functions retype_file and retype_path part of the public Python API?

    This would allow scripting by installing retype and importing retype_file or retype_path directly instead of having to use a Popen/subprocess-based solution...

    opened by abravalheri 1
  • Mermaid diagram svg sizing issues - fixed height

    Mermaid diagram svg sizing issues - fixed height

    Version: Retype 1.7.0

    I have a large diagram defined in mermaid and it which gets resized to fit. The horizontal size is as I'd expect, however there is a large amount of empty vertical space above and below. See the following image.

    image

    The above page was generated by the following md file. Note I've had to insert a / before the ``` to avoid github rendering it

    ---
    order: 2
    ---
    
    # Blah 1234
    
    ### Lifecycle
    Preceding text
    
    /```mermaid
    
    sequenceDiagram
    	participant User
    	participant GitRepo
    	participant Server
    	participant ResourceA
    	participant ResourceB
    	participant ResourceC
    	
    	User->>GitRepo: Commit changes
    	loop
    		Server->>GitRepo: Check for changes
    	end
    	User->>Server: Manually Trigger
    	note right of Server: Something pipeline scheduled
    	
    	rect rgba(0,255,0,.1)
    		note over Server,ResourceB: Blah blah blah
    		activate ResourceA
    		GitRepo->>ResourceA: Blah blah blah
    		note right of ResourceA: Blah blah blah:<br/>Blah blah blah
    		ResourceA->>Server: Blah blah blah
    		deactivate ResourceA	
    		
    		Server->>ResourceB: something something something
    		activate ResourceB
    		note left of ResourceB: something:<br/>- something something something<br/>- Adds/generates stuff<br/>-Does things<br/>-Does more things<br/>-Hey, more things
    		ResourceB->>Server: something something something something
    		deactivate ResourceB	
    	end
    	
    	note right of Server: Something something something<br/>Something something something
    	
    	rect rgba(0,255,0,.1)
    		note over Server,ResourceB: something something
    		Server->>ResourceB: Does something
    		activate ResourceB
    		note left of ResourceB: Something something something
    		note left of ResourceB: Something something something
    		
    		ResourceB->>Server: Something something something
    		deactivate ResourceB	
    	end
    	
    	note right of Server: something something completed<br/>Schedule something something
    	
    	rect rgba(0,255,0,.1)
    		note over Server,ResourceC: Blah something more
    		Server->>ResourceC: Downloads something artifacts
    		activate ResourceC
    		note left of ResourceC: Blah blah blahBlah blah blah<br/>produces something something
    		ResourceC->>Server: Blah blah blahBlah blah blah
    		deactivate ResourceC	
    	end
    	
    	note over User,Server: Blah blah bla<br/>Blah blah blah
    	
    
    /```
    
    Following text
    

    On further inspection you can see the height is fixed:

    image

    Removing that attribute fixes is for this particular case.

    opened by badbort 1
  • fix string normalization

    fix string normalization

    This change updates annotation comparison to first normalize the original string leaf nodes to their repr form which is what is generated from the ast (the pyi file).

    fixes #24

    opened by terencehonles 1
Releases(22.8.0)
Owner
Łukasz Langa
Python 3.8 & 3.9 Release Manager. llanga on Twitter. Python core developer, hobbyist musician, dad.
Łukasz Langa
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 4.1k Dec 28, 2022
Auto-generate PEP-484 annotations

PyAnnotate: Auto-generate PEP-484 annotations Insert annotations into your source code based on call arguments and return types observed at runtime. F

Dropbox 1.4k Dec 26, 2022
Turn your C++/Java code into a Python-like format for extra style points and to make everyone hates you

Turn your C++/Java code into a Python-like format for extra style points and to make everyone hates you

Tô Đức (Watson) 4 Feb 7, 2022
Tool for translation type comments to type annotations in Python

com2ann Tool for translation of type comments to type annotations in Python. The tool requires Python 3.8 to run. But the supported target code versio

Ivan Levkivskyi 123 Nov 12, 2022
Easy-apply-bot - A LinkedIn Easy Apply bot to help with my job search.

easy-apply-bot A LinkedIn Easy Apply bot to help with my job search. Getting Started First, clone the repository somewhere onto your computer, or down

Matthew Alunni 5 Dec 9, 2022
Mypy stubs, i.e., type information, for numpy, pandas and matplotlib

Mypy type stubs for NumPy, pandas, and Matplotlib This is a PEP-561-compliant stub-only package which provides type information for matplotlib, numpy

Predictive Analytics Lab 194 Dec 19, 2022
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 4.1k Jan 2, 2023
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 4.1k Dec 28, 2022
Runtime type annotations for the shape, dtype etc. of PyTorch Tensors.

torchtyping Type annotations for a tensor's shape, dtype, names, ... Turn this: def batch_outer_product(x: torch.Tensor, y: torch.Tensor) -> torch.Ten

Patrick Kidger 1.2k Jan 3, 2023
A hack for writing switch statements with type annotations in Python.

py_annotation_switch A hack for writing switch statements in type annotations for Python. Why should I use this? You most definitely should not use th

null 6 Oct 17, 2021
Applicator Kit for Modo allow you to apply Apple ARKit Face Tracking data from your iPhone or iPad to your characters in Modo.

Applicator Kit for Modo Applicator Kit for Modo allow you to apply Apple ARKit Face Tracking data from your iPhone or iPad with a TrueDepth camera to

Andrew Buttigieg 3 Aug 24, 2021
Retrieve annotated intron sequences and classify them as minor (U12-type) or major (U2-type)

(intron I nterrogator and C lassifier) intronIC is a program that can be used to classify intron sequences as minor (U12-type) or major (U2-type), usi

Graham Larue 4 Jul 26, 2022
Apply our monocular depth boosting to your own network!

MergeNet - Boost Your Own Depth Boost custom or edited monocular depth maps using MergeNet Input Original result After manual editing of base You can

Computational Photography Lab @ SFU 142 Dec 17, 2022
A GUI based datamoshing apllication for everyone! Apply this glitch to your videos and gifs. Supports all video formats!

A GUI based datamoshing apllication for everyone! Apply this glitch to your videos and gifs. Supports all video formats!

Akascape 131 Dec 31, 2022
Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

Andreas Schmid 1 Feb 7, 2022
Zotero2Readwise - A Python Library to retrieve annotations and notes from Zotero and upload them to your Readwise

Zotero ➡️ Readwise zotero2readwise is a Python library that retrieves all Zotero

Essi Alizadeh 49 Dec 20, 2022
Collection of library stubs for Python, with static types

typeshed About Typeshed contains external type annotations for the Python standard library and Python builtins, as well as third party packages as con

Python 3.3k Jan 2, 2023
PEP-484 stubs for Django

pep484 stubs for Django This package contains type stubs and a custom mypy plugin to provide more precise static types and type inference for Django f

TypedDjango 1.1k Dec 30, 2022
PEP-484 stubs for django-rest-framework

pep484 stubs for Django REST framework Mypy stubs for DRF 3.12.x. Supports Python 3.6, 3.7, 3.8 and 3.9. Installation pip install djangorestframework-

TypedDjango 303 Dec 27, 2022
open source tools to generate mypy stubs from protobufs

mypy-protobuf: Generate mypy stub files from protobuf specs We just released a new major release mypy-protobuf 2. on 02/02/2021! It includes some back

Dropbox 527 Jan 3, 2023