Utilities for refactoring imports in python-like syntax.

Overview

Build Status Azure DevOps coverage pre-commit.ci status

aspy.refactor_imports

Utilities for refactoring imports in python-like syntax.

Installation

pip install aspy.refactor_imports

Examples

aspy.refactor_imports.import_obj

Constructing an import object

>>> from aspy.refactor_imports.import_obj import FromImport
>>> from aspy.refactor_imports.import_obj import ImportImport
>>> FromImport.from_str('from foo import bar').to_text()
'from foo import bar\n'
>>> ImportImport.from_str('import bar as baz').to_text()
'import bar as baz\n'

Splitting an import object

>>> from aspy.refactor_imports.import_obj import ImportImport
>>> obj = ImportImport.from_str('import foo, bar, baz')
>>> [i.to_text() for i in obj.split_imports()]
['import foo\n', 'import bar\n', 'import baz\n']

Sorting import objects

>>> import pprint
>>> from aspy.refactor_imports.import_obj import FromImport
>>> objs = sorted([
    FromImport.from_str('from a import foo'),
    FromImport.from_str('from a.b import baz'),
    FromImport.from_str('from a import bar'),
    FromImport.from_str('from a import bar as buz'),
    FromImport.from_str('from a import bar as baz'),
])
>>> pprint.pprint([i.to_text() for i in objs])
['from a import bar\n',
 'from a import bar as baz\n',
 'from a import bar as buz\n',
 'from a import foo\n',
 'from a.b import baz\n']
# Or to partition into blocks (even with mixed imports)
>>> import buck.pprint as pprint
>>> from aspy.refactor_imports.import_obj import FromImport
>>> from aspy.refactor_imports.import_obj import ImportImport
>>> from aspy.refactor_imports.sort import sort
>>> partitioned = sort(
    [
        FromImport.from_str('from aspy import refactor_imports'),
        ImportImport.from_str('import sys'),
        FromImport.from_str('from pyramid.view import view_config'),
        ImportImport.from_str('import cached_property'),
    ],
    separate=True,
    import_before_from=True,
))
>>> pprint.pprint(partitioned)
(
    (ImportImport.from_str('import sys\n'),),
    (
        ImportImport.from_str('import cached_property\n'),
        FromImport.from_str('from pyramid.view import view_config\n'),
    ),
    (FromImport.from_str('from aspy import refactor_imports\n'),),
)

aspy.refactor_imports.classify

Classify a module

>>> from aspy.refactor_imports.classify import classify_import
>>> classify_import('__future__')
'FUTURE'
>>> classify_import('aspy')
'APPLICATION'
>>> classify_import('pyramid')
'THIRD_PARTY'
>>> classify_import('os')
'BUILTIN'
>>> classify_import('os.path')
'BUILTIN'

Also as convenient constants

## From aspy.refactor_imports.classify


class ImportType(object):
    __slots__ = ()

    FUTURE = 'FUTURE'
    BUILTIN = 'BUILTIN'
    THIRD_PARTY = 'THIRD_PARTY'
    APPLICATION = 'APPLICATION'

    __all__ = (FUTURE, BUILTIN, THIRD_PARTY, APPLICATION)
Comments
  • Get correct module info for zipimport modules (to then correctly classify them).

    Get correct module info for zipimport modules (to then correctly classify them).

    Do this because there are scenarios where zipped up packages end up being imported, and the import classification logic was marking those as part of the standard library.

    Trying to do full TDD. So opening with just a commit to add tests to hopefully get a nice CI failure first.

    Resolves #46

    opened by shea-parkes 31
  • Multiple imports on the same line

    Multiple imports on the same line

    It seems I need this for the issue I opened yesterday on reorder_python_imports regarding multiple imports on one line.

    Let me know what you think, I guess it's pretty aggressive though :)

    I removed the restriction on ImportImport as well as it is possible and legal to do (Although pep8 disapproves). I seem to have read somewhere that you should follow the pep but it's okay not to, or something like that, somehwere, can't for the life of me find now though

    opened by dinoshauer 10
  • Relative import sorted incorrect.

    Relative import sorted incorrect.

    I am using this, and I noticed the relative import sorted incorrect. Here is the code

    import pprint
    from aspy.refactor_imports.import_obj import FromImport
    
    objs = sorted([
        FromImport.from_str('from a import foo'),
        FromImport.from_str('from a.b import baz'),
        FromImport.from_str('from a import bar'),
        FromImport.from_str('from a import bar as buz'),
        FromImport.from_str('from a import bar as baz'),
        FromImport.from_str('from .a import bar as baz'),
        FromImport.from_str('from . import bar as baz'),
        FromImport.from_str('from .aa.bb import bar as baz'),
    ])
    pprint.pprint([i.to_text() for i in objs])
    

    Excepted:

    [u'from a import bar\n',
     u'from a import bar as baz\n',
     u'from a import bar as buz\n',
     u'from a import foo\n',
     u'from a.b import baz\n',
     u'from .aa.bb import bar as baz\n',
     u'from .a import bar as baz\n',
     u'from . import bar as baz\n',
    ]
    

    Got:

    [u'from . import bar as baz\n',
     u'from .a import bar as baz\n',
     u'from .aa.bb import bar as baz\n',
     u'from a import bar\n',
     u'from a import bar as baz\n',
     u'from a import bar as buz\n',
     u'from a import foo\n',
     u'from a.b import baz\n']
    
    question 
    opened by coldnight 6
  • Standardize handling of builtin imports between Python versions

    Standardize handling of builtin imports between Python versions

    In Python 3.10+, we have access to sys.stdlib_module_names, which lets us know all of the modules that are considered part of the Python stdlib — regardless of whether they are compiled into the current Python interpreter.

    In earlier versions of Python 3, we have only sys.builtin_module_names, which tells us the modules that are compiled into the current interpreter.

    To work around this, we need a solution to tell us whether a module that is not compiled into the current interpreter should be considered part of the stdlib. The previous approach considered all paths inside sys.path, excluding those ending with -packages (to exclude site-packages and dist-packages), to contain only stdlib modules. We also excluded the paths found in the PYTHONPATH environment variable, reasoning that if a path was present in sys.path because it was added by PYTHONPATH, it was a path falling outside the default Python interpreter paths. If we found the module in question within one of these paths, we'd report it as a builtin.

    With this change, we instead use distutils.sysconfig.get_python_lib(standard_lib=True), which directly reports the path used for the standard library (and not third-party extensions).

    Finally, we're making the order of resolution consistent among Python versions. We check, in priority order:

    1. If the module is specified in unclassifiable_application_modules (this is our "manual override")
    2. If the module is in the list of known builtins
    3. If the module can be found in the application path, influenced by the application_directories setting
    4. If the module can be found in the standard library path If none of the above match, we assume the import is a third-party module.

    Under Python 3.10+, we expect that check 4 never matches without check 2 having matched first.

    All of this also allows us to reduce much of the code duplication introduced in bdc924875711a6f7a9239cec92f9cafe4c3ba657.

    Tests are updated to reflect these new cases, and tox.ini adds all the interpreter versions we care about. We're also able to remove a previous version-specific test skip that was necessary due to the version-inconsistent behavior, but no longer performs differently among versions.

    Fixes #155.

    opened by steverice 5
  • Always sort 'from __future__' imports first (alternative)

    Always sort 'from __future__' imports first (alternative)

    Alternative to https://github.com/asottile/aspy.refactor_imports/pull/106 , reworking classify_import to take an import object. This is backwards incompatible, there are a few possible uses out there: https://github.com/search?l=Python&q=classify_import&type=Code .

    Fixes asottile/reorder_python_imports#214 .

    opened by adamchainz 2
  • Fix classification of application as builtin on Windows

    Fix classification of application as builtin on Windows

    This fixes a bug on Windows, where under certain circumstances an application is misclassified as a builtin, due to case-sensitive path comparison on a case-insensitive filesystem. See #52 for more details.

    Closes #52

    Proposed fixes

    • Compute the common path prefix of the module path and the application directory, using os.path.commonpath. Check if this path prefix is the same directory as the application directory, using ~os.path.samefile~ os.path.normcase.

    • Handle the case where the module path is on a different drive than the application directory. The os.path.commonpath function raises ValueError if the paths are on different drives.

    • Use ~os.path.samefile~ os.path.normcase to check if the real and absolute paths of the module are the same directory.

    Tests

    The test case uses an application directory named SRC and a sys.path entry of src. This is slightly different from the real world case described in the issue, but suffices to reproduce the issue. The test case is marked xfail on Linux, because this platform typically uses case-sensitive file systems.

    Surprisingly (at least to me), the test case also needs to be marked xfail on macOS, even though this platform is traditionally case-insensitive. The reason for this is that os.path.commonpath is case-sensitive on macOS (arguably a bug in CPython?).

    There is no test case yet for the branch where module path and application directory are on different drives, because I have not come up with a good way to test this.

    opened by cjolowicz 2
  • Add backports support

    Add backports support

    Since dataclasses doesn't exist before 3.7 so I use a backport library of it https://github.com/ericvsmith/dataclasses to support from python 3.6 to python 3.8. However, classify_import on dataclasses on python 3.6 returns different import ImportType (THIRD_PARTY) from the value that is returned by 3.7 (BUILTIN), so the sort order is different between python versions. I thought that the backport option can pin them as BUILTINs and fix the situation.

    opened by foriequal0 2
  • Behavior inconsistent between Python <3.10 and >=3.10

    Behavior inconsistent between Python <3.10 and >=3.10

    unclassifiable_application_modules

    Under Python 3.10+, the system builtins are checked for the import before anything else.

    Under Python <3.10, unclassifiable_application_modules is checked before anything else.

    As a result, if a builtin module is specified in unclassifiable_application_modules, it will be reported as an application module under Python <3.10 and a builtin under Python 3.10+.

    I'm not sure what the "correct" behavior for these are, but it does seem that both should be consistent among Python versions.

    application_directories

    Furthermore, the difference in find_stdlib under Python <3.10 is different from the base in sys.stdlib_module_names check under 3.10+. If the former is configured with an empty set of application_directories, it will identify an arbitrary module in the current directory as a builtin, while sys.stdlib_module_names does not. Here the behavior seems more clearly correct under Python 3.10+.

    opened by steverice 1
  • distutils classifies as THIRD_PARTY instead of BUILTIN

    distutils classifies as THIRD_PARTY instead of BUILTIN

    I'm using python3.8.

    from aspy.refactor_imports.classify import classify_import
    print(classify_import("distutils"))
    

    Expected: BUILTIN

    Actual: THIRD_PARTY

    Upon a closer look, I see that you specifically force distutils to be classified as THIRD_PARTY though I don't understand the reasoning given it still seems to be part of the standard library or at least seems to be documented as such. Also, I don't see functionality such as distutils.dir_util.copy_tree having moved into setuptools.

        # force distutils to be "third party" after being gobbled by setuptools
        elif base == 'distutils':
            return ImportType.THIRD_PARTY
    
    opened by bluelight773 1
  • Assertion failed: `assert spec.submodule_search_locations is not None`

    Assertion failed: `assert spec.submodule_search_locations is not None`

    I get the following assertion error while running tox -e linting on the pytest repo locally, which runs pre-commit, which runs reorder-python-imports:

    Reorder python imports...................................................Failed
    - hook id: reorder-python-imports
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/bin/reorder-python-imports", line 8, in <module>
        sys.exit(main())
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/reorder_python_imports.py", line 907, in main
        retv |= _fix_file(filename, args)
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/reorder_python_imports.py", line 475, in _fix_file
        new_contents = fix_file_contents(
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/reorder_python_imports.py", line 456, in fix_file_contents
        partitioned = step(partitioned)
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/reorder_python_imports.py", line 366, in apply_import_sorting
        sorted_blocks = sort(import_obj_to_partition.keys(), **sort_kwargs)
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/aspy/refactor_imports/sort.py", line 98, in sort
        imports_partitioned[classify_func(import_obj)].append(import_obj)
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/aspy/refactor_imports/sort.py", line 72, in classify_func
        tp = classify_import(obj.import_statement.module, **kwargs)
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/aspy/refactor_imports/classify.py", line 135, in classify_import
        found, module_path, is_builtin = _get_module_info(
      File "/home/ran/.cache/pre-commit/repoye77qfmi/py_env-python3.10/lib/python3.10/site-packages/aspy/refactor_imports/classify.py", line 103, in _get_module_info
        assert spec.submodule_search_locations is not None
    AssertionError
    

    reorder_python_imports is configured to run like this:

    -   repo: https://github.com/asottile/reorder_python_imports  
        rev: v2.6.0
        hooks:
        -   id: reorder-python-imports
            args: ['--application-directories=.:src', --py37-plus]
    

    This is on Python 3.10. I will try to add more details later if needed.

    opened by bluetech 1
  • aspy.refactor_imports.classify - _get_module_info fails for zipped third party libraries

    aspy.refactor_imports.classify - _get_module_info fails for zipped third party libraries

    There is a special case logic gate in classify.py:

    # special case pypy3 bug(?)
    elif not os.path.exists(spec.origin):
        return True, '(builtin)', True
    

    If you are using a zipped third-party package, then this will falsely identify it as part of the standard library.

    Here's a code snippet showing the value of spec.origin in such a case:

    Python 3.7.5 (default, Oct 31 2019, 15:18:51) [MSC v.1916 64 bit (AMD64)]
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.
    
    In [1]: import importlib.util
    
    In [2]: spec = importlib.util.find_spec('py4j')
    
    In [3]: spec
    Out[3]: ModuleSpec(name='py4j', loader=<zipimporter object "C:\DevApps\spark\spark-2.4.4-bin\python\lib\py4j-0.10.7-src.zip">, origin='C:\\DevApps\\spark\\spark-2.4.4-bin\\python\\lib\\py4j-0.10.7-src.zip\\py4j\\__init__.py', submodule_search_locations=['C:\\DevApps\\spark\\spark-2.4.4-bin\\python\\lib\\py4j-0.10.7-src.zip\\py4j'])
    
    In [4]: spec.origin
    Out[4]: 'C:\\DevApps\\spark\\spark-2.4.4-bin\\python\\lib\\py4j-0.10.7-src.zip\\py4j\\__init__.py'
    

    That spec.origin doesn't exist on the filesystem, so os.path.exists(spec.origin) is False.

    I realize we're using %PythonPath% which isn't really supported, but I do believe this would also be an issue in other scenarios. (Sourcing py4j from a zip included with a Spark distribution is actually a fairly common example I believe.)

    Would it be reasonable to get another logic gate in there to identify the zipped library scenario? I'd be happy to try and make the addition if you like.

    Thanks for the great tool. We're fine with mixing our third-party and first-party imports due to using %PythonPath%. We really appreciate the black compatibility (and the focus on clean diffs).

    opened by shea-parkes 1
Owner
Anthony Sottile
@pre-commit @pytest-dev @tox-dev
Anthony Sottile
Tool for automatically reordering python imports. Similar to isort but uses static analysis more.

reorder_python_imports Tool for automatically reordering python imports. Similar to isort but uses static analysis more. Installation pip install reor

Anthony Sottile 589 Dec 26, 2022
Tools for improving Python imports

imptools Tools for improving Python imports. Installation pip3 install imptools Overview Detailed docs import_path Import a module from any path on th

Danijar Hafner 7 Aug 7, 2022
Convert relative imports to absolute

absolufy-imports A tool and pre-commit hook to automatically convert relative imports to absolute. Installation $ pip install absolufy-imports Usage a

Marco Gorelli 130 Dec 30, 2022
Utilities for pycharm code formatting (flake8 and black)

Pycharm External Tools Extentions to Pycharm code formatting tools. Currently supported are flake8 and black on a selected code block. Usage Flake8 [P

Haim Daniel 13 Nov 3, 2022
Simple Python style checker in one Python file

pycodestyle (formerly called pep8) - Python style guide checker pycodestyle is a tool to check your Python code against some of the style conventions

Python Code Quality Authority 4.7k Jan 1, 2023
Optional static typing for Python 3 and 2 (PEP 484)

Mypy: Optional Static Typing for Python Got a question? Join us on Gitter! We don't have a mailing list; but we are always happy to answer questions o

Python 14.4k Jan 8, 2023
A Python Parser

parso - A Python Parser Parso is a Python parser that supports error recovery and round-trip parsing for different Python versions (in multiple Python

Dave Halter 520 Dec 26, 2022
A simple program which checks Python source files for errors

Pyflakes A simple program which checks Python source files for errors. Pyflakes analyzes programs and detects various errors. It works by parsing the

Python Code Quality Authority 1.2k Dec 30, 2022
Performant type-checking for python.

Pyre is a performant type checker for Python compliant with PEP 484. Pyre can analyze codebases with millions of lines of code incrementally – providi

Facebook 6.2k Jan 4, 2023
A static type analyzer for Python code

pytype - ?? ✔ Pytype checks and infers types for your Python code - without requiring type annotations. Pytype can: Lint plain Python code, flagging c

Google 4k Dec 31, 2022
The strictest and most opinionated python linter ever!

wemake-python-styleguide Welcome to the strictest and most opinionated python linter ever. wemake-python-styleguide is actually a flake8 plugin with s

wemake.services 2.1k Jan 1, 2023
Static type checker for Python

Static type checker for Python Speed Pyright is a fast type checker meant for large Python source bases. It can run in a “watch” mode and performs fas

Microsoft 9.2k Jan 3, 2023
Tool to check the completeness of MANIFEST.in for Python packages

check-manifest Are you a Python developer? Have you uploaded packages to the Python Package Index? Have you accidentally uploaded broken packages with

Marius Gedminas 270 Dec 26, 2022
A python documentation linter which checks that the docstring description matches the definition.

Darglint A functional docstring linter which checks whether a docstring's description matches the actual function/method implementation. Darglint expe

Terrence Reilly 463 Dec 31, 2022
Flake8 plugin that checks import order against various Python Style Guides

flake8-import-order A flake8 and Pylama plugin that checks the ordering of your imports. It does not check anything else about the imports. Merely tha

Python Code Quality Authority 270 Nov 24, 2022
Flake8 extension for checking quotes in python

Flake8 Extension to lint for quotes. Major update in 2.0.0 We automatically encourage avoiding escaping quotes as per PEP 8. To disable this, use --no

Zachary Heller 157 Dec 13, 2022
Check for python builtins being used as variables or parameters

Flake8 Builtins plugin Check for python builtins being used as variables or parameters. Imagine some code like this: def max_values(list, list2):

Gil Forcada Codinachs 98 Jan 8, 2023
flake8 plugin to run black for checking Python coding style

flake8-black Introduction This is an MIT licensed flake8 plugin for validating Python code style with the command line code formatting tool black. It

Peter Cock 146 Dec 15, 2022
Custom Python linting through AST expressions

bellybutton bellybutton is a customizable, easy-to-configure linting engine for Python. What is this good for? Tools like pylint and flake8 provide, o

H. Chase Stevens 249 Dec 31, 2022