Discover and load entry points from installed packages

Not exactly sure where this is coming from, but with version 0.2 (installed with conda), I cannot run juypter nbconvert:

$ jupyter nbconvert
Traceback (most recent call last):
  File ".../bin/jupyter-nbconvert", line 3, in <module>
    from nbconvert.nbconvertapp import main
  File ".../lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 99, in <module>
    class NbConvertApp(JupyterApp):
  File ".../lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 179, in NbConvertApp
    """.format(get_export_names()))
  File ".../lib/python2.7/site-packages/nbconvert/exporters/export.py", line 210, in get_export_names
    sorted(entrypoints.get_group_named('nbconvert.exporters'))
  File "/data/apps/anaconda/envs/work/lib/python2.7/site-packages/entrypoints.py", line 183, in get_group_named
    for ep in get_group_all(group, path=path):
  File ".../lib/python2.7/site-packages/entrypoints.py", line 195, in get_group_all
    if group in config:
TypeError: argument of type 'instance' is not iterable

config is a <ConfigParser.ConfigParser instance at 0x10592a320>.

conda info gives:

nbconvert 4.2.0 py27_0
entrypoints 0.2 py27_0

entrypoint fails to find the files if they are under a path with square brackets as glob tries to interpret them and fails. Using glob.escape(folder) fixes that. See https://github.com/ipython/ipython/issues/11512 for details

One use-case that the import ecosystem intends to support is importable wheels - wheels that much like zip eggs of the past can be added to sys.path and imported. Consider this environment:

draft $ python -m venv env
draft $ env/bin/pip install -q -U pip importlib_metadata
draft $ env/bin/pip download --no-deps keyring
Collecting keyring
  Using cached https://files.pythonhosted.org/packages/85/25/55798660a50bd2fd7a8ef58a4394fd4fbb474e6b87d55a8d3e49e300d2a4/keyring-16.0.0-py2.py3-none-any.whl
  Saved ./keyring-16.0.0-py2.py3-none-any.whl
draft $ env/bin/pip install entrypoints
Collecting entrypoints
  Using cached https://files.pythonhosted.org/packages/cc/8b/4eefa9b47f1910b3d2081da67726b066e379b04ca897acfe9f92bac56147/entrypoints-0.2.3-py2.py3-none-any.whl
Installing collected packages: entrypoints
Successfully installed entrypoints-0.2.3

keyring (which has entry points) is present as an importable wheel but isn't on sys.path (by default).

One can readily see that the entry points are visible using importlib_metadata.

draft $ env PYTHONPATH=./keyring-16.0.0-py2.py3-none-any.whl env/bin/python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib_metadata
>>> list(importlib_metadata.entry_points('keyring')['keyring.backends'])
['kwallet', 'secretservice', 'windows', 'macos']

But with entrypoints, the metadata is missing even though keyring is importable:

draft $ env PYTHONPATH=./keyring-16.0.0-py2.py3-none-any.whl env/bin/python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import keyring
>>> import entrypoints
>>> entrypoints.get_group_all('keyring.backends')
[]

I suggest entrypoints should support loading of entrypoints from importable wheels in addition to eggs.

I fail to install ipywidgets with pip under python 3.6 on Xubuntu 16.04 with the following message:

Collecting entrypoints>=0.2.2 (from nbconvert->notebook>=4.4.1->widgetsnbextension~=2.0.0->ipywidgets)
  Could not find a version that satisfies the requirement entrypoints>=0.2.2 (from nbconvert->notebook>=4.4.1->widgetsnbextension~=2.0.0->ipywidgets) (from versions: )
No matching distribution found for entrypoints>=0.2.2 (from nbconvert->notebook>=4.4.1->widgetsnbextension~=2.0.0->ipywidgets)

If I try to install entrypoints alone:

$ pip3.6 install entrypoints
Collecting entrypoints
  Could not find a version that satisfies the requirement entrypoints (from versions: )
No matching distribution found for entrypoints

/lib/python2.7/site-packages/entrypoints.py:168: DeprecationWarning: You passed a bytestring as `filenames`. This will not work on Python 3. Use `cp.read_file()` or switch to using Unicode strings across the board.
  cp.read(path)

on current master its in https://github.com/takluyver/entrypoints/blob/master/entrypoints.py#L171

The documentation and README seems to largely outsource the discussion of how to actually use this library to the Python Packaging documentation using this link: https://packaging.python.org/en/latest/distributing.html#entry-points

The problem is that it seems that that page has changed, the anchor link is dead and Ctrl+F for "entry" gives nothing. Is there a new link where this information is kept? Might it be a good idea to mirror that in the documentation in some minor way?

I am not really sure how this project relates to the entry_points argument in setuptools.setup, I was hoping the link might clarify that.

Should the entrypoint module handle multiple versions of a single package?

After upgrading a package, it seems that old .dist-info folders are hanging around. Now I'm experience this 2 places, (1) where I'm doing local development on a package (and the package version is updated after every commit: e.g,0.6.2.dev2+f0b8738.dirty meaning 2 revisions past the 0.6.2 tag, and that my local copy has changes). So this means more packages which exaggerates my issue a bit. And (2), I'm also seeing this when I'm doing some unusual package installation stuff pip install PKG -t INSTALL_DIR. So I fully understand why duplicate dist-info folders exist in that case.

What's unclear to me is what should be the normal behavior. I apologize for "packaging" question here, but the reason why I bring it up is that I do see different results between say:

list( pkg_resources.iter_entry_points("ksconf_cmd") )

and

entrypoints.get_group_all("ksconf_cmd")

Based on a quick test, pkg_resources seems to be pulling in the most recent version, where as entrypoints seems to be pulling in the older version. (I suppose there could also be a lexicographical sort or internal directory listing order factor in here too. Haven't dug that deep.)

So 2 questions:

1.) Is the assumption that pip (or whatever installer tool) is already taking are of duplicate .dist-info folders and only keeping the most recently installed one. 2.) If not, is there a way to get visibility into the multiple distributions using entrypoints?

Hello,

Could you please consider uploading sorce archive (i.e. *.tar.gz or *.zip) on pypi, so that "entrypoints" can be installed as an egg ? I am using buildout to install "ipython" that requires "nbconvert" and recent "nbconvert" requires "entrypoints".

Thanks in advance !

Hi,

jupyter-1.0.0 metapackage has a dependency on entrypoints

Trying to install jupyter issue an error on entrypoints:

Searching for entrypoints
Reading https://pypi.python.org/simple/entrypoints/
No local packages or download links found for entrypoints
error: Could not find suitable distribution for Requirement.parse('entrypoints')

Forcing to the whl file does no better:

Processing entrypoints-0.2.2-py2.py3-none-any.whl
error: Couldn't find a setup script in /tmp/easy_install-psn0idx6/entrypoints-0.2.2-py2.py3-none-any.whl

Bests

@takluyver

Since this package is now a dependency of nbconvert, would you mind adding conventional setupools installer (setup.py)? Even though it's a trivial package, some tools expect setup.py to exist (e.g. conda-build, tox and many others).

This is just a suggestion and would be up to you of course :)

Edit: I understand you're using your own pypi uploader/wrapper, this is a different question though.

I had heard that the implementation in pkg_resources for discovering entry points was slow, but until today I did not understand exactly why. This documentation might help others like me understand why they might choose entrypoints over pkg_resources.

I hope this documentation does not come across as overly critical of pkg_resources; I just intend for it to explain one motivation for this handle little library.

iter_files_distros(path=None, repeated_distro='first') will fail if one of the elements in sys.path is not a string. For example, my sys.path contained a PosixPath:

['/usr/local/bin', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages', PosixPath('/model_infrastructure')]

PosixPath objects do not implement an rstrip() method, and so the following happens [permalink]:

if path is None:
    path = sys.path

# `path` is now a list containing some strings and in my accidental case, a PosixPath

for folder in path:
    if folder.rstrip('/\\').endswith('.egg'):

# oops

I'll defer to someone with far more Python experience than I have to decide whether this is just a silly user error (i.e. entirely my fault), or something that should be handled within iter_files_distros(...).

Maybe it seems absurd that anything other than a string would find its way into sys.path, but for newcomers to Python (i.e. me, since ~3.5) who have grown up with pathlib, it doesn't feel too out of place. In fact, here's a PR.

For search engines and passers-by:

I had encountered this stacktrace in my Django + Dagster + MLflow + Gensim + Docker project:

AttributeError: 'PosixPath' object has no attribute 'rstrip'
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/utils.py", line 42, in solid_execution_error_boundary
    yield
  File "/usr/local/lib/python3.8/site-packages/dagster/utils/__init__.py", line 382, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/compute_generator.py", line 65, in _coerce_solid_compute_fn_to_iterator
    result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
  File "/model_infrastructure/solids/gensim.py", line 160, in cyclic_word2vec
    experiment = mlflow.get_experiment_by_name(experiment_name)
  File "/usr/local/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 815, in get_experiment_by_name
    return MlflowClient().get_experiment_by_name(name)
  File "/usr/local/lib/python3.8/site-packages/mlflow/tracking/client.py", line 434, in get_experiment_by_name
    return self._tracking_client.get_experiment_by_name(name)
  File "/usr/local/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 128, in get_experiment_by_name
    return self.store.get_experiment_by_name(name)
  File "/usr/local/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 263, in get_experiment_by_name
    response_proto = self._call_endpoint(GetExperimentByName, req_body)
  File "/usr/local/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 55, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "/usr/local/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 163, in call_endpoint
    response = http_request(
  File "/usr/local/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 47, in http_request
    from mlflow.tracking.request_header.registry import resolve_request_headers
  File "/usr/local/lib/python3.8/site-packages/mlflow/tracking/request_header/registry.py", line 39, in <module>
    _request_header_provider_registry.register_entrypoints()
  File "/usr/local/lib/python3.8/site-packages/mlflow/tracking/request_header/registry.py", line 21, in register_entrypoints
    for entrypoint in entrypoints.get_group_all("mlflow.request_header_provider"):
  File "/usr/local/lib/python3.8/site-packages/entrypoints.py", line 236, in get_group_all
    for config, distro in iter_files_distros(path=path):
  File "/usr/local/lib/python3.8/site-packages/entrypoints.py", line 128, in iter_files_distros
    if folder.rstrip('/\\').endswith('.egg'):

It took me a while to find the cause.

My BASE_DIR in Django's settings.py was a pathlib Path object:

from pathlib import Path

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

For whatever reason, I had to append BASE_DIR to sys.path from within a Dagster solid:

sys.path.append(BASE_DIR)

This resulted in a PosixPath object being added to sys.path. As explained above, sys.path is expected presently to hold only strings, and so iter_files_distros(path=None) will just call rstrip(...) on each element without type consideration.

A solution is ensure that a string is appended to sys.path:

sys.path.append(str(BASE_DIR.absolute()))

Migrating this, as recommended, from a downstream flake8 issue. As the issue disappears with the latest flake8 version from the master branch (which has migrated away from entrypoints), it was suggested that this might be an issue with entrypoints. I've provided the reproduction steps from the downstream issue below, please let me know if there's something entrypoint-specific that would be more helpful for your triage.

I've installed a custom flake8 plugin into a fresh Python 3.8.1 virtualenv using the following setup.py and pip install -e .

from setuptools import setup


setup(
    name="flake8-test",
    version="1.0.0",
    description="Flake8 test",
    author="hello",
    entry_points={
        "flake8.extension": ["TST = flake8_test:TestChecker"]
    },
    install_requires=["flake8"],
)

Using the following skeleton (flake8_test.py):

from flake8.options.manager import OptionManager


class TestChecker:
    name = "flake8-test"
    version = "1.0.0"

    def __init__(self, tree, lines):
        self.lines = lines
        self.tree = tree

    def run(self):
        yield (1, 1, "This is a test", TestChecker)

    @classmethod
    def add_options(cls, parser) -> None:
        """Add custom configuration option(s) to flake8."""
        parser.add_option(
            "--test-config-option",
            default=False,
            action="store_true",
            parse_from_config=True,
            help="This is a test"
        )

    @classmethod
    def parse_options(cls, options) -> None:
        """Parse the custom configuration options given to flake8."""
        cls.test_config_option = options.test_config_option

I recieve the following when invoking as a module:

$ python -m flake8 setup.py
Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\Lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\Lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\test\.venv\lib\site-packages\flake8\__main__.py", line 4, in <module>
    cli.main()
  File "C:\test\.venv\lib\site-packages\flake8\main\cli.py", line 18, in main
    app.run(argv)
  File "C:\test\.venv\lib\site-packages\flake8\main\application.py", line 393, in run
    self._run(argv)
  File "C:\test\.venv\lib\site-packages\flake8\main\application.py", line 380, in _run
    self.initialize(argv)
  File "C:\test\.venv\lib\site-packages\flake8\main\application.py", line 364, in initialize
    self.register_plugin_options()
  File "C:\test\.venv\lib\site-packages\flake8\main\application.py", line 205, in register_plugin_options
    self.check_plugins.register_options(self.option_manager)
  File "C:\test\.venv\lib\site-packages\flake8\plugins\manager.py", line 489, in register_options
    list(self.manager.map(register_and_enable))
  File "C:\test\.venv\lib\site-packages\flake8\plugins\manager.py", line 297, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "C:\test\.venv\lib\site-packages\flake8\plugins\manager.py", line 485, in register_and_enable
    call_register_options(plugin)
  File "C:\test\.venv\lib\site-packages\flake8\plugins\manager.py", line 397, in generated_function
    return method(optmanager, *args, **kwargs)
  File "C:\test\.venv\lib\site-packages\flake8\plugins\manager.py", line 216, in register_options
    add_options(optmanager)
  File "C:\test\flake8_test.py", line 18, in add_options
    parser.add_option(
  File "C:\test\.venv\lib\site-packages\flake8\options\manager.py", line 231, in add_option
    self.parser.add_option(option.to_optparse())
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\Lib\optparse.py", line 1008, in add_option
    self._check_conflict(option)
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\Lib\optparse.py", line 980, in _check_conflict
    raise OptionConflictError(
optparse.OptionConflictError: option --test-config-option: conflicting option string(s): --test-config-option

While flake8 behaves as expected:

$ flake8 setup.py
setup.py:13:2: W292 no newline at end of file

For broader context, this is causing CI for my plugin (flake8-annotations) to fail (Azure) since we're invoking flake8 as a module rather than from the defined entry point.