For a very large code base ( https://github.com/OpenMDAO/OpenMDAO-Framework ) , I get this in the output

.: L-:- None: pylint - failure Tool pylint failed to run (exception was raised)

.: L-:- None: vulture - failure Tool vulture failed to run (exception was raised)

There are no errors given by either tool as a result.

I tried using pylint and vulture with some smaller codes and it worked fine.

Great tool though!!!

It is quite disappointing when overall project, separate file and folder checks give different results. Here is a little table showing results depended on arguments given.

I've installed by cloning from git and then ran python setup.py install

Traceback (most recent call last):
  File "/usr/local/bin/prospector", line 9, in <module>
    load_entry_point('prospector==0.5.1', 'console_scripts', 'prospector')()
  File "/usr/local/lib/python2.7/dist-packages/prospector-0.5.1-py2.7.egg/prospector/run.py", line 186, in main
    summary, messages = prospector.execute()
  File "/usr/local/lib/python2.7/dist-packages/prospector-0.5.1-py2.7.egg/prospector/run.py", line 131, in execute
    tool.prepare(self.path, self.ignores, self.config, self.adaptors)
  File "/usr/local/lib/python2.7/dist-packages/prospector-0.5.1-py2.7.egg/prospector/tools/pylint/__init__.py", line 83, in prepare
    adaptor.adapt_pylint(linter)
  File "/usr/local/lib/python2.7/dist-packages/prospector-0.5.1-py2.7.egg/prospector/adaptor/common.py", line 8, in adapt_pylint
    linter.load_plugin_modules(['pylint_common'])
  File "/usr/local/lib/python2.7/dist-packages/pylint-1.1.0-py2.7.egg/pylint/lint.py", line 357, in load_plugin_modules
    module = load_module_from_name(modname)
  File "/usr/local/lib/python2.7/dist-packages/logilab_common-0.61.0-py2.7.egg/logilab/common/modutils.py", line 125, in load_module_from_name
    return load_module_from_modpath(dotted_name.split('.'), path, use_sys)
  File "/usr/local/lib/python2.7/dist-packages/logilab_common-0.61.0-py2.7.egg/logilab/common/modutils.py", line 167, in load_module_from_modpath
    mp_file, mp_filename, mp_desc = find_module(part, path)
ImportError: No module named pylint_common

So I tried:

sjuul@development01:~/workspace/oversight$ sudo pip install -U pylint-common
Requirement already up-to-date: pylint-common in /usr/local/lib/python2.7/dist-packages/pylint_common-0.1-py2.7.egg
Requirement already up-to-date: pylint>=1.0 in /usr/local/lib/python2.7/dist-packages/pylint-1.1.0-py2.7.egg (from pylint-common)
Requirement already up-to-date: astroid>=1.0 in /usr/local/lib/python2.7/dist-packages/astroid-1.0.1-py2.7.egg (from pylint-common)
Requirement already up-to-date: pylint-plugin-utils>=0.1 in /usr/local/lib/python2.7/dist-packages/pylint_plugin_utils-0.1.1-py2.7.egg (from pylint-common)
Requirement already up-to-date: logilab-common>=0.53.0 in /usr/local/lib/python2.7/dist-packages/logilab_common-0.61.0-py2.7.egg (from pylint>=1.0->pylint-common)

But that didn't help

For versions 1.1.6.3 and 1.1.6.4 I get following error during our CI runs.

We had no issue (except contradicting dependencies with flake8) before

The related error log:

$ prospector
Messages
========

.
  Line: None
    pylint: failure / Tool pylint failed to run (exception was raised)



Check Information
=================
         Started: 2019-06-13 08:48:04.082400
        Finished: 2019-06-13 08:48:19.840839
      Time Taken: 15.76 seconds
       Formatter: grouped
        Profiles: .prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
      Strictness: from profile
  Libraries Used: 
       Tools Run: pep8, profile-validator, pyflakes, pylint
  Messages Found: 1

ERROR: Job failed: exit status 1

pip check output:

$ cat pipcheck.txt
No broken requirements found.

Have to change our CI pipeline to add output of pip freeze

I was running prospector using python 3.6 and the latest version of prospector, but when attempting to run it, it would just throw the following error:

Traceback (most recent call last):
  File "<redacted>/bin/prospector", line 11, in <module>
    sys.exit(main())
  File "<redacted>/lib/python3.6/site-packages/prospector/run.py", line 173, in main
    prospector.execute()
  File "<redacted>/lib/python3.6/site-packages/prospector/run.py", line 53, in execute
    for tool in self.config.get_tools(found_files):
  File "<redacted>/python3.6/site-packages/prospector/config/__init__.py", line 45, in get_tools
    config_result = tool.configure(self, found_files)
  File "<redacted>/python3.6/site-packages/prospector/tools/pylint/__init__.py", line 185, in configure
    config_messages += self._pylintrc_configure(pylintrc, linter)
  File "<redacted>/lib/python3.6/site-packages/prospector/tools/pylint/__init__.py", line 103, in _pylintrc_configure
    are_plugins_loaded = linter.config_from_file(pylintrc)
  File "<redacted>/lib/python3.6/site-packages/prospector/tools/pylint/linter.py", line 24, in config_from_file
    self.read_config_file(config_file, quiet=True)
TypeError: read_config_file() got an unexpected keyword argument 'quiet'

From reading through the prospector codebase and the pylint codebase, it looks like the line self.read_config_file(config_file, quiet=True) is using the old keyword argument quiet prior to it being updated to be verbose instead within pylint. This can be found in this commit: https://github.com/PyCQA/pylint/commit/49c6da5e7c90f7c53162fff1d8d9a2b6474ee28a#diff-ade2cfcf3e840a993dbd465f517d91f5.

As reported in #179, the pycodestyle/pep8 tool fails to

• check Python files in the root PATH directory
• non-package directories in PATH when driven by prospector.

See the comments to #179 for more discussion.

This PR takes the simple approach of using found_files.iter_module_paths() for explicit files mode and rootpath for "single directory arg" mode (leaving file lookup to pep8/pycodestyle) instead of iter_package_paths().

Comes with extensive tests, also adding a bunch of tests for the general finder functionality.

Edit: PR looks intimidating now judging by the number of commits, but note that the actual changes to the prospector codebase are small - it's mostly test additions... ;-)

Describe the bug Just not working with the last version of PyLint

To Reproduce Steps to reproduce the behavior:

1. Install last version of Prospector and Pylint
2. Run prospector -X in an empty folder

=>

/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
  warnings.warn(
Traceback (most recent call last):
  File "/home/sbrunner/.asdf/installs/python/3.8.6/bin/prospector", line 8, in <module>
    sys.exit(main())
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/prospector/run.py", line 183, in main
    prospector.execute()
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/prospector/run.py", line 69, in execute
    messages += tool.run(found_files)
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/prospector/tools/pylint/__init__.py", line 259, in run
    self._linter.check(self._args)
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 996, in check
    self._check_files(
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1031, in _check_files
    with self._astroid_module_checker() as check_astroid_module:
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1150, in _astroid_module_checker
    checker.open()
  File "/home/sbrunner/.asdf/installs/python/3.8.6/lib/python3.8/site-packages/pylint/checkers/unsupported_version.py", line 51, in open
    self._py36_plus = py_version >= (3, 6)
TypeError: '>=' not supported between instances of 'NoneType' and 'tuple'

Expected behavior No output, no error

Screenshots not applicable

Environment (please complete the following information):

• OS: Ubuntu Linux
• Tool: pylint
• Prospector version: 1.5.1
• Python version: 3.8.6
• Pylint / astroid versions: 2.12.1 / 2.9.0

Additional context python3.8 -m pip freeze:

affine==2.3.0
aiocache==0.11.1
alembic==1.6.5
appdirs==1.4.4
argcomplete==1.12.3
arrow==1.1.0
astor==0.7.1
astroid==2.9.0
attrs==21.2.0
auto-changelog==0.5.3
azure-core==1.15.0
azure-storage-blob==12.9.0
Babel==2.9.1
backcall==0.2.0
bandit==1.7.0
bashcolor==1.1.1
bcrypt==3.2.0
Beaker==1.11.0
binaryornot==0.4.4
black==21.10b0
bleach==3.3.0
boto3==1.17.88
botocore==1.20.88
bottle==0.12.19
c2c.template==2.3.0
-e git+ssh://[email protected]/camptocamp/c2cciutils.git@b41af7949f102cd7206302956b062f95d73ae8da#egg=c2cciutils
c2cgeoform==2.1.23
-e git+ssh://[email protected]/camptocamp/c2cgeoportal.git@4442a8d28c6e0a3e5f8680c73153866f1a090beb#egg=c2cgeoportal_admin&subdirectory=admin
-e git+ssh://[email protected]/camptocamp/c2cgeoportal.git@4442a8d28c6e0a3e5f8680c73153866f1a090beb#egg=c2cgeoportal_commons&subdirectory=commons
-e git+ssh://[email protected]/camptocamp/c2cgeoportal.git@4442a8d28c6e0a3e5f8680c73153866f1a090beb#egg=c2cgeoportal_geoportal&subdirectory=geoportal
-e git+ssh://[email protected]/camptocamp/c2cwsgiutils.git@57746499c8830547fc4d7564277515e130b936ab#egg=c2cwsgiutils
CacheControl==0.12.6
cached-property==1.5.2
cachetools==4.2.2
cachy==0.3.0
cee-syslog-handler==0.6.0
Cerberus==1.3.4
certifi==2021.5.30
cffi==1.14.5
Chameleon==3.9.1
changelog-builder==0.1.7
chardet==4.0.0
cleo==0.8.1
click==7.1.2
click-log==0.3.2
click-plugins==1.1.1
cligj==0.7.2
clikit==0.6.2
codespell==2.1.0
colander==1.8.3
ColanderAlchemy==0.3.4
colorama==0.4.4
commitizen==2.17.12
confuse==1.5.0
conventional==0.5.3
conventional-commit==0.4.2
cookiecutter==1.7.3
crashtest==0.3.1
cryptography==3.4.7
cycler==0.10.0
decli==0.5.2
decorator==4.4.2
deform==2.0.15
defusedxml==0.7.1
-e git+ssh://[email protected]/sbrunner/deskew.git@5158d9e6f2692bea507ab81d7aac8e4b044557df#egg=deskew
distlib==0.3.2
distro==1.5.0
docformatter==1.4
docker==5.0.0
docker-compose==1.29.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.17.1
dodgy==0.2.1
dogpile.cache==1.1.3
dotty-dict==1.3.0
dparse==0.5.1
easygui==0.98.2
EasyProcess==0.3
EditorConfig==0.12.3
entrypoint2==0.2.4
filelock==3.0.12
Fiona==1.8.20
flake8==3.8.4
flake8-polyfill==1.0.2
GeoAlchemy2==0.9.1
geojson==2.5.0
git-changelog==0.4.2
gitdb==4.0.7
GitPython==3.1.18
gobang==0.1.0
google-api-core==1.29.0
google-api-python-client==2.2.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.4
googleapis-common-protos==1.53.0
graphite==0.70
graphyte==1.6.0
greenlet==1.1.0
gunicorn==20.1.0
html5lib==1.1
httplib2==0.19.1
hupper==1.10.3
idna==2.10
imageio==2.9.0
importlib-metadata==4.5.0
influxdb==5.3.1
iniconfig==1.1.1
inotify==0.2.10
invoke==1.6.0
ipython==7.27.0
iso8601==0.1.14
isodate==0.6.0
isoparser==0.3
isort==5.8.0
jedi==0.18.0
jeepney==0.6.0
Jinja2==2.11.3
jinja2-time==0.2.0
jira==3.0.1
jmespath==0.10.0
jsonref==0.2
jsonschema==3.2.0
jsonschema-gentypes==0.9.3
jsonschema2md==0.2.0.post1
keyring==21.8.0
kiwisolver==1.3.1
kubernetes==17.17.0
lark-parser==0.11.3
lazy-object-proxy==1.4.3
lingua==4.14
lnetatmo==2.1.0
lockfile==0.12.2
Mako==1.1.4
mappyfile==0.9.1
Markdown==3.3.4
markdown-table==2020.12.3
MarkupSafe==2.0.1
matplotlib==3.4.2
matplotlib-inline==0.1.2
mccabe==0.6.1
mp3play==0.1.15
msgpack==1.0.2
msrest==0.6.21
mss==6.1.0
munch==2.5.0
mypy==0.901
mypy-extensions==0.4.3
networkx==2.5.1
node-vm2==0.3.7
nose==1.3.7
numpy==1.21.0
oauthlib==3.1.1
opencv-python==4.5.3.56
opentsdb-py==0.6.0
orderedmultidict==1.0.1
OWSLib==0.24.1
packaging==20.9
pandas==1.2.5
papyrus==2.4
paramiko==2.7.2
parso==0.8.2
passwordgenerator==1.4
PasteDeploy==2.1.1
pastel==0.2.1
pathspec==0.9.0
pbr==5.6.0
pdfminer==20191125
pdfminer.six==20200517
pdfplumber==0.5.28
pep517==0.11.0
pep8-naming==0.10.0
peppercorn==0.6
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
pip-shims==0.5.3
pipenv==2021.11.15
pipfile==0.0.2
pkginfo==1.7.0
plaster==1.0
plaster-pastedeploy==0.7
platformdirs==2.2.0
playsound==1.3.0
plette==0.2.3
pluggy==0.13.1
poetry==1.1.8
poetry-core==1.0.4
polib==1.1.1
poyo==0.5.0
prompt-toolkit==3.0.19
prospector==1.5.1
protobuf==3.17.2
psycopg2==2.8.6
psycopg2-binary==2.9.1
ptyprocess==0.7.0
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.20.1
pycodestyle==2.6.0
pycparser==2.20
pycryptodome==3.10.1
pydocstyle==6.1.1
pyflakes==2.2.0
pygame==2.0.1
Pygments==2.9.0
PyGObject==3.40.1
pyguetzli==1.0.9
PyJWT==2.1.0
pylev==1.4.0
pylint==2.12.1
pylint-celery==0.3
pylint-django==2.4.4
pylint-flask==0.6
pylint-plugin-utils==0.6
PyNaCl==1.4.0
pyotp==2.6.0
pyparsing==2.4.7
PyPDF2==1.26.0
pyperclip==1.8.2
pyproj==3.1.0
pyramid==1.10.8
pyramid-beaker==0.8
pyramid-chameleon==0.3
pyramid-debugtoolbar==4.9
pyramid-jinja2==2.8
pyramid-mako==1.1.0
pyramid-multiauth==0.9.0
pyramid-tm==2.4
pyroma==3.2
pyrsistent==0.17.3
pyscreenshot==3.0
pytest==6.2.4
python-dateutil==2.8.1
python-dotenv==0.17.1
python-editor==1.0.4
python-gettext==4.0
python-gitlab==2.9.0
python-magic==0.4.22
python-semantic-release==7.16.2
python-slugify==4.0.1
python-vlc==3.0.12118
pytz==2021.1
pyupgrade==2.19.4
PyWavelets==1.1.1
PyYAML==5.4.1
pyyaml-include==1.2.post2
questionary==1.9.0
rasterio==1.2.6
readme-renderer==29.0
redis==3.5.3
regex==2021.4.4
repoze.lru==0.7
requests==2.25.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
requirements-detector==0.7
requirementslib==1.5.16
rfc3986==1.5.0
rsa==4.7.2
ruamel.yaml==0.17.17
ruamel.yaml.clib==0.2.2
runlike==0.7.0
s3transfer==0.4.2
safety==1.10.3
-e git+ssh://[email protected]/sbrunner/scan-to-paperless.git@3af86d80f908d68f9b389ec3e53ec5c3a78da45e#egg=scan_to_paperless
scikit-build==0.11.1
scikit-image==0.18.2
scipy==1.7.0
-e git+ssh://[email protected]/gberaudo/scrumtools.git@73b05150f2e5ea8007d834a7ec0a1dea3aa6dfd4#egg=scrumtools
SecretStorage==3.3.1
semver==2.13.0
sentry-sdk==1.1.0
setoptconf==0.2.0
setoptconf-tmp==0.3.1
setuptools-scm==6.0.1
Shapely==1.7.1
-e git+ssh://[email protected]/camptocamp/shared_config_manager.git@e594d5215ca3f45094c677c45573018f8fa8f538#egg=shared_config_manager&subdirectory=app
shellingham==1.4.0
six==1.16.0
smmap==4.0.0
snowballstemmer==2.1.0
snuggs==1.4.7
sortedcontainers==2.4.0
SQLAlchemy==1.3.23
stevedore==3.3.0
systemd==0.16.1
termcolor==1.1.0
text-unidecode==1.3
texttable==1.6.3
tifffile==2021.7.2
-e git+ssh://[email protected]/camptocamp/tilecloud.git@3cb56fd269b47678fdb9d3a1b4f9980985f0b68e#egg=tilecloud
-e git+ssh://[email protected]/camptocamp/tilecloud-chain.git@2e00f4e443abf797d6e7290e17dbccd579c0dc35#egg=tilecloud_chain
tk==0.1.0
tokenize-rt==4.1.0
toml==0.10.2
tomli==1.1.0
tomlkit==0.7.0
tqdm==4.61.0
traitlets==5.1.0
transaction==3.0.1
transifex-client==0.14.3
translationstring==1.4
twine==3.6.0
typer==0.3.2
types-pkg-resources==0.1.3
types-PyYAML==5.4.3
types-redis==3.5.16
types-requests==2.25.0
typing-extensions==3.10.0.0
Unidecode==1.2.0
universion==0.1.2
UNKNOWN @ file:///home/external/workspace/helm-mapserver
untokenize==0.1.1
uritemplate==3.0.1
urllib3==1.26.5
venusian==3.0.0
virtualenv==20.4.7
virtualenv-clone==0.5.4
vistir==0.5.2
Wand==0.6.7
wcwidth==0.2.5
webencodings==0.5.1
WebOb==1.8.7
websocket-client==0.59.0
whisper==1.1.8
wrapt==1.12.1
yoga==1.0.0
yoga-image-optimizer==1.0.1
yolk3k==0.9
zipp==3.4.1
zope.deprecation==4.4.0
zope.event==4.5.0
zope.interface==5.4.0
zope.sqlalchemy==1.4
zopflipy==1.5

Prospector runs became very very slow.

Does anybody else notice, that prospector became slower in the last few days. our CI runners do always freshly install prospector and do not version limit the dependencies.

I wanted to know first whether anybody else is encountering such issues?

I am still investigation, whether anything else changed in our CI infrastructure, but didn't find anything so far.

If nobody encounters anything like that within the next few days, then I assume it's not an issue with a prospector dependency and this should just be closed.

If however anybody encounter the same issues it would be interesting to indentify the dependency, that slows down

Using python2, pip install prospector fails with

Collecting prospector
  Using cached https://files.pythonhosted.org/packages/8a/94/d1643eb5460f3accc4609e2fe14ffe9e6dab2b1014b047fccab68b3ee5cb/prospector-1.1.1-py2.py3-none-any.whl
Collecting pylint>=2 (from prospector)
  Could not find a version that satisfies the requirement pylint>=2 (from prospector) (from versions: 0.15.2, 0.16.0, 0.18.0, 0.18.1, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.21.2, 0.21.3, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.25.1, 0.25.2, 0.26.0, 0.27.0, 0.28.0, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.9.0, 1.9.1, 1.9.2, 1.9.3)
No matching distribution found for pylint>=2 (from prospector)

while pip install prospector --no-binary prospector works as expected. This is probably due to incorrect dependency information in the universal wheel uploaded on PyPI.

Hi,

On the latest prospector version (1.1.3), I noticed that I get an error when trying to use flask dependencies

prospector --uses flask results in that error:

Traceback (most recent call last):
  File "/usr/local/bin/prospector", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/prospector/run.py", line 173, in main
    prospector.execute()
  File "/usr/local/lib/python3.6/site-packages/prospector/run.py", line 53, in execute
    for tool in self.config.get_tools(found_files):
  File "/usr/local/lib/python3.6/site-packages/prospector/config/__init__.py", line 41, in get_tools
    config_result = tool.configure(self, found_files)
  File "/usr/local/lib/python3.6/site-packages/prospector/tools/pylint/__init__.py", line 189, in configure
    self._prospector_configure(prospector_config, linter)
  File "/usr/local/lib/python3.6/site-packages/prospector/tools/pylint/__init__.py", line 37, in _prospector_configure
    linter.load_plugin_modules(['pylint_flask'])
  File "/usr/local/lib/python3.6/site-packages/pylint/lint.py", line 519, in load_plugin_modules
    module = modutils.load_module_from_name(modname)
  File "/usr/local/lib/python3.6/site-packages/astroid/modutils.py", line 196, in load_module_from_name
    return load_module_from_modpath(dotted_name.split('.'), path, use_sys)
  File "/usr/local/lib/python3.6/site-packages/astroid/modutils.py", line 239, in load_module_from_modpath
    module = imp.load_module(curname, mp_file, mp_filename, mp_desc)
  File "/usr/local/lib/python3.6/imp.py", line 245, in load_module
    return load_package(name, filename)
  File "/usr/local/lib/python3.6/imp.py", line 217, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/pylint_flask/__init__.py", line 79, in <module>
    MANAGER.register_transform(nodes.From,
AttributeError: module 'astroid.nodes' has no attribute 'From'

Here is my requirements.txt:

prospector == 1.1.3
pylint-flask == 0.5
astroid == 2.0.4
bandit == 1.4.0
safety==1.4.0

Please let me know if you need more details. I needed to specify pylint-flask directly, because even with --uses flask, the prospector did not downloaded pylint-flask automatically (not sure if it was meant to be downloaded automatically in that case).

@Pierre-Sassoulas @carlio Some magic was done for pathlib in Python 3.10 but it seems to no longer be needed in Python 3.11.

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Types of changes

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [ ] My change requires a change to the dependencies
• [ ] I have updated the dependencies accordingly
• [ ] I have read the CONTRIBUTING document.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Related Issue

#562

How Has This Been Tested?

pip install -e . # on prospector folder
prospector --output=pylint sphinx_prompt # on a project that use pyroma (witch failed before)

Checklist:

• [x] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [ ] My change requires a change to the dependencies
• [ ] I have updated the dependencies accordingly
• [x] I have read the CONTRIBUTING document.
• [ ] I have added tests to cover my changes.
• [x] All new and existing tests passed.

Describe the bug

When Pylint runs parallel checks, it pickles the linter object to pass to it's workers which in turn unpickle it. Under Prospector, the found files object includes filters which are closures which include the Prospector config. The setoptconf config object seems to get unpickled without it's _settings instance attribute which results in endless recursion.

To Reproduce

$ git clone -b docker https://github.com/rpatterson/python-project-structure.git
$ cd python-project-structure
$ make .env
$ editor .env  # Add the DOCKER_GID from your /etc/group
$ rm ./.prospector.yaml
$ docker compose run --rm docker bash
$ make build-docker-3.11
$ docker compose run --rm python-project-structure-devel tox exec --no-recreate-pkg --skip-pkg-install -e py311 -- prospector -X --direct-tool-stdout ./src/pythonprojectstructure/
...
Exception in initializer:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/concurrent/futures/process.py", line 235, in _process_worker
    initializer(*initargs)
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/pylint/lint/parallel.py", line 48, in _worker_initialize
    _worker_linter = dill.loads(linter)
		     ^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/dill/_dill.py", line 286, in loads
    return load(file, ignore, **kwds)
	   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/dill/_dill.py", line 272, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
	   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
    obj = StockUnpickler.load(self)
	  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/setoptconf/config.py", line 33, in __getattr__
    if name in self._settings:
	       ^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/setoptconf/config.py", line 33, in __getattr__
    if name in self._settings:
	       ^^^^^^^^^^^^^^
  File "/usr/local/src/python-project-structure/.tox/py311/lib/python3.11/site-packages/setoptconf/config.py", line 33, in __getattr__
    if name in self._settings:
	       ^^^^^^^^^^^^^^
  [Previous line repeated 970 more times]
RecursionError: maximum recursion depth exceeded
...

Expected behavior

Run successfully and report results.

Environment:

• OS: Alpine Linux v3.17 via the docker image (Pop!_OS 22.04 LTS is the host for the example above)
• Tool: pylint
• Prospector version: prospector[with_everything]==1.8.3
• Python version: 3.11.1

Additional context

See the ./requirements/py311/devel.txt file in the repository cloned above for full dependencies and versions.

Is your feature request related to a problem? Please describe. I have found an interesting issue when combining prospector with GitLab CI caching. The CI caching has the requirement that the directories and files to cache must be within the project's root directory. This leads to a .cache folder ending up inside the project's directory during CI/CD runs. Something in this folder is currently causing prospector to autodetect that I am using Django when I am not. I tried to apply ignore paths and ignore patterns to this problem, but it had no effect. Upon inspection of the code, I can see that the autodiscovery is invoked upon construction of the ProspectorConfig object, but none of the CLI arguments go on to affect autodiscovery.

Describe the solution you'd like I'd like to be able to configure autodiscovery to ignore this directory. It is probably best to have separate configuration options so as to not introduce weird interactions with existing usage of ignore paths and ignore patterns.

Describe alternatives you've considered I can disable autodiscovery.

Describe the bug

Running prospector's pyroma tool fails with:

Cannot run tool pyroma as support was not installed.
Please install by running 'pip install prospector[with_pyroma]'

To Reproduce

1. Create a virtual environment and activate it
2. Install prospector with all extras: pip install prospector[with_everything]
3. Run prospector with pyroma on some file: prospector -t pyroma example.py
4. It fails with:

Cannot run tool pyroma as support was not installed.
Please install by running 'pip install prospector[with_pyroma]'

5. Check the environment for pyroma:

$ pip freeze | grep pyroma
pyroma==4.1

Expected behavior

It should run the pyroma tool.

Environment:

• OS: macOS 12.6.1
• Prospector version 1.8.3
• Pyroma version: 4.1
• Python versions: 3.9.15 and 3.10.8

Additional context

I've modified prospector/tools/__init__.py on line 41 to print the import exception and I got:

cannot import name 'DEFAULT_TOOLS' from partially initialized module 'prospector.tools' (most likely due to a circular import) (/tmp/venv/lib/python3.9/site-packages/prospector/tools/__init__.py)

And this seems to be caused because tools/pyroma/__init__.py does import config:

from prospector.config import ProspectorConfig

That in turns import tools:

from prospector.tools import DEFAULT_TOOLS, DEPRECATED_TOOL_NAMES

I've tried removing the ProspectorConfig that is used only for a type hint, and without that Prospector runs pyroma without issues. I'm not sending a PR with this change because you might want to solve the issue in a different way in order to keep the type hints.

Describe the bug

After upgrading from 1.7 to 1.8 with pylint 2.15.8

To Reproduce Steps to reproduce the behavior:

1. Create any empty py file
2. Create .pylintrc file with following text:

[MASTER]
jobs=2

3. prospector
4. ... Never ends

Expected behavior Success check

Screenshots py-spy process analyzing:

Environment (please complete the following information):

• OS: Ubuntu 22.04
• Tool pylint
• Prospector version 1.8.0-1.8.2
• Python version 3.10

Additional context

If jobs is 1 - everything is OK. Does pylint fork makes prospector to be unstoppable?

After upgrading from 1.7 to 1.8, prospector does not respect noqa-comments anymore.

How to reproduce

Create an example file to test, example.py:

testVarible = 1

def a():
    pass

Then use Docker to test with both 1.7.7 and 1.8.2 prospector versions:

# Note that pylint version is set here due to https://github.com/PyCQA/prospector/issues/539
docker run --rm --workdir /usr/src/app -v $(pwd):/usr/src/app python:3.11 bash -c \
    "pip install prospector==1.7.7 pylint==2.14.5 && prospector --strictness high example.py"

docker run --rm --workdir /usr/src/app -v $(pwd):/usr/src/app python:3.11 bash -c \
    "pip install prospector==1.8.2 && prospector --strictness high example.py"

There should be 4 errors in both cases:

example.py
  Line: 1
    pycodestyle: N816 / variable 'testVarible' in global scope should not be mixedCase (col 2)
    pylint: invalid-name / Constant name "testVarible" doesn't conform to UPPER_CASE naming style
  Line: 3
    pycodestyle: E302 / expected 2 blank lines, found 1 (col 1)
    pylint: invalid-name / Function name "a" doesn't conform to snake_case naming style

Now add noqa comments

testVarible = 1  # noqa

def a():  # noqa
    pass

And rerun both Docker commands above.

There is 0 errors for 1.7.7, as expected. With 1.8.2, there are 3 errors for some reason:

example.py
  Line: 1
    pylint: invalid-name / Constant name "testVarible" doesn't conform to UPPER_CASE naming style
  Line: 3
    pycodestyle: E302 / expected 2 blank lines, found 1 (col 1)
    pylint: invalid-name / Function name "a" doesn't conform to snake_case naming style

Whats going on here?

Versions 1.8.0rc0, 1.8.0rc1, 1.8.1 and 1.8.2 all behave the same way. So something changed between 1.7.7 and 1.8.0rc0.