When I added git submodule checking, my Jenkins job run time went from 10 minutes up to 1 hour and 45 minutes. Ouch.

It turns out that every invocation of git submodule ... takes 50 seconds -- but only inside Jenkins. I cannot reproduce this from a command shell on the same machine.

I suspect a different %PATH% setting might be the cause. Perhaps Windows systems take a long time not finding executables on a long %PATH%?

(Meanwhile, I implemented a workaround, but that still leaves me a job that takes 30 minutes instead of the old 10 minutes.)

The existing MANIFEST.in can have been setup to ignore some files or complete directories. check-manifest should not complain that those are in version control but not in the sdist.

If MANIFEST.in exists, we read it and add relevant entries to the IGNORE definition. To properly support the exclude directive I needed to add an IGNORE_REGEXPS definition.

Tests pass on the two supported pythons that I have: 2.7 and 3.3.

When running check-manifest I'm getting the error "python.exe has stopped working. Windows is checking for a solution to the problem" (http://i.imgur.com/DdlZ2p5.png). There isn't an error in the shell window.

Host is Win7x64, 32bit python v3.2 check-manifest is v0.17 project is https://code.launchpad.net/leo-editor, local working copy at rev6315

The command shell log:

[py32] B:\apps\leo> check-manifest .
listing source files under version control: 2058 files and directories
copying source files to a temporary directory
building an sdist: leo-4.11.1.zip: 1987 files and directories

The error pops up at the "copying source files" stage.

#90 seems to suggest that (for example) .github/* will ignore the whole .github folder, once version 0.38 is released.

However, since .github/* does not match the .github folder added by add_directories in VCS.get_versioned_files, there will still be a false positive reported.

Ignoring dir does not work either, as now the files inside the directory don't get ignored. dir* does work I believe, but is dangerous because it will also ignore other folders with the same prefix such as dir2 and everything inside those folders.

Maybe this is intended behavior, but then it's a rather unexpected one from my perspective, as I see no reason to warn about a folder which has no content to warn about?

Currently if I have a git submodule, check-manifest complains about it missing in sdist and VCS at once: https://travis-ci.org/hynek/argon2_cffi/jobs/95785495

_git_ls_files() returns file paths with forward slashes. For files in submodules, add_prefix_to_each() is called to prefix all the submodule files with the relative path to the project root. On Windows, the relative path contains backslashes. Since not every file in the project is in a submodule, the combined list of files ended up with some files that use forward slashes and others that use backslashes.

For project structures where the submodules share a path segment with the project source, this caused add_directories() to create duplicate directory entries in the list of files. The duplicate entries would result in a traceback with a WindowsError when os.mkdir() was called to create the same directory a second time.

See #61 for more details.

This commit ensures the list of files contains only forward slashes before calling add_directories(), eliminating the duplicate directory entries.

I have svn, hg and git installed, but not bzr. Any VCS tests will currently fail due to a missing command. Eg: If a user only uses git, svn, hg and bzr tests will fail.

They should instead be skipped with an appropriate message: "skipping . 'foo' command not installed" or equivalent.

======================================================================
ERROR: test_get_vcs_files (tests.TestBzr)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 770, in test_get_vcs_files
    self._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 756, in _init_vcs
    self.vcs._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 831, in _init_vcs
    self._run('bzr', 'init')
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 724, in _run
    stderr=subprocess.STDOUT)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

======================================================================
ERROR: test_get_vcs_files_added_but_uncommitted (tests.TestBzr)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 781, in test_get_vcs_files_added_but_uncommitted
    self._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 756, in _init_vcs
    self.vcs._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 831, in _init_vcs
    self._run('bzr', 'init')
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 724, in _run
    stderr=subprocess.STDOUT)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

======================================================================
ERROR: test_get_vcs_files_in_a_subdir (tests.TestBzr)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 791, in test_get_vcs_files_in_a_subdir
    self._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 756, in _init_vcs
    self.vcs._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 831, in _init_vcs
    self._run('bzr', 'init')
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 724, in _run
    stderr=subprocess.STDOUT)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

======================================================================
ERROR: test_get_vcs_files_nonascii_filenames (tests.TestBzr)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 803, in test_get_vcs_files_nonascii_filenames
    self._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 756, in _init_vcs
    self.vcs._init_vcs()
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 831, in _init_vcs
    self._run('bzr', 'init')
  File "/mnt/home/user/repos/freebsd/ports/devel/py-check-manifest/work/check-manifest-0.22/tests.py", line 724, in _run
    stderr=subprocess.STDOUT)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

----------------------------------------------------------------------
Ran 87 tests in 8.683s

FAILED (errors=4)

• new command line option --ignore-bad-ideas wich takes a list of patterns like --ignore.
• new config section 'ignore-bad-ideas' similar to 'ignore'

Closes #67

I get something like this on Appveyor:

C:\projects\python-hunter>CALL C:\Python27\Scripts\tox   || EXIT 1 
GLOB sdist-make: C:\projects\python-hunter\setup.py 
check create: C:\projects\python-hunter\.tox\check 
check installdeps: docutils, check-manifest, flake8, readme, pygments 
check installed: bleach==1.4.2,check-manifest==0.30,docutils==0.12,flake8==2.5.1,html5lib==0.9999999,mccabe==0.3.1,pep8==1.5.7,pyflakes==1.0.0,Pygments==2.0.2,readme==0.6.0,six==1.10.0,wheel==0.24.0 
check runtests: PYTHONHASHSEED='632'
check runtests: commands[0] | python setup.py check --strict --metadata --restructuredtext
running check
check runtests: commands[1] | check-manifest C:\projects\python-hunter 
could not run ['git', 'ls-files', '-z']: [WinError 267] The directory name is invalid
ERROR: InvocationError: 'C:\\projects\\python-hunter\\.tox\\check\\Scripts\\check-manifest.EXE C:\\projects\\python-hunter' 
___________________________________ summary ___________________________________
ERROR:   check: commands failed
Command exited with code 1

https://ci.appveyor.com/project/ionelmc/python-hunter/build/master-66/job/fd169ykntce3ugva#L81

check-manifest copies the source files to a temporary directory, and then executes python2 setup.py sdist -d ... from there, but since that is not the git working directory, setuptools_scm can't find the git tag for versioning, resulting in:

$ check-manifest -v
listing source files under version control: 10 files and directories
building an sdist: hello-0.1.2.dev1+ng5ebdc38.tar.gz: 10 files and directories
copying source files to a temporary directory
building a clean sdist
['/home/philipsd6/devel/test_proj/venv/bin/python2', 'setup.py', 'sdist', '-d', '/tmp/philipsd6/check-manifest-DpRbmQ-sdist'] failed (status 1):
Traceback (most recent call last):
  File "setup.py", line 49, in <module>
    'Programming Language :: Python :: 3',
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
    dist.metadata.version = get_version(**value)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 104, in get_version
    version = _do_parse(root, parse)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 82, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/tmp/philipsd6/check-manifest-CP9IEQ-sources'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

How can we use both setuptools_scm for versioning from our git tags, but also validate our MANIFEST.in with check-manifest?

I use setuptools with declarative config (i.e. defined in setup.cfg) and build system in pyproject.toml:

[build-system]
requires = [
    "setuptools >= 40.6.0",
    "wheel",
]
build-backend = "setuptools.build_meta"

This works fine to build wheels and sdist using the PEP 517 hooks (here).

But check-manifest doesn't recognize this as valid Python project.

$ check-manifest
This is not a Python project (no setup.py).

Thanks for this tool, which we're using for h5py. :slightly_smiling_face:

We're running this inside pre-commit on https://pre-commit.ci/ (a dedicated CI platform for pre-commit). At the moment, we have version 0.39, which works. However, pre-commit is keen to upgrade to the latest (0.47), which fails with this message:

check-manifest...........................................................Failed
- hook id: check-manifest
- exit code: 2

['/pc/clone/G2JdS4VyRWO1npwWqn8cWA/py_env-python3/bin/python', '-m', 'build', '--sdist', '.', '--outdir', '/tmp/check-manifest-3l6quvt4-sdist'] failed (status 1):
* Creating venv isolated environment...
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/build-env-dkf2wbiu/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

It looks like if you depend on build[virtualenv], build will prefer virtualenv to venv, which I think should fix this (https://github.com/pypa/build/blob/96f9188ad181907fbd3e0efdf32dd3dc959d39c3/src/build/env.py#L61-L71 ).

It's frustrating that Debian is willing to break things that we can otherwise assume are always available. :disappointed:

After looking around in the Issues related to prune I think the answer is "no", but I thought I'd ask here explicitly though I think this is just (maybe?) related to Issue #115.

A project I work on has prune ** in our MANIFEST.in, which I'll copy here as well:

$ cat MANIFEST.in 
prune **
graft src

include setup.py
include setup.cfg
include LICENSE
include README.rst
include pyproject.toml
include MANIFEST.in
include AUTHORS

global-exclude __pycache__ *.py[cod]

I've recently noticed that while check-manifest is passing when run here, it isn't doing anything as it finds 0 files

$ check-manifest --version
check-manifest version 0.46
$ check-manifest --verbose
listing source files under version control: 0 files and directories
building an sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
copying source files to a temporary directory
building a clean sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
lists of files in version control and sdist match
$ echo $?
0

(Aside: This might be a nice place to fail out on the user)

However, as (according to the Including files in source distributions with MANIFEST.in PyPA guide)

Setuptools also has undocumented support for ** matching zero or more characters including forward slash, backslash, and colon.

build will happily make an sdist and wheel that has the correct file contents.

$ python -m build --outdir dist/ .
$ python -m tarfile --list dist/*.tar.gz | wc -l
105
$ python -m zipfile --list dist/*.whl | wc -l
81

$ python -m build --outdir dist/ .
$ python -m tarfile --list dist/*.tar.gz
pyhf-0.6.4.dev8/ 
pyhf-0.6.4.dev8/AUTHORS 
pyhf-0.6.4.dev8/LICENSE 
pyhf-0.6.4.dev8/MANIFEST.in 
pyhf-0.6.4.dev8/PKG-INFO 
pyhf-0.6.4.dev8/README.rst 
pyhf-0.6.4.dev8/pyproject.toml 
pyhf-0.6.4.dev8/setup.cfg 
pyhf-0.6.4.dev8/setup.py 
pyhf-0.6.4.dev8/src/ 
pyhf-0.6.4.dev8/src/pyhf/ 
pyhf-0.6.4.dev8/src/pyhf/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/_version.py 
pyhf-0.6.4.dev8/src/pyhf/cli/ 
pyhf-0.6.4.dev8/src/pyhf/cli/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/cli/cli.py 
pyhf-0.6.4.dev8/src/pyhf/cli/complete.py 
pyhf-0.6.4.dev8/src/pyhf/cli/infer.py 
pyhf-0.6.4.dev8/src/pyhf/cli/patchset.py 
pyhf-0.6.4.dev8/src/pyhf/cli/rootio.py 
pyhf-0.6.4.dev8/src/pyhf/cli/spec.py 
pyhf-0.6.4.dev8/src/pyhf/compat.py 
pyhf-0.6.4.dev8/src/pyhf/constraints.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/ 
pyhf-0.6.4.dev8/src/pyhf/contrib/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/cli.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/utils.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/ 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/brazil.py 
pyhf-0.6.4.dev8/src/pyhf/data/ 
pyhf-0.6.4.dev8/src/pyhf/data/citation.bib 
pyhf-0.6.4.dev8/src/pyhf/events.py 
pyhf-0.6.4.dev8/src/pyhf/exceptions/ 
pyhf-0.6.4.dev8/src/pyhf/exceptions/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/infer/ 
pyhf-0.6.4.dev8/src/pyhf/infer/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/infer/calculators.py 
pyhf-0.6.4.dev8/src/pyhf/infer/intervals.py 
pyhf-0.6.4.dev8/src/pyhf/infer/mle.py 
pyhf-0.6.4.dev8/src/pyhf/infer/test_statistics.py 
pyhf-0.6.4.dev8/src/pyhf/infer/utils.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/ 
pyhf-0.6.4.dev8/src/pyhf/interpolators/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code0.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code1.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code2.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code4.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code4p.py 
pyhf-0.6.4.dev8/src/pyhf/mixins.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/ 
pyhf-0.6.4.dev8/src/pyhf/modifiers/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/histosys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/lumi.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/normfactor.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/normsys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/shapefactor.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/shapesys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/staterror.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/ 
pyhf-0.6.4.dev8/src/pyhf/optimize/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/common.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/mixins.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_jax.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_minuit.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_numpy.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_pytorch.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_scipy.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_tflow.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/ 
pyhf-0.6.4.dev8/src/pyhf/parameters/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/paramsets.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/paramview.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/utils.py 
pyhf-0.6.4.dev8/src/pyhf/patchset.py 
pyhf-0.6.4.dev8/src/pyhf/pdf.py 
pyhf-0.6.4.dev8/src/pyhf/probability.py 
pyhf-0.6.4.dev8/src/pyhf/readxml.py 
pyhf-0.6.4.dev8/src/pyhf/schemas/ 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/ 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/defs.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/jsonpatch.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/measurement.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/model.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/patchset.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/workspace.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/HistFactorySchema.dtd 
pyhf-0.6.4.dev8/src/pyhf/simplemodels.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/ 
pyhf-0.6.4.dev8/src/pyhf/tensor/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/common.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/jax_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/numpy_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/pytorch_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/tensorflow_backend.py 
pyhf-0.6.4.dev8/src/pyhf/utils.py 
pyhf-0.6.4.dev8/src/pyhf/workspace.py 
pyhf-0.6.4.dev8/src/pyhf/writexml.py 
pyhf-0.6.4.dev8/src/pyhf.egg-info/ 
pyhf-0.6.4.dev8/src/pyhf.egg-info/PKG-INFO 
pyhf-0.6.4.dev8/src/pyhf.egg-info/SOURCES.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/dependency_links.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/entry_points.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/requires.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/top_level.txt
$ python -m zipfile --list dist/*.whl
File Name                                             Modified             Size
pyhf/__init__.py                               2021-09-01 03:41:30         6148
pyhf/_version.py                               2021-09-08 18:47:12          155
pyhf/compat.py                                 2021-09-01 03:41:30         4036
pyhf/constraints.py                            2021-09-01 03:41:30        10274
pyhf/events.py                                 2021-09-01 03:41:30         4401
pyhf/mixins.py                                 2021-09-01 03:41:30         2048
pyhf/patchset.py                               2021-09-01 03:41:30        11310
pyhf/pdf.py                                    2021-09-01 03:41:30        31029
pyhf/probability.py                            2021-09-01 03:41:30         9537
pyhf/readxml.py                                2021-09-07 03:51:54        13087
pyhf/simplemodels.py                           2021-09-01 03:41:30         5764
pyhf/utils.py                                  2021-09-07 03:51:54         5061
pyhf/workspace.py                              2021-09-07 03:51:54        33903
pyhf/writexml.py                               2021-09-07 03:51:54        10807
pyhf/cli/__init__.py                           2021-09-01 03:41:30          387
pyhf/cli/cli.py                                2021-09-01 03:41:30         1299
pyhf/cli/complete.py                           2021-08-18 14:06:12          982
pyhf/cli/infer.py                              2021-09-01 03:41:30         6605
pyhf/cli/patchset.py                           2021-09-01 03:41:30         4318
pyhf/cli/rootio.py                             2021-09-01 03:41:30         3072
pyhf/cli/spec.py                               2021-09-01 03:41:30        12266
pyhf/contrib/__init__.py                       2021-08-18 14:06:12           89
pyhf/contrib/cli.py                            2021-09-01 03:41:30         2051
pyhf/contrib/utils.py                          2021-09-01 03:41:30         2674
pyhf/contrib/viz/__init__.py                   2021-08-18 14:06:12           49
pyhf/contrib/viz/brazil.py                     2021-09-01 03:41:30        13476
pyhf/data/citation.bib                         2021-09-07 03:51:54          699
pyhf/exceptions/__init__.py                    2021-09-01 03:41:30         4784
pyhf/infer/__init__.py                         2021-09-01 03:41:30         8239
pyhf/infer/calculators.py                      2021-09-01 03:41:30        35522
pyhf/infer/intervals.py                        2021-09-01 03:41:30         2647
pyhf/infer/mle.py                              2021-09-01 03:41:30         8281
pyhf/infer/test_statistics.py                  2021-09-01 03:41:30        15867
pyhf/infer/utils.py                            2021-09-01 03:41:30         3336
pyhf/interpolators/__init__.py                 2021-09-01 03:41:30         1644
pyhf/interpolators/code0.py                    2021-09-01 03:41:30         3847
pyhf/interpolators/code1.py                    2021-09-01 03:41:30         4498
pyhf/interpolators/code2.py                    2021-09-01 03:41:30         5349
pyhf/interpolators/code4.py                    2021-09-01 03:41:30        16006
pyhf/interpolators/code4p.py                   2021-09-01 03:41:30         4954
pyhf/modifiers/__init__.py                     2021-09-01 03:41:30         7242
pyhf/modifiers/histosys.py                     2021-09-01 03:41:30         3290
pyhf/modifiers/lumi.py                         2021-09-01 03:41:30         2539
pyhf/modifiers/normfactor.py                   2021-09-01 03:41:30         2727
pyhf/modifiers/normsys.py                      2021-09-01 03:41:30         3312
pyhf/modifiers/shapefactor.py                  2021-09-01 03:41:30         5889
pyhf/modifiers/shapesys.py                     2021-09-07 14:57:40         7041
pyhf/modifiers/staterror.py                    2021-09-01 03:41:30         5274
pyhf/optimize/__init__.py                      2021-09-01 03:41:30         1493
pyhf/optimize/common.py                        2021-09-01 03:41:30         5542
pyhf/optimize/mixins.py                        2021-09-01 03:41:30         7237
pyhf/optimize/opt_jax.py                       2021-09-01 03:41:30         2491
pyhf/optimize/opt_minuit.py                    2021-09-01 03:41:30         5357
pyhf/optimize/opt_numpy.py                     2021-09-01 03:41:30         1051
pyhf/optimize/opt_pytorch.py                   2021-09-01 03:41:30         1353
pyhf/optimize/opt_scipy.py                     2021-09-01 03:41:30         3434
pyhf/optimize/opt_tflow.py                     2021-09-01 03:41:30         1583
pyhf/parameters/__init__.py                    2021-09-01 03:41:30          447
pyhf/parameters/paramsets.py                   2021-09-01 03:41:30         2018
pyhf/parameters/paramview.py                   2021-09-01 03:41:30         2877
pyhf/parameters/utils.py                       2021-09-01 03:41:30         2820
pyhf/schemas/HistFactorySchema.dtd             2021-01-21 19:23:20         6521
pyhf/schemas/1.0.0/defs.json                   2021-09-01 03:41:30        13069
pyhf/schemas/1.0.0/jsonpatch.json              2021-08-18 14:06:12          180
pyhf/schemas/1.0.0/measurement.json            2021-08-18 14:06:12          184
pyhf/schemas/1.0.0/model.json                  2021-08-18 14:06:12          172
pyhf/schemas/1.0.0/patchset.json               2021-08-18 14:06:12          178
pyhf/schemas/1.0.0/workspace.json              2021-08-18 14:06:12          180
pyhf/tensor/__init__.py                        2021-09-01 03:41:30         2041
pyhf/tensor/common.py                          2021-09-01 03:41:30         3278
pyhf/tensor/jax_backend.py                     2021-09-07 03:51:54        18044
pyhf/tensor/numpy_backend.py                   2021-09-02 17:02:22        17054
pyhf/tensor/pytorch_backend.py                 2021-09-02 17:02:22        18220
pyhf/tensor/tensorflow_backend.py              2021-09-07 03:51:54        23064
pyhf-0.6.4.dev8.dist-info/LICENSE              2021-09-08 18:47:12        10760
pyhf-0.6.4.dev8.dist-info/METADATA             2021-09-08 18:47:12        21377
pyhf-0.6.4.dev8.dist-info/WHEEL                2021-09-08 18:47:12           92
pyhf-0.6.4.dev8.dist-info/entry_points.txt     2021-09-08 18:47:12           39
pyhf-0.6.4.dev8.dist-info/top_level.txt        2021-09-08 18:47:12            5
pyhf-0.6.4.dev8.dist-info/RECORD               2021-09-08 18:47:12         6457

Given Issue #115, does this mean check-manifest is incompatible with this syntax and to use check-manifest users should manually add / exclude all top level files?

cc @alexander-held

For e.g. https://github.com/jugmac00/flask-reuploaded (but also for my other projects), I notice an increased runtime for the checks.

The biggest jump was from version 0.40 to 0.41, but it looks like it is getting slower with each release...

I do run check-manifest via pre-commit, but I could also reproduce the problem with a standalone version of check-manifest.

I only had a quick look and looks like the pyproject.toml support could have introduced the degraded performance. The runtime difference between 0.45 and 0.46 can be hardly explained by the changelog.

no check-manifest

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
bandit...................................................................Passed

real    0m1,113s
user    0m1,760s
sys     0m0,245s



0.40

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m1,577s


0.41

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m4,366s
user    0m4,471s
sys     0m0,511s


0.42

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m4,401s
user    0m4,557s
sys     0m0,456s

0.43

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m4,743s
user    0m4,808s
sys     0m0,503s

0.44

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

0.45

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m8,464s
user    0m8,068s
sys     0m0,878s

0.46

❯ time pre-commit run --all-files
isort....................................................................Passed
Debug Statements (Python)................................................Passed
flake8...................................................................Passed
pyupgrade................................................................Passed
check-python-versions....................................................Passed
check-manifest...........................................................Passed
bandit...................................................................Passed

real    0m10,674s
user    0m10,093s
sys     0m1,118s

tests without pre-commit

0.40

❯ time check-manifest 
lists of files in version control and sdist match

real	0m0,440s
user	0m0,388s
sys	0m0,055s

0.41

❯ time check-manifest 
lists of files in version control and sdist match

real	0m3,311s
user	0m2,848s
sys	0m0,245s

0.45

❯ time check-manifest 
lists of files in version control and sdist match

real	0m9,664s
user	0m8,324s
sys	0m0,812s

0.46

❯ time check-manifest 
lists of files in version control and sdist match

real	0m10,041s
user	0m8,671s
sys	0m0,875s

So, the changelog maybe is not really complete - as even without using pre-commit there is a performance degradation between 0.45 and 0.46 which only changelog entry says pre-commit now uses Python 3.

"missing from VCS" lists files I already excluded via .gitignore; needing to list them a 2nd time in the tool config is waste and more brittle that it'd need to be.

https://github.com/mherrmann/gitignore_parser is a maintained implementation.

check-manifest version 0.45

cf. #60

In 21d396902acf61ffeeaa3d81d075a1f156dbaf42 I added a dependency on distutils.text_file.TextFile() because I thought that's what setuptools used to read MANIFEST.in files.

Later I realized that was wrong and that setuptools has its own code for this. I decided to stay with distutils because the stdlib moves slower, and I thought this would protect check-manifest users from breaking changes in setuptools.

However this comes at the cost of accuracy, and also people are talking about deprecating distutils: https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134

Thanks for providing check-manifest. This is really useful.

However, I ran into a problem with v0.41. Since this version, when a pyproject.tom file is detected which has build-backend specified, then the sdist is built with pep517.build --source instead of the usual setup.py sdist.

However, this does not seem to work with poetry. In my pyproject.tom I have build-backend = "poetry.masonry.api", so check-manifest uses pep517.build --source. However, that command does not seem to care about the MANIFEST.in file, and does not put the files listed there in the sdist.

I'm not sure what's going wrong here. Should build backends generally consider MANIFEST.in and poetry just fails to do this? Or am I supposed to specify my sdist files differently from MANIFEST.in, maybe somewhere in pyproject.toml?