A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/


Code style: black Build Status Actions Status codecov.io Documentation Status Updates Downloads

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/.

  • bandersnatch >=4.0 supports Linux, MacOSX + Windows
  • Documentation

bandersnatch maintainers are looking for more help! Please refer to our MAINTAINER documentation to see the roles and responsibilities. We would also ask you read our Mission Statement to ensure it aligns with your thoughts for this project.

  • If interested contact @cooperlees

bandersnatch has its dependencies kept up to date by pyup.io!

  • If you'd like to have your dependencies kept up to date in your requirements.txt or setup.cfg, this is the service for you!


The following instructions will place the bandersnatch executable in a virtualenv under bandersnatch/bin/bandersnatch.

  • bandersnatch requires >= Python 3.8.0


This will pull latest build. Please use a specific tag if desired.

  • Docker image includes /bandersnatch/src/runner.py to periodically run a bandersnatch mirror
    • Please /bandersnatch/src/runner.py --help for usage
  • With docker, we recommend bind mounting in a read only bandersnatch.conf
    • Defaults to /conf/bandersnatch.conf
docker pull pypa/bandersnatch
docker run pypa/bandersnatch bandersnatch --help


This installs the latest stable, released version.

python3 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help


  • Run bandersnatch mirror - it will create an empty configuration file for you in /etc/bandersnatch.conf.
  • Review /etc/bandersnatch.conf and adapt to your needs.
  • Run bandersnatch mirror again. It will populate your mirror with the current status of all PyPI packages. Current mirror package size can be seen here: https://pypi.org/stats/
  • A blocklist or allowlist can be created to cut down your mirror size. You might want to Analyze PyPI downloads to determine which packages to add to your list.
  • Run bandersnatch mirror regularly to update your mirror with any intermediate changes.


Configure your webserver to serve the web/ sub-directory of the mirror. For nginx it should look something like this:

    server {
        listen [::1]:80;
        server_name <mymirrorname>;
        root <path-to-mirror>/web;
        autoindex on;
        charset utf-8;
  • Note that it is a good idea to have your webserver publish the HTML index files correctly with UTF-8 as the charset. The index pages will work without it but if humans look at the pages the characters will end up looking funny.

  • Make sure that the webserver uses UTF-8 to look up unicode path names. nginx gets this right by default - not sure about others.

For more information visit out official documentation for instructions on how to use a NGINX example Docker Image.

Cron jobs

You need to set up one cron job to run the mirror itself.

Here's a sample that you could place in /etc/cron.d/bandersnatch:

    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]

This assumes that you have a logger utility installed that will convert the output of the commands to syslog entries.

SystemD Timers are also another alternative in today's modern world.


bandersnatch does not keep much local state in addition to the mirrored data. In general you can just keep rerunning bandersnatch mirror to make it fix errors.

If you want to force bandersnatch to check everything against the master PyPI:

  • run bandersnatch mirror --force-check to move status files if they exist in your mirror directory in order get a full sync.

Be aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.

Other Commands

  • bandersnatch delete --help - Allows you to specify package(s) to be removed from your mirror (dangerous)
  • bandersnatch verify --help - Crawls your repo and fixes any missed files + deletes any unowned files found (dangerous)

Operational notes

Case-sensitive filesystem needed

You need to run bandersnatch on a case-sensitive filesystem.

OS X natively does this OK even though the filesystem is not strictly case-sensitive and bandersnatch will work fine when running on OS X. However, tarring a bandersnatch data directory and moving it to, e.g. Linux with a case-sensitive filesystem will lead to inconsistencies. You can fix those by deleting the status files and have bandersnatch run a full check on your data.

Windows requires elevated prompt

Bandersnatch makes use of symbolic links. On Windows, this permission is turned off by default for non-admin users. In order to run bandersnatch on Windows either call it from an elevated command prompt (i.e. right-click, run-as Administrator) or give yourself symlink permissions in the group policy editor.

Many sub-directories needed

The PyPI has a quite extensive list of packages that we need to maintain in a flat directory. Filesystems with small limits on the number of sub-directories per directory can run into a problem like this:

    2013-07-09 16:11:33,331 ERROR: Error syncing package: [email protected]
    OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'

Specifically we recommend to avoid using ext3. Ext4 and newer does not have the limitation of 32k sub-directories.

Client Compatibility

A bandersnatch static mirror is compatible only to the "static", cacheable parts of PyPI that are needed to support package installation. It does not support more dynamic APIs of PyPI that maybe be used by various clients for other purposes.

An example of an unsupported API is PyPI's XML-RPC interface, which is used when running pip search.

Bandersnatch Mission

The bandersnatch project strives to:

  • Mirror all static objects of the Python Package Index (https://pypi.org/)
  • bandersnatch's main goal is to support the main global index to local syncing only
  • This will allow organizations to have lower latency access to PyPI and save bandwidth on their WAN connections and more importantly the PyPI CDN
  • Custom features and requests may be accepted if they can be of a plugin form
    • e.g. refer to the blocklist and allowlist plugins


If you have questions or comments, please submit a bug report to https://github.com/pypa/bandersnatch/issues/new

  • IRC: #bandersnatch on Freenode (You can use webchat if you don't have an IRC client)

Code of Conduct

Everyone interacting in the bandersnatch project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PSF Code of Conduct.


This client is based on the original pep381client by Martin v. Loewis.

Richard Jones was very patient answering questions at PyCon 2013 and made the protocol more reliable by implementing some PyPI enhancements.

Christian Theune for creating and maintaining bandersnatch for many years!

  • add s3 support

    add s3 support

    I'm using the existing s3path lib to achive the pathlib part.

    This is ~~still under development.~~ ready to use, cheers ! 🍺

    Fixes #437

    enhancement help wanted 
    opened by LeoQuote 22
  • Only download latest versions

    Only download latest versions

    Is there any way to configure bandersnatch to only download the latest version of packages? I don't need nor want 600GB+ of every released version of every package. I'm not running an official mirror, I just want a local copy of latest packages.

    opened by wayneworkman 13
  • Bandersnatch not able to pull tensorflow binary

    Bandersnatch not able to pull tensorflow binary

    Referring to my attached bandersnatch.conf i am not able to pull down tensorflow and tensorflow-gpu binaries in the simple/ folder why is this so? In addition upon a successful pull of the python binaries, the mirror exit gracefully with a code 0.. Upon the next time i run a bandersnatch mirror --force-check again with the contents as per the last run, I get an exit code 1 with TypeError: must be a str or None not int why is this so? pp-pypi-mirror_config_bandersnatch.txt

    bug help wanted needs_external_pr 
    opened by goal86sg 13
  • Dockerimage not working

    Dockerimage not working

    The dockerimage doesn't work, as it seems to not find the 'default.conf':

    docker run -it --rm pypa/bandersnatch:latest
    Running bandersnatch every 3600s
    Config file '/conf/bandersnatch.conf' missing, creating default config.
    Please review the config file, then run 'bandersnatch' again.
    Could not create config file: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf'

    but even when i provide a /conf/bandersnatch.conf the image still fails, now with:

    docker run -it --rm -v $PWD/bandersnatch.conf:/conf/bandersnatch.conf:ro pypa/bandersnatch:latest
    Running bandersnatch every 3600s
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 183, in <module>
      File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 167, in main
        config = bandersnatch.configuration.BandersnatchConfig(
      File "/usr/local/lib/python3.8/site-packages/bandersnatch/configuration.py", line 33, in __call__
        cls._instances[cls] = super().__call__(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/bandersnatch/configuration.py", line 51, in __init__
        with importlib.resources.path("bandersnatch", "default.conf") as config_path:
      File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "/usr/local/lib/python3.8/importlib/resources.py", line 201, in path
        with open_binary(package, resource) as fp:
      File "/usr/local/lib/python3.8/importlib/resources.py", line 91, in open_binary
        return reader.open_resource(resource)
      File "<frozen importlib._bootstrap_external>", line 988, in open_resource
    FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf'
    enhancement help wanted good first issue 
    opened by MikeHofmann 13
  • Filtered Packages Not Being Removed

    Filtered Packages Not Being Removed

    We are running a PyPI mirror with bandersnatch and have recently enabled a number of filters. One goal of our filtering is to reduce the mirror size. However, after enabling the filters, no packages seem to get deleted. How do we go about getting packages cleaned off of the mirror that do not meet our filtering configuration? Does only the verify command delete packages?

    Our config file looks similar to this:

    directory = /srv/pypi
    json = true
    master = https://pypi.org/
    timeout = 10
    workers = 3
    hash-index = false
    stop-on-error = false
    delete-packages = true
    verifiers = 2
    enabled =
    packages =
    packages =
    keep = 3
    opened by daheise 12
  • Problems with 'runner.py'

    Problems with 'runner.py'

    Hello there,

    I noticed these problems in the runner.py while I was using the docker image:


    • Firstly, args doesn't have a conf attribute, it has been defined as config.

    • Secondly, I think, It's not a good idea to hardcode the bandersnatch path. Because even in the Docker image you'll find the bandersnatch inside /usr/local/bin/

    • Thirdly, the interval type should be int not str: https://github.com/pypa/bandersnatch/blob/1b573a141fd6ee4b171c612b1422089b119e544f/src/runner.py#L21

    • Because, this line will show an error if you execute it: https://github.com/pypa/bandersnatch/blob/1b573a141fd6ee4b171c612b1422089b119e544f/src/runner.py#L28

    Even when I solved the above problems in my container, I got another error. My question is, why don't you guys utilize the default configuration when the configuration hasn't been given?

    2020-01-04 16:00:45,222 WARNING: Config file '/conf/bandersnatch.conf' missing, creating default config.
    2020-01-04 16:00:45,222 WARNING: Please review the config file, then run 'bandersnatch' again.
    2020-01-04 16:00:45,222 ERROR: Could not create config file: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf'
    Sleeping for 3599.6514344215393s
    opened by GreatBahram 12
  • Split Docker Build to accept build args to optionally include swift support

    Split Docker Build to accept build args to optionally include swift support

    added dockerignore for slimer build context parametrized dockerfile

    making swift support optional

    nil/bandersnatch 3.9 9f730c999230 2 minutes ago 156MB nil/bandersnatch 3.9-swift 91d50bd7103d 4 minutes ago 280MB

    hopefully addressing https://github.com/pypa/bandersnatch/issues/740 also

    opened by nlaurance-pyie 12
  • Change to re.match usage to re.search

    Change to re.match usage to re.search

    Using re.search instead of re.match allows for non regex formatted matching (static strings) and non-anchored regex matching. This also eliminates the need for .* all over the place when making patterns.

    opened by lps-rocks 11
  • Failed to mirror package via Proxy Server since version 4.0.0

    Failed to mirror package via Proxy Server since version 4.0.0

    Since version 4.0.0 of bandersnatch , I can't use my proxy with environment variables http_proxy or https_proxy

    [email protected]:/# bandersnatch --debug mirror
    2020-05-04 11:36:23,022 DEBUG: Initializing Master's aiohttp ClientSession
    2020-05-04 11:36:23,023 INFO: Status file /srv/pypi/status missing. Starting over.
    2020-05-04 11:36:23,023 INFO: Syncing with https://pypi.org.
    2020-05-04 11:36:23,023 INFO: Current mirror serial: 0
    2020-05-04 11:36:23,024 INFO: Syncing all packages.
    2020-05-04 11:36:23,031 DEBUG: Closing Master's aiohttp ClientSession
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
        return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
      File "/usr/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
        raise exceptions[0]
      File "/usr/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
        sock = await self._connect_sock(
      File "/usr/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
        await self.sock_connect(sock, address)
      File "/usr/lib/python3.8/asyncio/selector_events.py", line 494, in sock_connect
        return await fut
      File "/usr/lib/python3.8/asyncio/selector_events.py", line 526, in _sock_connect_cb
        raise OSError(err, f'Connect call failed {address}')
    ConnectionRefusedError: [Errno 111] Connect call failed ('', 443)
    opened by alebourdoulous 11
  • bug fix: reading either default or system configuration instead of both

    bug fix: reading either default or system configuration instead of both

    I fixed this issue. #95 I think the problem was configuration module tries to read default and system configuration. The problem is when you comment any section in system configuration, it'll read that section through default configuration.

    opened by GreatBahram 10
  • Accept extra parameters provided to pip inside requirement files

    Accept extra parameters provided to pip inside requirement files

    I have some django packages which depend on custom packages that we self host on our internal pypiserver. To make it work we add this first line to requirements.txt:

    --extra-index-url http://xxx.yyy:8080/ --trusted-host xxx.yyy

    This line generates an exception in bandersnatch:

    2021-09-22 09:52:33,508 INFO: Selected storage backend: filesystem (configuration.py:128)
    2021-09-22 09:52:33,509 INFO: Selected compare method: hash (configuration.py:174)
    2021-09-22 09:52:33,610 INFO: considering /requirements/xxx-requirements.txt (allowlist_name.py:111)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/packaging/requirements.py", line 102, in __init__
        req = REQUIREMENT.parseString(requirement_string)
      File "/usr/local/lib/python3.9/site-packages/pyparsing.py", line 1955, in parseString
        raise exc
      File "/usr/local/lib/python3.9/site-packages/pyparsing.py", line 3250, in parseImpl
        raise ParseException(instring, loc, self.errmsg, self)
    pyparsing.ParseException: Expected W:(abcd...), found '-'  (at char 0), (line:1, col:1)
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 223, in <module>
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 219, in main
        return asyncio.run(async_main(args, config))
      File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
        return loop.run_until_complete(main)
      File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
        return future.result()
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 184, in async_main
        return await bandersnatch.mirror.mirror(config)
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 1019, in mirror
        mirror = BandersnatchMirror(
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 201, in __init__
        super().__init__(master=master, workers=workers)
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 43, in __init__
        self.filters = LoadedFilters(load_all=True)
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 151, in __init__
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 180, in _load_filters
        plugin_instance = plugin_class()
      File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 55, in __init__
      File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 30, in initialize_plugin
        self.allowlist_package_names = self._determine_unfiltered_package_names()
      File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 150, in _determine_unfiltered_package_names
        filtered_requirements |= _parse_package_lines(req_fh.readlines())
      File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 127, in _parse_package_lines
        requirement = Requirement(package_line.strip())
      File "/usr/local/lib/python3.9/site-packages/packaging/requirements.py", line 104, in __init__
        raise InvalidRequirement(
    packaging.requirements.InvalidRequirement: Parse error at "'--extra-'": Expected W:(abcd...)```
    bug enhancement 
    opened by mbacicc 4
  • Allow the use of regex or *.txt in allowlist requirements

    Allow the use of regex or *.txt in allowlist requirements

    It would be nice to have the possibility to do this:

    requirements_path = /requirements
    requirements =

    Or provide a regex to find for specific files by changing this function:


    opened by mbacicc 1
  • Move to using minio official docker image

    Move to using minio official docker image

    Right now we use a fork to run minio for CI: https://github.com/pypa/bandersnatch/blob/main/.github/workflows/ci_ubuntu.yml#L18

    Lets see if we can:

    • Pass arguments to use image as is
      • Maybe there is a way today I just didn't find
    • Add a second ENTRYPOINT script that starts an anonymous minio server (Merge to upstream repo)
    • If they won't accept, maybe create our own fork of the main and possibly automate keep the image up to date
    opened by cooperlees 1
  • Many options do not have a configured fallback value even if the documentation says so

    Many options do not have a configured fallback value even if the documentation says so

    What's the issue:

    Some of the configurable knobs don't actually have a fallback value defined even though the documentation explicitly states these values do have a default. Case in point, let's take a gander at the [mirror].timeout value -- it's documented as having a default of 10 seconds:

    The default value for this setting is 10 seconds. https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html#timeout

    buttt looking at the source code it actually doesn't have one: https://github.com/pypa/bandersnatch/blob/3b78e040a330e3308389e3d32d8c13e8b8c41af5/src/bandersnatch/mirror.py#L1012-L1014

    Potential solutions:

    • Fix the lack of fallback=$val for all of the options that are documented having a default
    • Fix the documentation to avoid stating these options have a default

    I personally prefer the first one since having defaults makes sense so fixing the code to actually implement them is a good idea ... although this is a good opportunity to do a configuration adiut and see if any options shouldn't exist / or have a default / should have their default changed.

    Additional context:

    I was notified of this issue from a quick exchange in the Python Discord server: https://discord.com/channels/267624335836053506/463035462760792066/877942046952947722

    bug documentation 
    opened by ichard26 1
  • `verify` operation does not have plugin support

    `verify` operation does not have plugin support

    I have filtered out packages other than linux, but when the verify operation is performed, he will download mac win and other packages that do not meet expectations。

    • cmd:bandersnatch -c /etc/bandersnatch_5v.conf verify --dry-run --json-update --delete
    • conf:
    packages =
    ; vim: set ft=cfg:
    # test to plugins
    enabled =
    platforms =


    2021-07-27 11:45:11,363 INFO: Parsing pyqtpurchasing (verify.py:128)
    2021-07-27 11:45:11,363 INFO: [DRY RUN] Would of grabbed latest json for pyqtpurchasing (verify.py:140)
    2021-07-27 11:45:11,381 INFO: Initialized release plugin blocklist_release, filtering [] (blocklist_name.py:111)
    2021-07-27 11:45:11,388 INFO: https://files.pythonhosted.org/packages/c2/49/f45599d8a9dac9fab18f0074e6fbd38e1262e667075671586daf39eb7a4e/PyQtPurchasing-5.10-5.10.0-cp35.cp36.cp37-abi3-macosx_10_6_intel.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,409 INFO: https://files.pythonhosted.org/packages/f0/77/ccfb2bf18fec3a7cb2d3e65e60eca9c34948b5508365fde5d8733195cc28/PyQtPurchasing-5.10-5.10.0-cp35.cp36.cp37-none-win32.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,409 INFO: https://files.pythonhosted.org/packages/ee/1f/5c9b069f4390d1e3aab0a249cf13a2df227f3c31495bb8d7c369f020892d/PyQtPurchasing-5.10-5.10.0-cp35.cp36.cp37-none-win_amd64.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,409 INFO: https://files.pythonhosted.org/packages/21/c4/ed9526a559cedd451a48f67d64b5f86af2da02f2d44e63814609af0ce79d/PyQtPurchasing-5.10.1-5.10.1-cp35.cp36.cp37.cp38-abi3-macosx_10_6_intel.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,420 INFO: https://files.pythonhosted.org/packages/be/49/c6834af097c7e4c205a6edaadb50dc2feca910ad8a4fdbd1c710d8a1f335/PyQtPurchasing-5.10.1-5.10.1-cp35.cp36.cp37.cp38-none-win32.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,420 INFO: https://files.pythonhosted.org/packages/0c/27/7ab0598c562dcd14920e414e2efaf971696240389da7da9bc45f80988f14/PyQtPurchasing-5.10.1-5.10.1-cp35.cp36.cp37.cp38-none-win_amd64.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,420 INFO: https://files.pythonhosted.org/packages/e9/fb/2757a5bfea3378e6e477244853c36c2978ef1b7985199ba997d619ca826b/PyQtPurchasing-5.11.2-5.11.1-cp35.cp36.cp37.cp38-abi3-macosx_10_6_intel.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,432 INFO: https://files.pythonhosted.org/packages/c4/95/5830475fccb4819968f60eeac8e208838060b001b3a3f07f544f57b67a68/PyQtPurchasing-5.11.2-5.11.1-cp35.cp36.cp37.cp38-none-win32.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,432 INFO: https://files.pythonhosted.org/packages/31/dd/e90e62a03967d6e4daf96d0f468532e4fb923881af1f2e5fa1419941cb75/PyQtPurchasing-5.11.2-5.11.1-cp35.cp36.cp37.cp38-none-win_amd64.whl would be fetched (verify.py:163)
    2021-07-27 11:45:11,432 INFO: https://files.pythonhosted.org/packages/c1/31/eacc67d01b925acfd73bc059d565464dd52a4594197b6538aee5e316e254/PyQtPurchasing-5.11.3-5.11.2-cp35.cp36.cp37.cp38-abi3-macosx_10_6_intel.whl would be fetched (verify.py:163)
    bug help wanted 
    opened by zmylk 1
  • Problem with metadata fetching

    Problem with metadata fetching

    I previously used Bandersnatch 4.4 to mirror specific packages, and it worked perfectly, but since I updated it to version 5.0, it does download the package releases into my allowlist but it downloads the metadata of all packages are worrying about my allowlist. I looked in the documentation but I don't understand how I can prevent it from downloading package metadata that I don't want to mirror, can you help me please?

    Here's my config:

    enabled =
    plugins =
    platforms =
    packages =

    And the output when I execute bandersnatch mirror

    2021-06-17 12:13:19,437 INFO: Selected storage backend: filesystem (configuration.py:126)
    2021-06-17 12:13:19,437 INFO: Selected compare method: hash (configuration.py:172)
    2021-06-17 12:13:19,574 INFO: Initialized project plugin blocklist_project, filtering [] (blocklist_name.py:27)
    2021-06-17 12:13:19,669 INFO: Initialized release plugin allowlist_release, filtering [<Requirement('cryptography>=2.6.1')>, <Requirement('idna>=2.6')>, <Requirement('pcodedmp>=1.2.6')>, <Requirement('pygobject>=3.30.4')>, <Requirement('python-debianbts>=2.8.2')>, <Requirement('urllib3>=1.24.1')>, <Requirement('setuptools>=40.8.0')>, <Requirement('cupshelpers>=1.0')>, <Requirement('keyring>=17.1.1')>, <Requirement('psutil>=5.7.2')>, <Requirement('uptime>=3.0.1')>, <Requirement('easygui>=0.98.1')>, <Requirement('pysftp>=0.2.9')>, <Requirement('entrypoints>=0.3')>, <Requirement('wheel>=0.32.3')>, <Requirement('pycairo>=1.16.2')>, <Requirement('pysmbc>=')>, <Requirement('pystemd>=0.7.0')>, <Requirement('distro>=1.3.0')>, <Requirement('pyinstaller>=4.0')>, <Requirement('toml>=0.9.0')>, <Requirement('prometheus-client>=0.8.0')>, <Requirement('colorclass>=2.2.0')>, <Requirement('keyrings-alt>=3.1.1')>, <Requirement('typing-extensions>=3.6.4')>, <Requirement('msoffcrypto-tool>=4.10.2')>, <Requirement('olefile>=0.46')>, <Requirement('pip>=18.1')>, <Requirement('python-magic>=0.4.18')>, <Requirement('six>=1.12.0')>, <Requirement('asn1crypto>=0.24.0')>, <Requirement('raptorq>=1.4.2')>, <Requirement('pycrypto>=2.6.1')>, <Requirement('pylru>=1.2.0')>, <Requirement('paho-mqtt>=1.5.1')>, <Requirement('jsonpickle>=1.4.1')>, <Requirement('pyinotify>=0.9.6')>, <Requirement('ansible>=2.9.12')>, <Requirement('requests>=2.21.0')>, <Requirement('tornado>=6.0.4')>, <Requirement('pycups>=1.9.73')>, <Requirement('distro-info>=0.21')>, <Requirement('pyparsing>=2.4.7')>, <Requirement('altgraph>=0.17')>, <Requirement('semantic-version>=2.6.0')>, <Requirement('paho-mqtt>=1.5.0')>, <Requirement('pyyaml>=5.3.1')>, <Requirement('markupsafe>=1.1.1')>, <Requirement('bcrypt>=3.2.0')>, <Requirement('jmespath>=0.10.0')>, <Requirement('setuptools-rust>=0.11.4')>, <Requirement('pynacl>=1.4.0')>, <Requirement('zstandard>=0.14.0')>, <Requirement('unidecode>=1.1.1')>, <Requirement('jinja2>=2.11.2')>, <Requirement('pycurl>=')>, <Requirement('cerberus>=1.3.2')>, <Requirement('chrome-gnome-shell>=0.0.0')>, <Requirement('reportbug>=7.5.3-deb10u1')>, <Requirement('paramiko>=2.7.2')>, <Requirement('chardet>=3.0.4')>, <Requirement('zipp>=3.1.0')>, <Requirement('certifi>=2018.8.24')>, <Requirement('pycparser>=2.20')>, <Requirement('pyxdg>=0.25')>, <Requirement('python-debian>=0.1.35')>, <Requirement('httplib2>=0.11.3')>, <Requirement('oletools>=0.55.1')>, <Requirement('secretstorage>=2.3.1')>, <Requirement('cffi>=1.14.3')>, <Requirement('pysimplesoap>=1.16.2')>, <Requirement('pyinstaller-hooks-contrib>=2020.8')>, <Requirement('unattended-upgrades>=0.1')>, <Requirement('python-apt>=')>, <Requirement('importlib-metadata>=1.7.0')>] (allowlist_name.py:170)
    2021-06-17 12:13:19,672 INFO: Initialized release plugin blocklist_release, filtering [] (blocklist_name.py:110)
    2021-06-17 12:13:19,687 INFO: Initialized exclude_platform plugin with ['.win32', '-win32', 'win_amd64', 'win-amd64', 'macosx_', 'macosx-', '.freebsd', '-freebsd'] (filename_name.py:85)
    2021-06-17 12:13:19,937 INFO: Status file /home/admsrv/.bandersnatch/status missing. Starting over. (mirror.py:594)
    2021-06-17 12:13:19,937 INFO: Syncing with https://pypi.org. (mirror.py:58)
    2021-06-17 12:13:19,937 INFO: Current mirror serial: 0 (mirror.py:263)
    2021-06-17 12:13:19,937 INFO: Resuming interrupted sync from local todo list. (mirror.py:270)
    2021-06-17 12:13:21,566 INFO: Trying to reach serial: 10666172 (mirror.py:295)
    2021-06-17 12:13:21,566 INFO: 277245 packages to sync. (mirror.py:297)
    2021-06-17 12:13:21,566 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:77)
    opened by Lulu300 7
  • 'prerelease_release' plugin causes a KeyError in package metadata verify

    'prerelease_release' plugin causes a KeyError in package metadata verify

    I am unable to perform a "verify" on my package mirror if I have the prerelease_release plugin enabled in my bandersnatch.conf file. If I remove this plugin, it works fine.

    (venv) [email protected]:/mnt/pypi# bandersnatch --config ./bandersnatch.conf verify
    2021-06-11 08:18:55,619 INFO: Starting verify for /mnt/pypi/pypi with 3 workers (verify.py:242)
    2021-06-11 08:18:55,792 INFO: Parsing wix-protos-wixerd-api-gateway-server-api (verify.py:128)
    2021-06-11 08:18:55,885 INFO: Initialized prerelease plugin with [re.compile('.+rc\\d+$'), re.compile('.+a(lpha)?\\d+$'), re.compile('.+b(eta)?\\d+$'), re.compile('.+dev\\d+$')] (prerelease_name.py:33)
    Traceback (most recent call last):
      File "/mnt/pypi/venv/bin/bandersnatch", line 8, in <module>
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/main.py", line 213, in main
        return asyncio.run(async_main(args, config))
      File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
        return loop.run_until_complete(main)
      File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
        return future.result()
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/main.py", line 127, in async_main
        return await bandersnatch.verify.metadata_verify(config, args)
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/verify.py", line 259, in metadata_verify
        await verify_producer(
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/verify.py", line 222, in verify_producer
        await asyncio.gather(
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/verify.py", line 212, in consume
        await verify(
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch/verify.py", line 155, in verify
      File "/mnt/pypi/venv/lib/python3.8/site-packages/bandersnatch_filter_plugins/prerelease_name.py", line 39, in filter
        version = metadata["version"]
    KeyError: 'version'

    This is an except from my config file. If I comment out just the prerelease_release line, the verify operation starts as expected.

    enabled =

    This issue may be similar to https://github.com/pypa/bandersnatch/issues/505 as they are both failing because the metadata dictionary does not contain the key they are expecting.

    bug help wanted 
    opened by RWoodring79 5
  • Support new PEP 0658 Metadata

    Support new PEP 0658 Metadata


    If this passes, lets ensure bandersnatch mirrors the new files etc.

    enhancement help wanted 
    opened by cooperlees 0
  • bandersnatch_docker_upload should only run on PR or official repo

    bandersnatch_docker_upload should only run on PR or official repo

    bandersnatch_docker_upload is now in the Actions list of forked repos, which I guess is not supposed to happen.

    bug enhancement help wanted 
    opened by happyaron 2
  • Add CI Success / Fail Discord notifications

    Add CI Success / Fail Discord notifications

    Lets have IRC like CI notifications in Discord.

    Some Discord alert bots: https://github.com/marketplace/actions/actions-status-discord https://github.com/marketplace/actions/discord-workflow-status-notifier

    opened by cooperlees 0
  • 5.0.0(Apr 28, 2021)

    New Features

    • bandersnatch is now a >= 3.8 Python project
    • New size_project_metadata filter plugin, which can deny download of projects larger than defined threshold - PR #806
    • Add option to compare file size and upload time instead of sha256sum for downloading - PR #822
    • Add optional uvloop support - PR #891 - Thanks cooperlees
    • Move to official docker upload action w/arm64 images uploaded - PR #896 - Thanks cooperlees


    • blacklist/whitelist will no longer work in bandersnatch configuration - PR #897 - Thanks cooperlees
      • Please use allowlist/denylist respectively

    Bug Fixes

    • Unused storage plugins are loaded and cause non-fatal errors if dependencies are missing - PR #799 - Thanks electricworry
    • Replaced usages of asynctest with unittest.mock in tests - PR #807 and PR #856 - Thanks ichard26
    • Remove debugging line that loads entire files into memory. - PR #858 - Thanks asrp
    • Removed terrible isinstance check of unittest.Mock in mirror.py - PR #859 - Thanks ichard26
    • Put potential time consuming IO operations into executor - PR #877
    • Migrated Markdown documentation from recommonmark to MyST-Parser + docs config clean up - PR #879 - Thanks ichard26
    • Use shutil.move() for temp file management - PR #883 - Thanks happyaron
    • Fixed logging bug in SizeProjectMetadataFilter to show it activated - PR #889 - Thanks cooperlees
    • Attempt to wrap all potentially block calls in a ThreadPoolExecutor - PR #894 - Thanks cooperlees
    Source code(tar.gz)
    Source code(zip)
  • 4.4.0(Dec 31, 2020)

    New Features

    • Build a swift and non swift docker image - PR #754
    • Split Docker Build to accept build args to optionally include swift support - PR #741 - Thanks nlaurance-pyie
    • Slimmer docker image - PR #738 - Thanks nlaurance-pyie
    • Renamed black/white to block/allow lists - PR #737 - Thanks nlaurance-pyie
    • packages allowlist can be defined from requirements like files - PR #739 - Thanks nlaurance-pyie
    • Simplify logging around filters - PR #678 - Thanks @dalley

    Bug Fixes

    • Handling of timeouts that can occur in verify. - PR #785 - Thanks electricworry
    • Added retry logic on timeouts when fetching metadata - PR #773 - Thanks gerrod3
    • Fix links, improve docs CI, and improve external object linking - PR #776 - Thanks ichard26
    • Handle 404 status for json verify - PR #763 - Thanks electricworry
    • Clean up isort config after upgrade to 5+ - PR #767 - Thanks ichard26
    • Remove duplicate max() target serial finding code + update typing - PR #745
    • swift.py: use BaseFileLock's lock_file property - PR #699 - Thanks hauntsaninja
    • Move to latest isort + mypy fixes - PR #706
    • Update change log url in project metadata - PR #673 - Thanks @abn
    Source code(tar.gz)
    Source code(zip)
  • 4.3.0(Aug 25, 2020)

    New Features

    • Add SOCKS proxy support to aiohttp via aiohttp-socks - PR #668
    • Add support for skipping mirroring release files (metadata only) - PR #670 - Thanks @abn

    Bug Fixes

    • Move GitHub actions to v2 tags - PR #666 - Thanks @ryuichi1208
    Source code(tar.gz)
    Source code(zip)
  • 4.2.0(Aug 21, 2020)

    New Features

    Thanks to RedHat engineers @dalley + @gerrod3 for all this refactor work in PR #591

    • New generic Mirror class to perform Python metadata syncing
      • (previous Mirror class has been renamed to BandersnatchMirror)
    • Package's filter methods are now part of its public API
    • New errors.py file to house Bandersnatch specific errors

    Internal API Changes

    • Old Mirror class has been renamed to BandersnatchMirror. Performs same functionality with use of new Mirror API.
    • BandersnatchMirror now performs all filesystem operations throughout the sync process including the ones previously in Package.
    • Package no longer performs filesystem operations. Properties json_file, json_pypi_symlink, simple_directory and methods save_json_metadata, sync_release_files, gen_data_requires_python, generate_simple_page, sync_simple_page, _save_simple_page_version, _prepare_versions_path, _file_url_to_local_url, _file_url_to_local_path, download_file have all been moved into BandersnatchMirror. Package's sync has been refactored into Bandersnatch's process_package.
    • Package class is no longer created with an instance of Mirror
    • StaleMetadata exception has been moved to new errors.py file
    • PackageNotFound exception has been moved to new errors.py file

    Bug Fixes

    • Fixed Fix latest_release plugin to ensure latest version is included - PR #660 - Thanks @serverwentdown
    Source code(tar.gz)
    Source code(zip)
  • 4.1.1(Aug 12, 2020)

  • 4.1.0(Aug 10, 2020)

    New Features

    • bandersnatch is now 100% type annotated - PRs #546 #561 #592 #593 - Thanks @ichard26 + @rkm
    • Move to storage abstraction - PR #445 - Thanks @techalchemy
      • Can now support more than just filesystem e.g. swift
    • Add sync subcommand to force a sync on a particular PyPI package - PR #572 - Thanks @z4yx
    • Added new allowlist filter - PR #626 - Thanks @gerrod3
    • Make webdir/pypi/json/PKG symlinks relative - PR #637 - Thanks @indrat
      • Makes mirror files more portable
    • Add main and program name override to ArgumentParser - PR #643 - Thanks @rkm
      • Allow non pkg_resources install to work

    Internal API Changes

    • Refactored the removal of releases for release_plugins to happen inside of Package PR #608 - Thanks @gerrod3
    • Minor refactor of Package class PR #606 - Thanks @dralley
    • Refactored filter loading into seperate class PR #599 - Thanks @gerrod3
    • Move legacy directory cleanup to mirror.py PR #586
    • Move verify to use Master for HTTP calls - PR #555
    • Move http request code for package metadata to master.py - PRs #550 - Thanks @dralley

    Bug Fixes

    • Fixed allow/blocklist release filtering pre-releases - PR #641 - Thanks @gerrod3
    • Casefold (normalize per PEP503) package names in blacklist/whitelist plugins config - PR #629 - Thanks @lepaperwan
    • Fix passing package info to filters in verify action. PR #638 - Thanks @indrat
    • Fix todo file removal - PR #571
    • Introduce a new global-timeout config option for aiohttp coroutines - Default 5 hours - PR #540 - Thanks @techalchemy
    • Many doc fixes - PRs #542 #551 #557 #605 #628 #630 - Thanks @pgrimaud + @ichard26 + @hugovk
    • Move to setting timeout only on session + 10 * total_timeout (over sock timeouts) - PR #535
    • Stop using include_package_data option in setup.cfg to get config files included in more installs - PR #519
    Source code(tar.gz)
    Source code(zip)
  • 4.0.3(May 7, 2020)

    • Change aiohttp-xmlrpc to use Master.session to ensure config shared - PR #506 - Thanks @alebourdoulous for reporting
      • e.g. Maintin trust of proxy server environment variables
    Source code(tar.gz)
    Source code(zip)
  • 4.0.2(Apr 26, 2020)

    • Raise for error HTML response on all aiohttp session requests - PR #494 / #496 - Thanks @windtail
    • Pass str to shutil.move due to Python bug - PR #497 - Thanks @SanketDG
    • Some more type hints added to verify.py - PR #488 - Thanks @SanketDG
    • Ignore atime on stat in test test_package_sync_does_not_touch_existing_local_file comparision as it casues stat compare fail on a slower run - PR #487 - Thanks @SanketDG
    Source code(tar.gz)
    Source code(zip)
  • 4.0.1(Apr 5, 2020)

  • 4.0.0(Mar 29, 2020)

    • Replace requests with aiohttp - PR #440
    • Replace xmlrpc2 with aiohttp-xmlrpc - PR #404
    • Only store PEP503 Normalized Simple API directories - PR #465 + #455
    • Flag errors when KeyboardInterrupt raised during sync - PR #421
    • Finish Windows Support + Add CI - PRs #469 + #471 - Thanks @FaustinCarter
    • Autobuild Docker images with master - PR #88 - Thanks @abitrolly
    • Only print conf deprecations if found in config - PR #327
    • Add PyPI metadata and Python version plugin filters - PR #391 - Thanks @TemptorSent
    • Add in GitHub Actions CI for Linux (Ubuntu), MacOSX + Windows
    Source code(tar.gz)
    Source code(zip)
  • 3.6.0(Sep 24, 2019)

  • 3.5.0(Sep 14, 2019)

  • 3.4.0(May 30, 2019)

  • 3.3.0(Apr 12, 2019)

  • 3.2.0(Jan 25, 2019)

  • 3.1.3(Dec 26, 2018)

  • 3.1.2(Dec 3, 2018)

  • 3.1.1(Nov 26, 2018)

  • 3.1.0(Nov 26, 2018)

  • 3.0.1(Oct 30, 2018)

Python Packaging Authority
Python Packaging Authority
An installation and dependency system for Python

Pyflow Simple is better than complex - The Zen of Python Pyflow streamlines working with Python projects and files. It's an easy-to-use CLI app with a

David O'Connor 904 Oct 19, 2021
A set of tools to keep your pinned Python dependencies fresh.

pip-tools = pip-compile + pip-sync A set of command line tools to help you keep your pip-based packages fresh, even when you've pinned them. You do pi

Jazzband 5.3k Oct 22, 2021
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 22.4k Oct 24, 2021
:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED Forks: https://github.com/orsinium/forks DepHell -- project management for Python. Why it is better than all other tools: Form

DepHell 1.7k Oct 22, 2021
pip-run - dynamic dependency loader for Python

pip-run provides on-demand temporary package installation for a single interpreter run. It replaces this series of commands (or their Windows equivale

Jason R. Coombs 61 Oct 2, 2021
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.

Spack Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, a

Spack 2.3k Oct 15, 2021
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 16.8k Oct 24, 2021
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 16.8k Oct 22, 2021
Python PyPi staging server and packaging, testing, release tool

devpi: PyPI server and packaging/testing/release tool This repository contains three packages comprising the core devpi system on the server and clien

null 500 Oct 23, 2021
local pypi server (custom packages and auto-mirroring of pypi)

localshop A PyPI server which automatically proxies and mirrors PyPI packages based upon packages requested. It has support for multiple indexes and t

Michael van Tellingen 372 Aug 29, 2021
pipreqs - Generate pip requirements.txt file based on imports of any project. Looking for maintainers to move this project forward.

pipreqs - Generate requirements.txt file for any project based on imports Installation pip install pipreqs Usage Usage: pipreqs [options] <path>

Vadim Kravcenko 3.7k Oct 22, 2021
OS-agnostic, system-level binary package manager and ecosystem

Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for

Conda 4.3k Oct 23, 2021
OS-agnostic, system-level binary package manager and ecosystem

Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for

Conda 4.3k Oct 16, 2021
The Python Package Index

Warehouse Warehouse is the software that powers PyPI. See our development roadmap, documentation, and architectural overview. Getting Started You can

Python Packaging Authority 2.8k Oct 24, 2021
Conan - The open-source C/C++ package manager

Conan Decentralized, open-source (MIT), C/C++ package manager. Homepage: https://conan.io/ Github: https://github.com/conan-io/conan Docs: https://doc

Conan.io 5.3k Oct 22, 2021
A Poetry plugin for dynamically extracting the package version.

Poetry Version Plugin A Poetry plugin for dynamically extracting the package version. It can read the version from a file __init__.py with: # __init__

Sebastián Ramírez 148 Oct 9, 2021
Install and Run Python Applications in Isolated Environments

pipx — Install and Run Python Applications in Isolated Environments Documentation: https://pipxproject.github.io/pipx/ Source Code: https://github.com

null 4.2k Oct 22, 2021
The Python package installer

pip - The Python Package Installer pip is the package installer for Python. You can use pip to install packages from the Python Package Index and othe

Python Packaging Authority 7.5k Oct 23, 2021
A PDM plugin that packs your packages into a zipapp

pdm-packer A PDM plugin that packs your packages into a zipapp Requirements pdm-packer requires Python >=3.7 Installation If you have installed PDM wi

Frost Ming 10 Sep 23, 2021