A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

Python Packaging Authority

Last update: Dec 28, 2022

Related tags

Package Management mirror pypi mirroring bandersnatch pypi-mirror-client pep381

Overview

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/.

bandersnatch >=4.0 supports Linux, MacOSX + Windows
Documentation

bandersnatch maintainers are looking for more help! Please refer to our MAINTAINER documentation to see the roles and responsibilities. We would also ask you read our Mission Statement to ensure it aligns with your thoughts for this project.

If interested contact @cooperlees

bandersnatch has its dependencies kept up to date by pyup.io!

If you'd like to have your dependencies kept up to date in your requirements.txt or setup.cfg, this is the service for you!

Installation

The following instructions will place the bandersnatch executable in a virtualenv under bandersnatch/bin/bandersnatch.

bandersnatch requires >= Python 3.8.0

Docker

This will pull latest build. Please use a specific tag if desired.

Docker image includes /bandersnatch/src/runner.py to periodically run a bandersnatch mirror
- Please /bandersnatch/src/runner.py --help for usage
With docker, we recommend bind mounting in a read only bandersnatch.conf
- Defaults to /conf/bandersnatch.conf

docker pull pypa/bandersnatch
docker run pypa/bandersnatch bandersnatch --help

pip

This installs the latest stable, released version.

python3 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help

Quickstart

Run bandersnatch mirror - it will create an empty configuration file for you in /etc/bandersnatch.conf.
Review /etc/bandersnatch.conf and adapt to your needs.
Run bandersnatch mirror again. It will populate your mirror with the current status of all PyPI packages. Current mirror package size can be seen here: https://pypi.org/stats/
A blocklist or allowlist can be created to cut down your mirror size. You might want to Analyze PyPI downloads to determine which packages to add to your list.
Run bandersnatch mirror regularly to update your mirror with any intermediate changes.

Webserver

Configure your webserver to serve the web/ sub-directory of the mirror. For nginx it should look something like this:

    server {
        listen 127.0.0.1:80;
        listen [::1]:80;
        server_name <mymirrorname>;
        root <path-to-mirror>/web;
        autoindex on;
        charset utf-8;
    }

Note that it is a good idea to have your webserver publish the HTML index files correctly with UTF-8 as the charset. The index pages will work without it but if humans look at the pages the characters will end up looking funny.
Make sure that the webserver uses UTF-8 to look up unicode path names. nginx gets this right by default - not sure about others.

For more information visit out official documentation for instructions on how to use a NGINX example Docker Image.

Cron jobs

You need to set up one cron job to run the mirror itself.

Here's a sample that you could place in /etc/cron.d/bandersnatch:

    LC_ALL=en_US.utf8
    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]

This assumes that you have a logger utility installed that will convert the output of the commands to syslog entries.

SystemD Timers are also another alternative in today's modern world.

Maintenance

bandersnatch does not keep much local state in addition to the mirrored data. In general you can just keep rerunning bandersnatch mirror to make it fix errors.

If you want to force bandersnatch to check everything against the master PyPI:

run bandersnatch mirror --force-check to move status files if they exist in your mirror directory in order get a full sync.

Be aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.

Other Commands

bandersnatch delete --help - Allows you to specify package(s) to be removed from your mirror (dangerous)
bandersnatch verify --help - Crawls your repo and fixes any missed files + deletes any unowned files found (dangerous)

Operational notes

Case-sensitive filesystem needed

You need to run bandersnatch on a case-sensitive filesystem.

OS X natively does this OK even though the filesystem is not strictly case-sensitive and bandersnatch will work fine when running on OS X. However, tarring a bandersnatch data directory and moving it to, e.g. Linux with a case-sensitive filesystem will lead to inconsistencies. You can fix those by deleting the status files and have bandersnatch run a full check on your data.

Windows requires elevated prompt

Bandersnatch makes use of symbolic links. On Windows, this permission is turned off by default for non-admin users. In order to run bandersnatch on Windows either call it from an elevated command prompt (i.e. right-click, run-as Administrator) or give yourself symlink permissions in the group policy editor.

Many sub-directories needed

The PyPI has a quite extensive list of packages that we need to maintain in a flat directory. Filesystems with small limits on the number of sub-directories per directory can run into a problem like this:

    2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449
    OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'

Specifically we recommend to avoid using ext3. Ext4 and newer does not have the limitation of 32k sub-directories.

Client Compatibility

A bandersnatch static mirror is compatible only to the "static", cacheable parts of PyPI that are needed to support package installation. It does not support more dynamic APIs of PyPI that maybe be used by various clients for other purposes.

An example of an unsupported API is PyPI's XML-RPC interface, which is used when running pip search.

Bandersnatch Mission

The bandersnatch project strives to:

Mirror all static objects of the Python Package Index (https://pypi.org/)
bandersnatch's main goal is to support the main global index to local syncing only
This will allow organizations to have lower latency access to PyPI and save bandwidth on their WAN connections and more importantly the PyPI CDN
Custom features and requests may be accepted if they can be of a plugin form
- e.g. refer to the blocklist and allowlist plugins

Contact

If you have questions or comments, please submit a bug report to https://github.com/pypa/bandersnatch/issues/new

IRC: #bandersnatch on Freenode (You can use webchat if you don't have an IRC client)

Code of Conduct

Everyone interacting in the bandersnatch project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PSF Code of Conduct.

Kudos

This client is based on the original pep381client by Martin v. Loewis.

Richard Jones was very patient answering questions at PyCon 2013 and made the protocol more reliable by implementing some PyPI enhancements.

Christian Theune for creating and maintaining bandersnatch for many years!

Comments

Bandersnatch not able to pull tensorflow binary

Referring to my attached bandersnatch.conf i am not able to pull down tensorflow and tensorflow-gpu binaries in the simple/ folder why is this so? In addition upon a successful pull of the python binaries, the mirror exit gracefully with a code 0.. Upon the next time i run a bandersnatch mirror --force-check again with the contents as per the last run, I get an exit code 1 with TypeError: must be a str or None not int why is this so? pp-pypi-mirror_config_bandersnatch.txt
bug help wanted needs_external_pr

opened by goal86sg 13

Dockerimage not working

The dockerimage doesn't work, as it seems to not find the 'default.conf':

docker run -it --rm pypa/bandersnatch:latest
Running bandersnatch every 3600s
Config file '/conf/bandersnatch.conf' missing, creating default config.
Please review the config file, then run 'bandersnatch' again.
Could not create config file: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf'

but even when i provide a /conf/bandersnatch.conf the image still fails, now with:

docker run -it --rm -v $PWD/bandersnatch.conf:/conf/bandersnatch.conf:ro pypa/bandersnatch:latest
Running bandersnatch every 3600s
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 183, in <module>
    exit(main())
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 167, in main
    config = bandersnatch.configuration.BandersnatchConfig(
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/configuration.py", line 33, in __call__
    cls._instances[cls] = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/configuration.py", line 51, in __init__
    with importlib.resources.path("bandersnatch", "default.conf") as config_path:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/importlib/resources.py", line 201, in path
    with open_binary(package, resource) as fp:
  File "/usr/local/lib/python3.8/importlib/resources.py", line 91, in open_binary
    return reader.open_resource(resource)
  File "<frozen importlib._bootstrap_external>", line 988, in open_resource
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf'

enhancement help wanted good first issue

opened by MikeHofmann 13

Only download latest versions

Is there any way to configure bandersnatch to only download the latest version of packages? I don't need nor want 600GB+ of every released version of every package. I'm not running an official mirror, I just want a local copy of latest packages.
enhancement

opened by wayneworkman 13
Split Docker Build to accept build args to optionally include swift support

added dockerignore for slimer build context parametrized dockerfile

making swift support optional

nil/bandersnatch 3.9 9f730c999230 2 minutes ago 156MB nil/bandersnatch 3.9-swift 91d50bd7103d 4 minutes ago 280MB

hopefully addressing https://github.com/pypa/bandersnatch/issues/740 also

opened by nlaurance-pyie 12
Problems with 'runner.py'
Hello there,

I noticed these problems in the runner.py while I was using the docker image:

https://github.com/pypa/bandersnatch/blob/1b573a141fd6ee4b171c612b1422089b119e544f/src/runner.py#L27

Firstly, args doesn't have a conf attribute, it has been defined as config.

Secondly, I think, It's not a good idea to hardcode the bandersnatch path. Because even in the Docker image you'll find the bandersnatch inside /usr/local/bin/

Thirdly, the interval type should be int not str: https://github.com/pypa/bandersnatch/blob/1b573a141fd6ee4b171c612b1422089b119e544f/src/runner.py#L21

Because, this line will show an error if you execute it: https://github.com/pypa/bandersnatch/blob/1b573a141fd6ee4b171c612b1422089b119e544f/src/runner.py#L28

Even when I solved the above problems in my container, I got another error. My question is, why don't you guys utilize the default configuration when the configuration hasn't been given?

2020-01-04 16:00:45,222 WARNING: Config file '/conf/bandersnatch.conf' missing, creating default config. 2020-01-04 16:00:45,222 WARNING: Please review the config file, then run 'bandersnatch' again. 2020-01-04 16:00:45,222 ERROR: Could not create config file: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/bandersnatch/default.conf' Sleeping for 3599.6514344215393s
needs_external_pr
opened by GreatBahram 12
Filtered Packages Not Being Removed
We are running a PyPI mirror with bandersnatch and have recently enabled a number of filters. One goal of our filtering is to reduce the mirror size. However, after enabling the filters, no packages seem to get deleted. How do we go about getting packages cleaned off of the mirror that do not meet our filtering configuration? Does only the verify command delete packages?

Our config file looks similar to this:

[mirror] directory = /srv/pypi json = true master = https://pypi.org/ timeout = 10 workers = 3 hash-index = false stop-on-error = false delete-packages = true verifiers = 2 [plugins] enabled = latest_release blacklist_project regex_project [blacklist] packages = example1 [filter_regex] packages = .+-evil$ [latest_release] keep = 3
needs_external_pr
opened by daheise 12

Docker running + storing packages on an overlayfs causing conflicting exceptions

Description

I want to mirror only a handful of packages with bandersnatch. To achieve this I use a docker-compose file which loads the preset configuration as shown below. After issuing the command below I get a couple of traces of different nature:

ERROR: Continuing to next file after error downloading -> (sounds like files cannot be found on the server side)
ERROR: Error syncing package: fastapi@15646760 (mirror.py:377) -> (sounds like files are already present)

Is this expected behaviour and can I ignore the errors safely?

Log output

user@machine pip % docker exec -it pip-bandersnatch-1 bandersnatch -c /conf/bandersnatch.conf 
mirror
2022-11-07 11:00:48,862 INFO: Selected storage backend: filesystem (configuration.py:131)
2022-11-07 11:00:48,864 INFO: Selected compare method: hash (configuration.py:177)
2022-11-07 11:00:49,537 INFO: Initialized project plugin allowlist_project, filtering ['typer', 'docker', 'fastapi'] (allowlist_name.py:31)
2022-11-07 11:00:49,621 INFO: Initialized release plugin allowlist_release, filtering [<Requirement('docker[all]')>, <Requirement('typer[all]')>, <Requirement('fastapi[all]')>] (allowlist_name.py:177)
2022-11-07 11:00:49,699 INFO: Status file /srv/pypi/status missing. Starting over. (mirror.py:566)
2022-11-07 11:00:49,700 INFO: Syncing with https://pypi.org. (mirror.py:57)
2022-11-07 11:00:49,700 INFO: Current mirror serial: 0 (mirror.py:278)
2022-11-07 11:00:49,700 INFO: Syncing all packages. (mirror.py:293)
2022-11-07 11:01:17,621 INFO: Package 'typer' is allowlisted (allowlist_name.py:88)
2022-11-07 11:01:17,946 INFO: Package 'fastapi' is allowlisted (allowlist_name.py:88)
2022-11-07 11:01:17,952 INFO: Package 'docker' is allowlisted (allowlist_name.py:88)
2022-11-07 11:01:24,487 INFO: Trying to reach serial: 15683776 (mirror.py:310)
2022-11-07 11:01:24,487 INFO: 3 packages to sync. (mirror.py:312)
2022-11-07 11:01:24,530 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:76)
2022-11-07 11:01:24,531 INFO: No release file filters are enabled. Skipping release file filtering (mirror.py:80)
2022-11-07 11:01:24,533 INFO: Fetching metadata for package: docker (serial 15631690) (package.py:58)
2022-11-07 11:01:24,536 INFO: Fetching metadata for package: fastapi (serial 15646760) (package.py:58)
2022-11-07 11:01:24,539 INFO: Fetching metadata for package: typer (serial 15668790) (package.py:58)
2022-11-07 11:01:27,253 INFO: Storing index page(s): typer - in /srv/pypi/web/simple/typer (mirror.py:698)
2022-11-07 11:01:31,154 INFO: Storing index page(s): docker - in /srv/pypi/web/simple/docker (mirror.py:698)
2022-11-07 11:01:36,188 INFO: Downloading: https://files.pythonhosted.org/packages/c4/3d/2f36f343b2927f2bcef8707510a8560dc07f76a2da95c1c3274942d7a1e3/fastapi-0.18.0.tar.gz (mirror.py:875)
2022-11-07 11:01:37,381 INFO: Downloading: https://files.pythonhosted.org/packages/a7/2d/97d6367bec2367b586349e9a35cfa8e269a41a110e5c7910759709854b4a/fastapi-0.19.0.tar.gz (mirror.py:875)
2022-11-07 11:01:38,489 INFO: Downloading: https://files.pythonhosted.org/packages/fc/1f/eab23d52f245e3b09aee2ae85e2507a0f4a56f7ec562698fad9e88b895cb/fastapi-0.2.0.tar.gz (mirror.py:875)
2022-11-07 11:01:39,291 INFO: Downloading: https://files.pythonhosted.org/packages/e6/a3/bffda6970d7153cb519968e3b41275365f190327345073bbbc7ff20d9140/fastapi-0.2.1.tar.gz (mirror.py:875)
2022-11-07 11:01:40,160 INFO: Downloading: https://files.pythonhosted.org/packages/c8/37/9931d412ba3132f2c4332682e0b8bb8ddc0f60471e0a8d6896fa45c73f1c/fastapi-0.20.0.tar.gz (mirror.py:875)
2022-11-07 11:01:41,238 INFO: Downloading: https://files.pythonhosted.org/packages/ba/9a/c1a0504028822ec649a45b52dce11d25b60d7a4b93510a39c9e41435fdc2/fastapi-0.20.1.tar.gz (mirror.py:875)
2022-11-07 11:01:42,337 INFO: Downloading: https://files.pythonhosted.org/packages/52/46/76fc59fd9b472efbc6b00a537b061c22db8ee2b249e87a1ee5c8e9eff306/fastapi-0.21.0.tar.gz (mirror.py:875)
2022-11-07 11:01:43,406 INFO: Downloading: https://files.pythonhosted.org/packages/2c/1d/bcf27e8f99a716cc60dfc9c1258d90d82f05b6c650d2c768eb2ae71d9b63/fastapi-0.22.0.tar.gz (mirror.py:875)
2022-11-07 11:01:44,476 INFO: Downloading: https://files.pythonhosted.org/packages/5e/58/3eeecfc9acb2a5d79b4b523d6cc6f97f122a5ead5ad366f01e750036a74e/fastapi-0.23.0.tar.gz (mirror.py:875)
2022-11-07 11:01:45,535 INFO: Downloading: https://files.pythonhosted.org/packages/71/48/64f0a29bd601a90e323562547a6989db221ad5f69c77e0b7565f8ec477d0/fastapi-0.24.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:01:45,550 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/71/48/64f0a29bd601a90e323562547a6989db221ad5f69c77e0b7565f8ec477d0/fastapi-0.24.0-py3-none-any.whl (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/71/48/64f0a29bd601a90e323562547a6989db221ad5f69c77e0b7565f8ec477d0'
2022-11-07 11:01:45,555 INFO: Downloading: https://files.pythonhosted.org/packages/9c/85/d38aa42a29231906bb559cca8bb2f032f170bfbf647fe446bc90258294ba/fastapi-0.24.0.tar.gz (mirror.py:875)
2022-11-07 11:01:46,654 INFO: Downloading: https://files.pythonhosted.org/packages/b7/48/6a691a87c3eb8638e2e6adb8e1a0e34d35067122a63db29e8c2b435ff3e0/fastapi-0.25.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:01:46,738 INFO: Downloading: https://files.pythonhosted.org/packages/50/16/a7b8f7cdb679e8f81547e0f9d6d9049815a5cbf187738a60be6e682e0089/fastapi-0.25.0.tar.gz (mirror.py:875)
2022-11-07 11:01:47,898 INFO: Downloading: https://files.pythonhosted.org/packages/ad/df/6d757dab8ea480be15f5b600ee90aaf149c16a3780b8cd50bc749f65fdf1/fastapi-0.26.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:01:47,984 INFO: Downloading: https://files.pythonhosted.org/packages/db/a2/8683239773f12c1f97ffd2d76223882a5ecd86abfabc45ceb078d1b668f2/fastapi-0.26.0.tar.gz (mirror.py:875)
2022-11-07 11:01:49,097 INFO: Downloading: https://files.pythonhosted.org/packages/c3/fb/2af84a2e65f5453561c4cc4f571aa282dc8780df2a212fd17951aadddca4/fastapi-0.27.0.tar.gz (mirror.py:875)
2022-11-07 11:01:50,183 INFO: Downloading: https://files.pythonhosted.org/packages/eb/36/31d0a08c3303a4f56494f489d615f49c13603579c6ed49b301f21b711a9b/fastapi-0.27.1.tar.gz (mirror.py:875)
2022-11-07 11:01:51,287 INFO: Downloading: https://files.pythonhosted.org/packages/48/6a/78cecd90b1fc4e62408a53ae4e6de1daa21762c0e1f251fb8a26e2ad061b/fastapi-0.27.2.tar.gz (mirror.py:875)
2022-11-07 11:01:52,401 INFO: Downloading: https://files.pythonhosted.org/packages/e6/b1/cec35fe22b1f15afabf5d02f7b3c3885dacf6cc5364f14ffa58da7febfbc/fastapi-0.28.0.tar.gz (mirror.py:875)
2022-11-07 11:01:53,491 INFO: Downloading: https://files.pythonhosted.org/packages/fb/2a/32279b88584b345333581974843820d0bb8a8ecf956b065a013138362a59/fastapi-0.29.0.tar.gz (mirror.py:875)
2022-11-07 11:01:54,609 INFO: Downloading: https://files.pythonhosted.org/packages/6c/6a/6bca3022a045066fa2e64eca6b64a6ddc1fe6e2b753fc39128b46df6678b/fastapi-0.29.1.tar.gz (mirror.py:875)
2022-11-07 11:01:55,715 INFO: Downloading: https://files.pythonhosted.org/packages/3e/5a/bd109788bf5c76661511bc51ac1c3b67f5040f9579ebc10bd31559f5e5f9/fastapi-0.3.0.tar.gz (mirror.py:875)
2022-11-07 11:01:56,528 INFO: Downloading: https://files.pythonhosted.org/packages/3c/16/f389a874e9a1235227c1c20e709c5503a0621195796286492599d7bd87df/fastapi-0.30.0.tar.gz (mirror.py:875)
2022-11-07 11:01:57,661 INFO: Downloading: https://files.pythonhosted.org/packages/f9/f3/6ecd806db796606431a78de4eff9edcf42f788ff653e9acf4c7f6057c38f/fastapi-0.30.1.tar.gz (mirror.py:875)
2022-11-07 11:01:58,742 INFO: Downloading: https://files.pythonhosted.org/packages/85/69/739de0af8f8db61d9cc1367048eca6c5d161e8869eeed6c9bfcb7434ef6c/fastapi-0.31.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:01:58,857 INFO: Downloading: https://files.pythonhosted.org/packages/3d/4e/7ea13b3e4c0590b57d8138041cfb511dd2b450f97316f138951348851dc0/fastapi-0.31.0.tar.gz (mirror.py:875)
2022-11-07 11:01:59,948 INFO: Downloading: https://files.pythonhosted.org/packages/2c/da/9f1ee461ce48e34b2accd0cbb45d3c527049508e965ace846a48c83957d2/fastapi-0.32.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:00,025 INFO: Downloading: https://files.pythonhosted.org/packages/d1/ce/2e59c0404fce6237d2fc99df0858459ea605238d16e45720a64824f8851c/fastapi-0.32.0.tar.gz (mirror.py:875)
2022-11-07 11:02:01,272 INFO: Downloading: https://files.pythonhosted.org/packages/94/01/9fdade622c17ebe2f24dd32df87ae8cd69dc5d6046eb2fec5f5a9c6fdefe/fastapi-0.33.0.tar.gz (mirror.py:875)
2022-11-07 11:02:02,468 INFO: Downloading: https://files.pythonhosted.org/packages/45/9e/483c79a63ca3062cc844442b1e88293dcf524ca724742538f036e617dca1/fastapi-0.34.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:02,561 INFO: Downloading: https://files.pythonhosted.org/packages/d8/bd/b4083b9f38a3f62dbb3dca2d7094d5941efdbff39ca1b57da3364da0dfb1/fastapi-0.34.0.tar.gz (mirror.py:875)
2022-11-07 11:02:03,683 INFO: Downloading: https://files.pythonhosted.org/packages/07/b7/ebc45989b2c7bfca3b49a579ad38d0011ba3054ce4a6666278933145a1dd/fastapi-0.35.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:03,779 INFO: Downloading: https://files.pythonhosted.org/packages/67/35/259ef7a337036fbe21be0748515481f0f7dd328100db27c7e3e2acebbe8d/fastapi-0.35.0.tar.gz (mirror.py:875)
2022-11-07 11:02:04,905 INFO: Downloading: https://files.pythonhosted.org/packages/bb/94/7b5953d610858b0d0324b849e13cdf97bef476a27d8d8ca3494c35a6eb54/fastapi-0.36.0.tar.gz (mirror.py:875)
2022-11-07 11:02:06,041 INFO: Downloading: https://files.pythonhosted.org/packages/34/12/093fe39888016473561d302327a013301a4ca8b9509c7a6b9af26aecfdd0/fastapi-0.37.0.tar.gz (mirror.py:875)
2022-11-07 11:02:07,144 INFO: Downloading: https://files.pythonhosted.org/packages/1b/f5/595bb169d0d85f7d3811fcf268c0872a05cb48cdd1ac1e0a14b063de3972/fastapi-0.38.0.tar.gz (mirror.py:875)
2022-11-07 11:02:08,258 INFO: Downloading: https://files.pythonhosted.org/packages/af/63/76229a4268091757dfce3f63487e8c9b92639c30a8396757679e3a20c6b6/fastapi-0.38.1.tar.gz (mirror.py:875)
2022-11-07 11:02:09,404 INFO: Downloading: https://files.pythonhosted.org/packages/8b/20/eb050de401ef0a302f20452e123af841af47306b034f9044c279ae5b37d3/fastapi-0.39.0.tar.gz (mirror.py:875)
2022-11-07 11:02:10,491 INFO: Downloading: https://files.pythonhosted.org/packages/8c/65/12ba1e298bca05dc2109f96c35c4a54c5aeadeb81e9fd88ebbead9cb7bdd/fastapi-0.4.0.tar.gz (mirror.py:875)
2022-11-07 11:02:11,416 INFO: Downloading: https://files.pythonhosted.org/packages/c4/cd/685523fc953335ac62dd4ceb9347994a85b768d2b32d1d829ce82e8048dd/fastapi-0.40.0.tar.gz (mirror.py:875)
2022-11-07 11:02:12,550 INFO: Downloading: https://files.pythonhosted.org/packages/6e/31/5f3b44f0f572c523e78a05bf2bca155466b95e4746ad5a4d500736b86f9c/fastapi-0.41.0.tar.gz (mirror.py:875)
2022-11-07 11:02:13,667 INFO: Downloading: https://files.pythonhosted.org/packages/90/31/099a3abb5d43db1a0fc29144e4dd8681ca33a6005473fd315ff364c6e5c6/fastapi-0.42.0.tar.gz (mirror.py:875)
2022-11-07 11:02:14,783 INFO: Downloading: https://files.pythonhosted.org/packages/b9/73/c59aca57ed410d531ca31cda428c6e03c1c07bdc466cd3b5d3dbebc5fdfc/fastapi-0.43.0.tar.gz (mirror.py:875)
2022-11-07 11:02:15,881 INFO: Downloading: https://files.pythonhosted.org/packages/ea/24/d65623a36f970764797edb4b4bbe8d9166f4911a28371410d01bd6ef0cd6/fastapi-0.44.0.tar.gz (mirror.py:875)
2022-11-07 11:02:16,994 INFO: Downloading: https://files.pythonhosted.org/packages/66/9e/cf4837c3cc4d96caddf97a2974c8bb02204ffae7fc9c0ceaff9c21cf3c49/fastapi-0.44.1.tar.gz (mirror.py:875)
2022-11-07 11:02:18,553 INFO: Downloading: https://files.pythonhosted.org/packages/6b/f4/8aa81e9e5de3877a3149f7e53b21cc3de28d2a474692f17b79b6baf1deb8/fastapi-0.45.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:18,613 INFO: Downloading: https://files.pythonhosted.org/packages/ce/eb/93adebcaf0ddddc4d185520d6bad5c00d2b3b5f70abe15ac16555e3a5b50/fastapi-0.45.0.tar.gz (mirror.py:875)
2022-11-07 11:02:19,850 INFO: Downloading: https://files.pythonhosted.org/packages/7b/c4/3593af517fbdeb51443d8376f2cbe3c63cb46531aa66415516251119a398/fastapi-0.46.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:19,914 INFO: Downloading: https://files.pythonhosted.org/packages/b4/cc/44941903f220aad11f7252d729dc32a7cb2ec20d4ffd78053abea4a4d050/fastapi-0.46.0.tar.gz (mirror.py:875)
2022-11-07 11:02:21,087 INFO: Downloading: https://files.pythonhosted.org/packages/c7/67/d489088f26e9d4846043cc15160dcabba117474ee6b76c7db7fb215f7c71/fastapi-0.47.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:21,152 INFO: Downloading: https://files.pythonhosted.org/packages/de/b1/cca52dc8e2bbf91d4e411bc87f3160cc5f6856e75a9e0b081fef25bc77ac/fastapi-0.47.0.tar.gz (mirror.py:875)
2022-11-07 11:02:22,338 INFO: Downloading: https://files.pythonhosted.org/packages/a3/a7/4804d7abf8a1544d079d50650af872387154ebdac5bd07d54b2e60e2b334/fastapi-0.47.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:22,345 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/a3/a7/4804d7abf8a1544d079d50650af872387154ebdac5bd07d54b2e60e2b334/fastapi-0.47.1-py3-none-any.whl (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/a3/a7/4804d7abf8a1544d079d50650af872387154ebdac5bd07d54b2e60e2b334'
2022-11-07 11:02:22,349 INFO: Downloading: https://files.pythonhosted.org/packages/24/98/4fc080a7caafd79deb3abdd899be45337e78e6768aba08678aee7c4edfc3/fastapi-0.47.1.tar.gz (mirror.py:875)
2022-11-07 11:02:23,516 INFO: Downloading: https://files.pythonhosted.org/packages/7b/1a/c53cf201d35cc1bb69fd2f5af9976cb12c2f0156c4662d61ce5c22c76ded/fastapi-0.48.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:23,567 INFO: Downloading: https://files.pythonhosted.org/packages/37/d9/679462f5d48458afc8afcc3c8cc62177843aaafb416162666c3fa7b162f3/fastapi-0.48.0.tar.gz (mirror.py:875)
2022-11-07 11:02:24,753 INFO: Downloading: https://files.pythonhosted.org/packages/b4/0c/d7ce15c837486502ae146aec1caab3515d6fa434ae2554d88e3d589f180a/fastapi-0.49.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:24,800 INFO: Downloading: https://files.pythonhosted.org/packages/10/7e/691edfb8fc0bc7f201be0dcd92f58836a57abffe3dee10d24adb301c6da7/fastapi-0.49.0.tar.gz (mirror.py:875)
2022-11-07 11:02:25,920 INFO: Downloading: https://files.pythonhosted.org/packages/2f/57/ad36eb7f47c463e3bf5ec57874c14469b6065b1ab2d510f7b2b4219f9148/fastapi-0.49.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:26,002 INFO: Downloading: https://files.pythonhosted.org/packages/db/fd/3d6b6a70892784f7bfba11a3755d7271613d0f0db438dfb4b001f39f21d0/fastapi-0.49.1.tar.gz (mirror.py:875)
2022-11-07 11:02:27,202 INFO: Downloading: https://files.pythonhosted.org/packages/b8/2a/13ca48d28a51ebad2f0e12f9084252ac02d60e7c207737c5fae5d1639d0e/fastapi-0.49.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:27,213 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/b8/2a/13ca48d28a51ebad2f0e12f9084252ac02d60e7c207737c5fae5d1639d0e/fastapi-0.49.2-py3-none-any.whl (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/pypi/web/packages/b8/2a/13ca48d28a51ebad2f0e12f9084252ac02d60e7c207737c5fae5d1639d0e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1180, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/b8/2a/13ca48d28a51ebad2f0e12f9084252ac02d60e7c207737c5fae5d1639d0e'
2022-11-07 11:02:27,217 INFO: Downloading: https://files.pythonhosted.org/packages/c7/97/b0fc25252a3d74c27e392505660ed1d9e36f116a2783dfccd692c759c3b3/fastapi-0.49.2.tar.gz (mirror.py:875)
2022-11-07 11:02:28,463 INFO: Downloading: https://files.pythonhosted.org/packages/c3/31/127290b5be5e72423e8937506830964b50e818a2039db3ec1d46b0abade7/fastapi-0.5.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:28,540 INFO: Downloading: https://files.pythonhosted.org/packages/e4/27/eb1e323b2eb60dab88d93ec6dae4f76970b1e441a70d8573e030ae0da02c/fastapi-0.5.0.tar.gz (mirror.py:875)
2022-11-07 11:02:29,423 INFO: Downloading: https://files.pythonhosted.org/packages/62/20/6f9895adaf7a5c9005eed6c6a6855bf0a39507037b03571dc9285ce16af1/fastapi-0.5.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:29,500 INFO: Downloading: https://files.pythonhosted.org/packages/3e/08/1ac8eda55669cd5c2b2fd343ee81e460cf6a24f49368fd7de28c0c0a6023/fastapi-0.5.1.tar.gz (mirror.py:875)
2022-11-07 11:02:29,514 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/3e/08/1ac8eda55669cd5c2b2fd343ee81e460cf6a24f49368fd7de28c0c0a6023/fastapi-0.5.1.tar.gz (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/pypi/web/packages/3e/08/1ac8eda55669cd5c2b2fd343ee81e460cf6a24f49368fd7de28c0c0a6023'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1180, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/3e/08/1ac8eda55669cd5c2b2fd343ee81e460cf6a24f49368fd7de28c0c0a6023'
2022-11-07 11:02:29,523 INFO: Downloading: https://files.pythonhosted.org/packages/91/56/20f373a15e83e429fbbbf5e09357dde46cb9171f89e76e0ee3eaf4fc3578/fastapi-0.50.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:29,627 INFO: Downloading: https://files.pythonhosted.org/packages/82/fd/f967d9845c85d475e1a3fe228ae72b04e0dcb615638b8b2e10f794f825ff/fastapi-0.50.0.tar.gz (mirror.py:875)
2022-11-07 11:02:30,826 INFO: Downloading: https://files.pythonhosted.org/packages/e7/dc/930093c07ef0ce8fa960139bbbc6cf0b2b4343c5b0faf4222e2e7afa826a/fastapi-0.51.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:30,897 INFO: Downloading: https://files.pythonhosted.org/packages/d8/a4/a192938c358ba12043fb5f34ef9319d4ba776c53bfc4bd893c0d21fdbd9e/fastapi-0.51.0.tar.gz (mirror.py:875)
2022-11-07 11:02:32,081 INFO: Downloading: https://files.pythonhosted.org/packages/18/5b/46c084c174fc69b2a7e1d9c22d014f39fb677d9a7635f24734ef56e0fb53/fastapi-0.52.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:32,144 INFO: Downloading: https://files.pythonhosted.org/packages/7f/3a/d19b7af86f61877c2d976b3b220ae8b2b22c45fa8f0d0787e9f32d40fcc2/fastapi-0.52.0.tar.gz (mirror.py:875)
2022-11-07 11:02:33,328 INFO: Downloading: https://files.pythonhosted.org/packages/f9/eb/f899ccd1f052bb306d3938696458a403ee61a116fd784bc9fb266f6ce211/fastapi-0.53.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:33,401 INFO: Downloading: https://files.pythonhosted.org/packages/ee/fa/a9bdf3410669cb2f10e9fe4e11d1ae13338ed7947745e8b3797a6bff5c8a/fastapi-0.53.0.tar.gz (mirror.py:875)
2022-11-07 11:02:34,567 INFO: Downloading: https://files.pythonhosted.org/packages/cb/19/5bb41e0f33413cf768a63451945a2166ca76c14d4a42f79c7b628d7bfaae/fastapi-0.53.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:34,632 INFO: Downloading: https://files.pythonhosted.org/packages/fd/1c/752cee1610bea575e60fa975ff3d5b535ede94a7742da6f43b9b5c3f6da8/fastapi-0.53.1.tar.gz (mirror.py:875)
2022-11-07 11:02:35,814 INFO: Downloading: https://files.pythonhosted.org/packages/f7/6b/d7c462d6dbf9460981389b02045bdf730f55e079c3497f197db44fef14e4/fastapi-0.53.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:35,881 INFO: Downloading: https://files.pythonhosted.org/packages/e3/aa/d807237eca90e3a1f49139407eda6d7a380a77a6747b1c37265bf7ecf43f/fastapi-0.53.2.tar.gz (mirror.py:875)
2022-11-07 11:02:37,062 INFO: Downloading: https://files.pythonhosted.org/packages/f6/26/fe0ef3f1ba5b0b2264da858a4407ed1fcd94f49655281487e6bd2709a8b7/fastapi-0.54.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:37,130 INFO: Downloading: https://files.pythonhosted.org/packages/f7/b9/6aad3271f8171bc098bf8a184c03d8c376ca6fb6b01a63484d37c17f9f60/fastapi-0.54.0.tar.gz (mirror.py:875)
2022-11-07 11:02:38,333 INFO: Downloading: https://files.pythonhosted.org/packages/f9/45/9c54e719b0625ea13ed921853269ece018dfea9b7a07ddef935bbf716dc9/fastapi-0.54.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:38,391 INFO: Downloading: https://files.pythonhosted.org/packages/4d/54/45770888a4a73666390120b444131568637e70b5e002860bad99fa1dcb27/fastapi-0.54.1.tar.gz (mirror.py:875)
2022-11-07 11:02:39,592 INFO: Downloading: https://files.pythonhosted.org/packages/80/fb/c1109c43c32bac260906e640cfc88e23fba751ba61319af779dd390aaa9b/fastapi-0.54.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:39,660 INFO: Downloading: https://files.pythonhosted.org/packages/be/a9/28297b80b3573662af8ae205c162362aa6a5308dc4ba9e381471590d9b96/fastapi-0.54.2.tar.gz (mirror.py:875)
2022-11-07 11:02:40,888 INFO: Downloading: https://files.pythonhosted.org/packages/29/7f/013c7f28edd70f8825d3e95b8d19467a151cd698977a0f0a36a04110475c/fastapi-0.55.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:40,960 INFO: Downloading: https://files.pythonhosted.org/packages/bf/40/37def65cd25fc032e1b555ed8038d62a3d77613a5c47bc03311e1a9859c7/fastapi-0.55.0.tar.gz (mirror.py:875)
2022-11-07 11:02:42,207 INFO: Downloading: https://files.pythonhosted.org/packages/05/24/29cf80ab90b921fc245118762afc67e561b591a6b8b9523f650a2ddf59e5/fastapi-0.55.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:42,269 INFO: Downloading: https://files.pythonhosted.org/packages/0a/13/785d0e5967128628f1d6c05ba16d4c9bf9506d7fd35bbe5b2e269d2804ab/fastapi-0.55.1.tar.gz (mirror.py:875)
2022-11-07 11:02:43,598 INFO: Downloading: https://files.pythonhosted.org/packages/bd/0e/f313c9973973b74aef39397ce2b3e6c8d4e6e0172da43499ea80692d20b8/fastapi-0.56.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:43,653 INFO: Downloading: https://files.pythonhosted.org/packages/bc/81/1caf557ef8e721c45bb77258eb4dfda715f99bc90dbed2d59e66e5a4c545/fastapi-0.56.0.tar.gz (mirror.py:875)
2022-11-07 11:02:44,905 INFO: Downloading: https://files.pythonhosted.org/packages/10/1b/6995baf88eaf291cd20e6b3a9eaab13173746bd39591e9d0faf25db5469b/fastapi-0.56.1.tar.gz (mirror.py:875)
2022-11-07 11:02:46,186 INFO: Downloading: https://files.pythonhosted.org/packages/55/fe/c4f3d3f4056ed63ec0d852746529f05a9afe740c51bdc23ac037042ebc1b/fastapi-0.57.0.tar.gz (mirror.py:875)
2022-11-07 11:02:47,493 INFO: Downloading: https://files.pythonhosted.org/packages/0a/26/754130d8f3c85f56e98758e91bd848d50389f215cf46e5d08689adb1b9ab/fastapi-0.58.0.tar.gz (mirror.py:875)
2022-11-07 11:02:48,765 INFO: Downloading: https://files.pythonhosted.org/packages/b0/47/a46950e883896d0280f13a4dd5b8bef2c72ebe9fd5509e360c19e2d181bf/fastapi-0.58.1.tar.gz (mirror.py:875)
2022-11-07 11:02:50,060 INFO: Downloading: https://files.pythonhosted.org/packages/9f/c2/31ae21414301c60d0bbb83676d843aa303f2e3bc47987cb471438e8b30b6/fastapi-0.59.0.tar.gz (mirror.py:875)
2022-11-07 11:02:51,321 INFO: Downloading: https://files.pythonhosted.org/packages/c9/68/f9606baa28ff12e65a55ffc76d6978f42e41a8166baa5678822d32cc60b7/fastapi-0.6.0.tar.gz (mirror.py:875)
2022-11-07 11:02:52,238 INFO: Downloading: https://files.pythonhosted.org/packages/db/ff/19c04ede806ff37de2a64590daae0d2bf3fda0be2068eb475a258503b217/fastapi-0.6.1.tar.gz (mirror.py:875)
2022-11-07 11:02:53,133 INFO: Downloading: https://files.pythonhosted.org/packages/2c/a4/5be495c24d80e5f0117182a836cd04c10026b38349f386d4ad7d908852e0/fastapi-0.6.2.tar.gz (mirror.py:875)
2022-11-07 11:02:54,027 INFO: Downloading: https://files.pythonhosted.org/packages/52/ef/3f770b61e25e3863fb0199014b1099728a2d6c887bb9bd59f0383e8333b9/fastapi-0.6.3.tar.gz (mirror.py:875)
2022-11-07 11:02:54,916 INFO: Downloading: https://files.pythonhosted.org/packages/5e/36/d6a12c5532a31b072f69674cb55f52c911410a60ef40687f2a9f06bfbc4f/fastapi-0.6.4.tar.gz (mirror.py:875)
2022-11-07 11:02:55,869 INFO: Downloading: https://files.pythonhosted.org/packages/b0/f7/6abe5e057df3afcaa0566903ae8d60ecad64a13599e937254f23b49e7ea2/fastapi-0.60.0.tar.gz (mirror.py:875)
2022-11-07 11:02:57,165 INFO: Downloading: https://files.pythonhosted.org/packages/f7/c8/fa9ef0711be4f38dd8f381ee3e610330aaf5dfe26c880bf18565d7b1753e/fastapi-0.60.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:57,225 INFO: Downloading: https://files.pythonhosted.org/packages/83/08/3d0113f3c4c0ad650ca438d583187af434f7136843c3ed04216866db21d4/fastapi-0.60.1.tar.gz (mirror.py:875)
2022-11-07 11:02:58,504 INFO: Downloading: https://files.pythonhosted.org/packages/30/1a/0346ce8f8146d6f74d91ad15429f8bee32b7eecc500efae401fcd2578acd/fastapi-0.60.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:02:58,575 INFO: Downloading: https://files.pythonhosted.org/packages/ac/bc/9ec5a7400d02df102d15b53f689993ba491a11b5cfc86c90a4accb019195/fastapi-0.60.2.tar.gz (mirror.py:875)
2022-11-07 11:02:59,967 INFO: Downloading: https://files.pythonhosted.org/packages/ac/af/e10003597b5dd7a98add4b208be133161e0d886db203eca733dbe3b200b2/fastapi-0.61.0.tar.gz (mirror.py:875)
2022-11-07 11:03:01,271 INFO: Downloading: https://files.pythonhosted.org/packages/48/65/454fb440d48098845875b5ba8599efafee1efabb97720a584c78674e6d26/fastapi-0.61.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:01,318 INFO: Downloading: https://files.pythonhosted.org/packages/da/32/ea2b86e56674ff3d1bb0bbb2e3b74a04694ac0be3331e9f9d431c350bcc2/fastapi-0.61.1.tar.gz (mirror.py:875)
2022-11-07 11:03:01,331 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/da/32/ea2b86e56674ff3d1bb0bbb2e3b74a04694ac0be3331e9f9d431c350bcc2/fastapi-0.61.1.tar.gz (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/pypi/web/packages/da/32/ea2b86e56674ff3d1bb0bbb2e3b74a04694ac0be3331e9f9d431c350bcc2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1180, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/da/32/ea2b86e56674ff3d1bb0bbb2e3b74a04694ac0be3331e9f9d431c350bcc2'
2022-11-07 11:03:01,335 INFO: Downloading: https://files.pythonhosted.org/packages/4c/0b/5df17eaadb7fe39dad349f484e551e802ce0581be672822f010c530d5e75/fastapi-0.61.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:01,448 INFO: Downloading: https://files.pythonhosted.org/packages/36/0c/888ebd0cdef9045b811c3e06b3860742a41d4b9699ba4e0b81fa82d9d817/fastapi-0.61.2.tar.gz (mirror.py:875)
2022-11-07 11:03:02,922 INFO: Downloading: https://files.pythonhosted.org/packages/09/9b/e45a36554f7452f6d372e607a647c3827434f5b353ef4630a7432514c928/fastapi-0.62.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:03,015 INFO: Downloading: https://files.pythonhosted.org/packages/6a/7c/b694a1e8d0620cfd9ad1f50e64f69f6b1317f55f7583931599ba7d36d2b8/fastapi-0.62.0.tar.gz (mirror.py:875)
2022-11-07 11:03:04,262 INFO: Downloading: https://files.pythonhosted.org/packages/9f/33/1b643f650688ad368983bbaf3b0658438038ea84d775dd37393d826c3833/fastapi-0.63.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:04,364 INFO: Downloading: https://files.pythonhosted.org/packages/7f/5b/23057d0cdeac3643c08937ef1348590e284842636c837108971d8af6b51d/fastapi-0.63.0.tar.gz (mirror.py:875)
2022-11-07 11:03:05,613 INFO: Downloading: https://files.pythonhosted.org/packages/0f/cc/929a628d250bc81ff39aa2a3f8d66b3ea2e07c3c93135e9b0303058ccd59/fastapi-0.64.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:05,688 INFO: Downloading: https://files.pythonhosted.org/packages/a1/fe/064bdb5598ad9dfe942b58884296bcd7b81d1e13e5af48fd328e38b2de97/fastapi-0.64.0.tar.gz (mirror.py:875)
2022-11-07 11:03:07,184 INFO: Downloading: https://files.pythonhosted.org/packages/2c/2a/4b5dcc2eabaa439f81387b0d3f80ebb2ea52b0436720b713fef79db32b7e/fastapi-0.65.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:07,254 INFO: Downloading: https://files.pythonhosted.org/packages/8d/31/eb754cf36c72ad0b46ca3e8de2ceadabfaee85cec74cebe226280c40f1dc/fastapi-0.65.0.tar.gz (mirror.py:875)
2022-11-07 11:03:08,744 INFO: Downloading: https://files.pythonhosted.org/packages/4e/b9/a91a699f5c201413b3f61405dbccc29ebe5ad25945230e9cec98fdb2434c/fastapi-0.65.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:08,824 INFO: Downloading: https://files.pythonhosted.org/packages/3a/e5/5eee1f9204886acb45da099f91bf34985b2c536b2746ef9bc3a937751ff7/fastapi-0.65.1.tar.gz (mirror.py:875)
2022-11-07 11:03:10,327 INFO: Downloading: https://files.pythonhosted.org/packages/dc/a8/a6be420593c4061c086e6d2ba47db46401d9af2b98b6cd33d35284f706d3/fastapi-0.65.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:10,398 INFO: Downloading: https://files.pythonhosted.org/packages/8c/5d/66e0777c6702d0f1bc238aca5ef4f8914eb564b27a04c87e158cd26bd69a/fastapi-0.65.2.tar.gz (mirror.py:875)
2022-11-07 11:03:10,414 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/8c/5d/66e0777c6702d0f1bc238aca5ef4f8914eb564b27a04c87e158cd26bd69a/fastapi-0.65.2.tar.gz (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/8c/5d/66e0777c6702d0f1bc238aca5ef4f8914eb564b27a04c87e158cd26bd69a'
2022-11-07 11:03:10,418 INFO: Downloading: https://files.pythonhosted.org/packages/54/d2/1e5d77c3c16dad9582c4c86722c998b12ba7937e68e7e212dca71808626a/fastapi-0.65.3-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:10,536 INFO: Downloading: https://files.pythonhosted.org/packages/59/27/c7402c3e9ba3db1a6e7de187ad9de5d9c3b5ee6582fac43267e0c67f8937/fastapi-0.65.3.tar.gz (mirror.py:875)
2022-11-07 11:03:11,989 INFO: Downloading: https://files.pythonhosted.org/packages/f8/cb/2682d18ac7a553a0203f48190f474ab6dde6cd8815ac6822a46d611e6661/fastapi-0.66.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:12,108 INFO: Downloading: https://files.pythonhosted.org/packages/63/30/c5c72431045f502d523bdf749ce9afa90ee1591426fcdd3db05f216ca8b7/fastapi-0.66.0.tar.gz (mirror.py:875)
2022-11-07 11:03:13,646 INFO: Downloading: https://files.pythonhosted.org/packages/1b/a0/51ad4a441b28f3c3bfa0b864df4c09c9ba54e67e066f39028130fe3beff7/fastapi-0.66.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:13,762 INFO: Downloading: https://files.pythonhosted.org/packages/f3/84/ca8505d899255f00eb5c8b21817b3a77a946b22c0c469f20b54c03fb47e2/fastapi-0.66.1.tar.gz (mirror.py:875)
2022-11-07 11:03:15,369 INFO: Downloading: https://files.pythonhosted.org/packages/52/be/2a26007dc86c51e87d70021f6c1b3442726c5918fe57d27927badf687122/fastapi-0.67.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:15,433 INFO: Downloading: https://files.pythonhosted.org/packages/49/48/26f1b57a5742b364e33ba2908faf794b024658cf26ac340ae692edd4e677/fastapi-0.67.0.tar.gz (mirror.py:875)
2022-11-07 11:03:16,941 INFO: Downloading: https://files.pythonhosted.org/packages/c1/66/5275b973ab5704e47e6735051f30c1e0c827edec2825b84e77cc21c8ba6c/fastapi-0.68.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:17,016 INFO: Downloading: https://files.pythonhosted.org/packages/64/df/939fed987038be2344d1187b666f7ecf172f81d6c9327357ccce05f4e40f/fastapi-0.68.0.tar.gz (mirror.py:875)
2022-11-07 11:03:18,467 INFO: Downloading: https://files.pythonhosted.org/packages/df/44/ee1976b03404318590bbe4b0ef27007ce2c42b15757aa0c72bc99a4ebae7/fastapi-0.68.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:18,541 INFO: Downloading: https://files.pythonhosted.org/packages/e0/2a/03c3e1ede3aa83c45da1d156b0180288e573f5e2643ec4fd6d697cebebdf/fastapi-0.68.1.tar.gz (mirror.py:875)
2022-11-07 11:03:20,089 INFO: Downloading: https://files.pythonhosted.org/packages/92/a8/10d8a171afa66e6e8a5cfe82771d11c4c84a2a874c19c0b55c7099001fac/fastapi-0.68.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:20,166 INFO: Downloading: https://files.pythonhosted.org/packages/00/6e/4a1dd8aeb5f673f9b772b64a35e7d11e94057306caf3108d52fd47eac7cf/fastapi-0.68.2.tar.gz (mirror.py:875)
2022-11-07 11:03:21,776 INFO: Downloading: https://files.pythonhosted.org/packages/cc/84/23b7d8b4e0b389d74b8cc5dbd2b40d44e275f1e6d04d547f8ac1f89c2d9c/fastapi-0.69.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:21,846 INFO: Downloading: https://files.pythonhosted.org/packages/c6/51/7b4d02f3936a4693b871b7b7ae41f19f770c850b54cc5d0b0e98b976e51f/fastapi-0.69.0.tar.gz (mirror.py:875)
2022-11-07 11:03:23,694 INFO: Downloading: https://files.pythonhosted.org/packages/e1/2c/009a8c8b5b3a1d03ee0e65d82d0a0e44c4a937c101fa0397efac12a1f4a3/fastapi-0.7.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:23,787 INFO: Downloading: https://files.pythonhosted.org/packages/39/c8/4021029cc8a20d83048ec4f9aff57061508a00043590b76164fdf983c674/fastapi-0.7.0.tar.gz (mirror.py:875)
2022-11-07 11:03:24,804 INFO: Downloading: https://files.pythonhosted.org/packages/a3/d2/1d27313c69cfb998521a476931266d84b7bd59d479f7ea6bfbe2d94d9964/fastapi-0.7.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:24,891 INFO: Downloading: https://files.pythonhosted.org/packages/5c/9f/0a27b476708aa6c47abefe74a2ef5db14205bd5938d736e7aea9e6ff6bd1/fastapi-0.7.1.tar.gz (mirror.py:875)
2022-11-07 11:03:25,905 INFO: Downloading: https://files.pythonhosted.org/packages/72/44/a35678119f8a6b25f12224bc55463ecc813c4a0a2c3546818890a50db482/fastapi-0.70.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:25,978 INFO: Downloading: https://files.pythonhosted.org/packages/e6/35/384bc3bf9a4926a45052037e0bf0806ffeda237b8b731cb04a5e1e1c6a45/fastapi-0.70.0.tar.gz (mirror.py:875)
2022-11-07 11:03:27,848 INFO: Downloading: https://files.pythonhosted.org/packages/90/89/06fe73e915538261626ac4b95b5b8f61938b6150cbed5f9f99c5a16f9719/fastapi-0.70.1.tar.gz (mirror.py:875)
2022-11-07 11:03:29,483 INFO: Downloading: https://files.pythonhosted.org/packages/d7/84/55a906526454816a5d194c8572cb4c4cca8d56f8a1192bf87a9ac101238f/fastapi-0.71.0.tar.gz (mirror.py:875)
2022-11-07 11:03:31,076 INFO: Downloading: https://files.pythonhosted.org/packages/81/da/d3df7f1e153638842a3cf4b1ed0172680d24d6ecd4491ccb8cc132b90778/fastapi-0.72.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:31,085 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/81/da/d3df7f1e153638842a3cf4b1ed0172680d24d6ecd4491ccb8cc132b90778/fastapi-0.72.0-py3-none-any.whl (mirror.py:686)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/pypi/web/packages/81/da/d3df7f1e153638842a3cf4b1ed0172680d24d6ecd4491ccb8cc132b90778'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1180, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/81/da/d3df7f1e153638842a3cf4b1ed0172680d24d6ecd4491ccb8cc132b90778'
2022-11-07 11:03:31,094 INFO: Downloading: https://files.pythonhosted.org/packages/af/d8/34d3939d705060c5a0d714db371cf863568c97f88a059489baa442e79c1a/fastapi-0.72.0.tar.gz (mirror.py:875)
2022-11-07 11:03:32,972 INFO: Downloading: https://files.pythonhosted.org/packages/27/a7/439a8891b22b4f1e45483b8a3b209f2eabcfdd44e75c339d5f2d315342fa/fastapi-0.73.0.tar.gz (mirror.py:875)
2022-11-07 11:03:34,404 INFO: Downloading: https://files.pythonhosted.org/packages/3f/c1/7c8107f72bcee63f61ce2fe9bb6503a0f67a893c63e063e184f592b9c532/fastapi-0.74.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:34,488 INFO: Downloading: https://files.pythonhosted.org/packages/2a/d4/6a88d73df0b26b6b00378efe6bbde410377f0d3338834e6c7115e86e858a/fastapi-0.74.0.tar.gz (mirror.py:875)
2022-11-07 11:03:36,138 INFO: Downloading: https://files.pythonhosted.org/packages/99/e0/ef1628c4aa96b0a1a2c81a3cac501ebd58849516ea6d1d0e4faf87bf3b34/fastapi-0.74.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:36,231 INFO: Downloading: https://files.pythonhosted.org/packages/c8/ce/07fc3196b05948e16fbc46944728218d4af9a7fb6536e74804c26e303eb8/fastapi-0.74.1.tar.gz (mirror.py:875)
2022-11-07 11:03:37,904 INFO: Downloading: https://files.pythonhosted.org/packages/4f/46/226355e82ccb4be82e06269e9a546f16c1d87fbda2286fb5d36a1c31af9e/fastapi-0.75.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:37,973 INFO: Downloading: https://files.pythonhosted.org/packages/6d/0b/64ecaddaa9d6d4bbf40422664650f7ec45de9281306c69f22f16f92c510c/fastapi-0.75.0.tar.gz (mirror.py:875)
2022-11-07 11:03:40,937 INFO: Downloading: https://files.pythonhosted.org/packages/2c/1c/195178dfef05e6688d9201aca157b5b8a23171dfc685450fba138547d465/fastapi-0.75.2.tar.gz (mirror.py:875)
2022-11-07 11:03:42,525 INFO: Downloading: https://files.pythonhosted.org/packages/7f/6d/9986b5ebbef1872885d9bd76e7ce24def372beddfe5d6e98b5ae36f2e46f/fastapi-0.76.0.tar.gz (mirror.py:875)
2022-11-07 11:03:44,421 INFO: Downloading: https://files.pythonhosted.org/packages/db/9e/67413d504b1cc044881e7d181c0206a27dd3534fcf6fdf92443477473986/fastapi-0.77.0.tar.gz (mirror.py:875)
2022-11-07 11:03:46,086 INFO: Downloading: https://files.pythonhosted.org/packages/16/5b/d45cca598cec06bead962b84648cbdb0c0d966f2453694bce6f5ce4c0bc9/fastapi-0.77.1-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:46,146 INFO: Downloading: https://files.pythonhosted.org/packages/eb/ab/bdf0fbdbeb1badc95fd1d6510c575457e53efc23c67021684002200e0aa3/fastapi-0.77.1.tar.gz (mirror.py:875)
2022-11-07 11:03:48,262 INFO: Downloading: https://files.pythonhosted.org/packages/61/0f/427f0af121b226e62237e430f5bf4485e0ae1565b3f5b782613b59f30abc/fastapi-0.78.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:48,325 INFO: Downloading: https://files.pythonhosted.org/packages/b7/2a/4676736c3b1864ca483265db43cad9710b83a95c2530aa6bfe671b83fd46/fastapi-0.78.0.tar.gz (mirror.py:875)
2022-11-07 11:03:50,371 INFO: Downloading: https://files.pythonhosted.org/packages/ae/ed/ea37410618f1c206ba857d391d7b2de6de7a758ea586662aef77c945d3b4/fastapi-0.79.0.tar.gz (mirror.py:875)
2022-11-07 11:03:52,244 INFO: Downloading: https://files.pythonhosted.org/packages/48/eb/feec98de25762193702e909535fbf4a1b3cb63617e6ee2e72c95ae4789ba/fastapi-0.79.1.tar.gz (mirror.py:875)
2022-11-07 11:03:54,925 INFO: Downloading: https://files.pythonhosted.org/packages/4d/58/e91d3dc0b12f26ea73891fd07c5b2d8a3492e732fefe506584e24de9d8e3/fastapi-0.8.0.tar.gz (mirror.py:875)
2022-11-07 11:03:55,902 INFO: Downloading: https://files.pythonhosted.org/packages/82/93/6bd83151004f2188e016589e4e4f48890087cdf84b3a25212216773258f1/fastapi-0.80.0.tar.gz (mirror.py:875)
2022-11-07 11:03:58,233 INFO: Downloading: https://files.pythonhosted.org/packages/d0/64/63e77ebc5c641dd3d021acd658de8b8710060c742b3557c02f8b4df272a2/fastapi-0.81.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:03:58,314 INFO: Downloading: https://files.pythonhosted.org/packages/cb/3b/f77f6c450882b23343552d06ad09658d14d535dd6b571fa09b8444cb4628/fastapi-0.81.0.tar.gz (mirror.py:875)
2022-11-07 11:04:00,790 INFO: Downloading: https://files.pythonhosted.org/packages/56/c7/e36aa8a7a04a2536b559abd7ced3a69fbabb324b27911b7a4c50276167cf/fastapi-0.82.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:04:00,859 INFO: Downloading: https://files.pythonhosted.org/packages/10/95/33ddb2614d59317a18501ef3f598be0e726f2ce7993feadbdf076a9413a1/fastapi-0.82.0.tar.gz (mirror.py:875)
2022-11-07 11:04:03,266 INFO: Downloading: https://files.pythonhosted.org/packages/bc/34/6746797c8b5275649f99ed57e5a6dd60f912649c0f8ac985b362efd9e8c7/fastapi-0.83.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:04:03,332 INFO: Downloading: https://files.pythonhosted.org/packages/61/4e/4913c1672374a8161af5e0f865f1a33d8faa185aecaf61b8c288dca1d4fa/fastapi-0.83.0.tar.gz (mirror.py:875)
2022-11-07 11:04:05,991 INFO: Downloading: https://files.pythonhosted.org/packages/6b/1c/e80add75c4c8383d6041a10ca5ee9cdb185ac9d7a8f9ffa94975010e72ca/fastapi-0.84.0.tar.gz (mirror.py:875)
2022-11-07 11:04:08,415 INFO: Downloading: https://files.pythonhosted.org/packages/f8/f6/5334a17a8acb95b2b71825db6c8c3d2b984d1d0c31266fcda02480ab62ab/fastapi-0.85.0-py3-none-any.whl (mirror.py:875)
2022-11-07 11:04:08,504 INFO: Downloading: https://files.pythonhosted.org/packages/f2/74/15805b9921c0fa99a732bbab71fc548a269d027b6540f48043f87338a273/fastapi-0.85.0.tar.gz (mirror.py:875)
2022-11-07 11:04:11,223 INFO: Downloading: https://files.pythonhosted.org/packages/76/06/167f461067a95fb631fdf65657089364dc43844bc38d8c39374c3abbc746/fastapi-0.85.1.tar.gz (mirror.py:875)
2022-11-07 11:04:13,552 INFO: Downloading: https://files.pythonhosted.org/packages/99/d4/4256804791b1e822f46bf79015270e9e9f14e9b634cb8231a88f6dc100c3/fastapi-0.85.2-py3-none-any.whl (mirror.py:875)
2022-11-07 11:04:13,620 INFO: Downloading: https://files.pythonhosted.org/packages/f9/7a/8a2cd82257017a4c15f13a8e831bc22c52cc0a3e9fad5a2cc00c491e733f/fastapi-0.85.2.tar.gz (mirror.py:875)
2022-11-07 11:04:16,248 INFO: Downloading: https://files.pythonhosted.org/packages/3d/c4/0cbe3b1213cdb4b3dd8692ed30788b01bf49d88d71afcb1c296a608f6029/fastapi-0.86.0.tar.gz (mirror.py:875)
2022-11-07 11:04:18,920 INFO: Downloading: https://files.pythonhosted.org/packages/d7/2b/08583e9aeaacde54a1bc814a29916e390ebd36f7d54d53e9ef5b3236d21c/fastapi-0.9.0.tar.gz (mirror.py:875)
2022-11-07 11:04:19,956 INFO: Downloading: https://files.pythonhosted.org/packages/98/c7/5c98837ca82d4e49c16891100724328931bd07b79e61528291bbf94436bb/fastapi-0.9.1.tar.gz (mirror.py:875)
2022-11-07 11:04:20,958 ERROR: Error syncing package: fastapi@15646760 (mirror.py:377)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 130, in package_syncer
    await self.process_package(package)
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 337, in process_package
    await self.sync_release_files(package)
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 693, in sync_release_files
    raise deferred_exception  # raise the exception after trying all files
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 662, in sync_release_files
    downloaded_file = await self.download_file(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 879, in download_file
    dirname.mkdir(parents=True)
  File "/usr/local/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/srv/pypi/web/packages/71/48/64f0a29bd601a90e323562547a6989db221ad5f69c77e0b7565f8ec477d0'
2022-11-07 11:04:20,959 INFO: Generating global index page. (simple.py:223)
2022-11-07 11:04:21,118 INFO: 2 packages had changes (mirror.py:990)
2022-11-07 11:04:21,118 INFO: Writing diff file to mirrored-files (mirror.py:1000)

Environment configuration

bandersnatch version: 6.0.1

docker-compose file:

services:
  bandersnatch:
    image: pypa/bandersnatch:latest
    platform: linux/amd64
    volumes:
      - "./config:/conf"
      - "./data:/srv/pypi/web"
  bandersnatch_nginx:
    image: bandersnatch_nginx
    build: .
    platform: linux/amd64
    volumes:
      - "./data:/data/pypi/web:ro"
    ports:
      - "40080:80"

Config file:

[mirror]
directory = /srv/pypi
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 10
global-timeout = 1800
workers = 3
hash-index = false
simple-format = ALL
stop-on-error = false
storage-backend = filesystem
verifiers = 3
compare-method = hash

[plugins]
enabled =
    allowlist_project
    allowlist_release

[allowlist]
packages =
    typer[all]
    fastapi[all]
    docker[all]

bug help wanted

opened by tibeer 11

Change to re.match usage to re.search

Using re.search instead of re.match allows for non regex formatted matching (static strings) and non-anchored regex matching. This also eliminates the need for .* all over the place when making patterns.

opened by lps-rocks 11

Failed to mirror package via Proxy Server since version 4.0.0

Since version 4.0.0 of bandersnatch , I can't use my proxy with environment variables http_proxy or https_proxy

root@01e29aacbf76:/# bandersnatch --debug mirror
2020-05-04 11:36:23,022 DEBUG: Initializing Master's aiohttp ClientSession
2020-05-04 11:36:23,023 INFO: Status file /srv/pypi/status missing. Starting over.
2020-05-04 11:36:23,023 INFO: Syncing with https://pypi.org.
2020-05-04 11:36:23,023 INFO: Current mirror serial: 0
2020-05-04 11:36:23,024 INFO: Syncing all packages.
2020-05-04 11:36:23,031 DEBUG: Closing Master's aiohttp ClientSession
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 494, in sock_connect
    return await fut
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 526, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 443)
.....

bug

opened by alebourdoulous 11

Got no project package links index.html using allowlist plugin

my bandersnatch.conf:

[mirror]
directory = /opt/bandersnatch
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 1000
global-timeout = 3600
workers = 3
hash-index = false
stop-on-error = false
storage-backend = filesystem
; log-config = /opt/bandersnatch/bandersnatch-log.conf
; root_uri = https://example.com
verifiers = 3
; keep_index_versions = 0
; vim: set ft=cfg:
; diff-file = /srv/pypi/mirrored-files
; diff-append-epoch = true

[plugins]
enabled =
    blocklist_project
    blocklist_release
    allowlist_project
    allowlist_release
[blocklist]

; https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html
; https://pypi.org/stats/
[allowlist]
packages =
    cntk
    tensorflow-gpu
    tensorflow
    tensorflow-cpu
    torch

bug help wanted

opened by JJLAAA 10

Consistently sync failure from pypi.org as we didn't handle aiohttp timeouts correctly

I'm using bandersnatch to sync from pypi.org every 5 mins. I noticed the mirror work always failed on syncing corona-cli since yesterday.

Is it an issue caused by configuration and any workaround is suggested?

2020-03-31 11:23:35,098 ERROR: Error syncing package: corona-cli@6911612
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/bandersnatch/package.py", line 145, in sync
    metadata_response = await metadata_generator.asend(None)
  File "/usr/local/lib/python3.7/site-packages/bandersnatch/master.py", line 105, in get
    await self.check_for_stale_cache(path, required_serial, got_serial)
  File "/usr/local/lib/python3.7/site-packages/bandersnatch/master.py", line 74, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib64/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib64/python3.7/site-packages/aiohttp/client.py", line 404, in _request
    tm = TimeoutHandle(self._loop, real_timeout.total)
AttributeError: 'float' object has no attribute 'total'

bug

opened by zxkane 10

Move Docker to 3.11

We work in 3.11 so might as well start building docker in the faster available Python Runtime.

Please report any bugs!

Will look to go >= 3.10 later this year ... And move code to take advantage of that if we can.
enhancement

opened by cooperlees 2
Bump sphinx from 5.3.0 to 6.1.1
Bumps sphinx from 5.3.0 to 6.1.1.

Release notes

Sourced from sphinx's releases.

v6.1.1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.1.0

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b2

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

Changelog

Sourced from sphinx's changelog.

Release 6.1.1 (released Jan 05, 2023)

Bugs fixed

#11091: Fix util.nodes.apply_source_workaround for literal_block nodes with no source information in the node or the node's parents.

Release 6.1.0 (released Jan 05, 2023)

Dependencies

Adopted the Ruff_ code linter.

.. _Ruff: https://github.com/charliermarsh/ruff

Incompatible changes

#10979: gettext: Removed support for pluralisation in get_translation. This was unused and complicated other changes to sphinx.locale.

Deprecated

sphinx.util functions:

Renamed sphinx.util.typing.stringify() to sphinx.util.typing.stringify_annotation()

Moved sphinx.util.xmlname_checker() to sphinx.builders.epub3._XML_NAME_PATTERN

Moved to sphinx.util.display:

sphinx.util.status_iterator

sphinx.util.display_chunk

sphinx.util.SkipProgressMessage

sphinx.util.progress_message

Moved to sphinx.util.http_date:

sphinx.util.epoch_to_rfc1123

sphinx.util.rfc1123_to_epoch

Moved to sphinx.util.exceptions:

sphinx.util.save_traceback

... (truncated)

Commits

77aaa86 Bump to 6.1.1 final

476c115 Suppress ValueError in apply_source_workaround (#11092)

c80d656 Bump version

4e1004a Bump to 6.1.0 final

a2176d4 Fix deprecation warnings

2c104e9 Merge branch '6.0.x'

a27d262 Bump to 6.0.1 final

821569e Add note for Pygments

222d366 imgmath: Fix relative file path (#10965)

c499f66 Add SIM113 lint (#11057)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

skip news
opened by dependabot[bot] 2
heavy io doing glob('**') in /web/simple for global index
https://github.com/pypa/bandersnatch/blob/16137fd3861dd41cd34c060989140edeb00f8e43/src/bandersnatch/simple.py#L78-L89

The function shown above generate a package list for the global indexes i.e. /web/simple/index{.html,_v1.html,_v1.json}.

According to the pathlib.Path.glob document:

Note: Using the “**” pattern in large directory trees may consume an inordinate amount of time.

This function is extremely slow (minutes) for a frequent mirror on HDD. bandersnatch now do this slow glob in a directory having over 400,000 sub-directories even for just 1 package update. Some tracing shows that, each sub-directory (and possibly the index.html file) is stat()ed during this process.

It holds in my mirror that, each sub-directory in /web/simple contains a index.html, so I think it's sufficient to just readdir(simple_dir) for a comprehensive package list here. So I tried to replace the set comprehension to the one below:

{ x.name for x in simple_path.iterdir() if 'index.' not in x.name }

After this change, generating global index now takes only seconds. But I'm still not sure if there's any other considerations or legacy problems preventing doing this, hence this issue instead of a direct PR is opened.
enhancement help wanted
opened by TechCiel 3
Make CI pass in 3.11 + Docker build in 3.11
We do not pass in 3.11 ... Lets make it after latest packaging ...

Known issues:

s3path: https://github.com/liormizr/s3path/issues/112

Once solved and integration tests pass lets change Docker to default to 3.11 too.
bug enhancement help wanted
opened by cooperlees 1
How to count the size of pypi directory

At present, I can obtain the official directory size of pypi（https://pypi.org/stats/）, while I am synchronizing the pypi directory. However, the du or duc command takes too long to count. Is there a more convenient way to do this?
invalid question

opened by leochen12-rgb 3

Releases(6.1.0)

6.1.0(Jan 3, 2023)
New Features

Add a packages config under [filter_prerelease] for prerelease filter to filter only specified packages. PR #1328

Dropped Support

Dropped LegacyVersion support for latest_name plugin PR #1315

No longer issue PURGE requests on StalePage exceptions as PyPI now requires authentication PR #1335

CI

Format with black --preview style PR #1313

I contribute to black and want to help find bugs ...

Source code(tar.gz)
Source code(zip)
6.0.1(Oct 24, 2022)
Bug Fixes

Fixed JSON only mirroring adding correct path to diff_file_list PR #1248

Fixed requirements file parsing when it contains pip options PR #1231

New Features

Added the possibility to limit the time range in which the docker image runner can execute the bandersnatch mirroring activity. This can be done by providing an optional command line argument with like --hours-range 23-7 where the interval is expressed as <start_hour>-<end_hour>. PR #1232

Add support for globbing in the allowed requirements list. User can specify requirements*.txt or *.txt to merge multiple requirements files PR #1230

Source code(tar.gz)
Source code(zip)
6.0.0(Sep 1, 2022)
Add PEP691 simple index support PR #1154

Add example nginx serving for PEP 691 PR #1161

Big change to Simple API generation code and Simple API storage will double with the default of ALL formats. Please tune your config if you only want one format.
Source code(tar.gz)
Source code(zip)
5.3.0(Jul 29, 2022)
New Features

Add delete and verify support in s3 (and maybe swift) storage backend PR #1142

Add --skip-simple-root option for bandersnatch sync command PR #1145

Other

Replace pyup with dependabot

Resulted in lots of dependencies updating

Docs typo fixes

Source code(tar.gz)
Source code(zip)
5.2.0(May 2, 2022)
5.2.0 (2022-05-02)

New Features

Add PEP 629 header to Simple API HTML PR #1122

Added support to exclude specific Python minor versions PR #1110 - Thanks davidkim83

Add S3 Docker Image building PR #1092

Other

Got Swift usage documentation

Move Docker containers to Python 3.10 PR #1092

Python 3.10 is now supported PR #1073 - Thanks isidentical

Move minio docker image running to official image

Source code(tar.gz)
Source code(zip)
5.1.1(Dec 15, 2021)
Bug fixes

Fix debug output for package changes at end of mirror operation PR #1066 -- Thanks forky2

Other

Update lots of dependency version pins

Fix documentation build CI

Source code(tar.gz)
Source code(zip)
5.1.0(Nov 3, 2021)
New Features

Add support of downloading packages from alternative mirror PR #928 -- Thanks happyaron

Add support of s3 storage PR #886 -- Thanks LeoQuote

Bug fixes

Add bandersnatch command line help to the documentation main page PR #920 - Thanks ichard26

Generate data-yanked tag in simple page PR #931 - Thanks happyaron

Protect repository metadata from being trashed when disk is full PR #962 - Thanks happyaron

Fix tox to used pinned requirements*.txt files for deps - PR #1011 - Thanks cooperlees

Documentation

Add Storage Options documentation PR #1006 - Thanks cooperlees

Update all dependencies to latest except pyparsing + docutils

Dependencies

Change pyup to run monthly + auto add skip_news PR #1043 - Thanks cooperlees

Source code(tar.gz)
Source code(zip)
5.0.0(Apr 28, 2021)
New Features

bandersnatch is now a >= 3.8 Python project

New size_project_metadata filter plugin, which can deny download of projects larger than defined threshold - PR #806

Add option to compare file size and upload time instead of sha256sum for downloading - PR #822

Add optional uvloop support - PR #891 - Thanks cooperlees

Move to official docker upload action w/arm64 images uploaded - PR #896 - Thanks cooperlees

Deprecations

blacklist/whitelist will no longer work in bandersnatch configuration - PR #897 - Thanks cooperlees

Please use allowlist/denylist respectively

Bug Fixes

Unused storage plugins are loaded and cause non-fatal errors if dependencies are missing - PR #799 - Thanks electricworry

Replaced usages of asynctest with unittest.mock in tests - PR #807 and PR #856 - Thanks ichard26

Remove debugging line that loads entire files into memory. - PR #858 - Thanks asrp

Removed terrible isinstance check of unittest.Mock in mirror.py - PR #859 - Thanks ichard26

Put potential time consuming IO operations into executor - PR #877

Migrated Markdown documentation from recommonmark to MyST-Parser + docs config clean up - PR #879 - Thanks ichard26

Use shutil.move() for temp file management - PR #883 - Thanks happyaron

Fixed logging bug in SizeProjectMetadataFilter to show it activated - PR #889 - Thanks cooperlees

Attempt to wrap all potentially block calls in a ThreadPoolExecutor - PR #894 - Thanks cooperlees

Source code(tar.gz)
Source code(zip)
4.4.0(Dec 31, 2020)
New Features

Build a swift and non swift docker image - PR #754

Split Docker Build to accept build args to optionally include swift support - PR #741 - Thanks nlaurance-pyie

Slimmer docker image - PR #738 - Thanks nlaurance-pyie

Renamed black/white to block/allow lists - PR #737 - Thanks nlaurance-pyie

packages allowlist can be defined from requirements like files - PR #739 - Thanks nlaurance-pyie

Simplify logging around filters - PR #678 - Thanks @dalley

Bug Fixes

Handling of timeouts that can occur in verify. - PR #785 - Thanks electricworry

Added retry logic on timeouts when fetching metadata - PR #773 - Thanks gerrod3

Fix links, improve docs CI, and improve external object linking - PR #776 - Thanks ichard26

Handle 404 status for json verify - PR #763 - Thanks electricworry

Clean up isort config after upgrade to 5+ - PR #767 - Thanks ichard26

Remove duplicate max() target serial finding code + update typing - PR #745

swift.py: use BaseFileLock's lock_file property - PR #699 - Thanks hauntsaninja

Move to latest isort + mypy fixes - PR #706

Update change log url in project metadata - PR #673 - Thanks @abn

Source code(tar.gz)
Source code(zip)
4.3.0(Aug 25, 2020)
New Features

Add SOCKS proxy support to aiohttp via aiohttp-socks - PR #668

Add support for skipping mirroring release files (metadata only) - PR #670 - Thanks @abn

Bug Fixes

Move GitHub actions to v2 tags - PR #666 - Thanks @ryuichi1208

Source code(tar.gz)
Source code(zip)
4.2.0(Aug 21, 2020)
New Features

Thanks to RedHat engineers @dalley + @gerrod3 for all this refactor work in PR #591

New generic Mirror class to perform Python metadata syncing

(previous Mirror class has been renamed to BandersnatchMirror)

Package's filter methods are now part of its public API

New errors.py file to house Bandersnatch specific errors

Internal API Changes

Old Mirror class has been renamed to BandersnatchMirror. Performs same functionality with use of new Mirror API.

BandersnatchMirror now performs all filesystem operations throughout the sync process including the ones previously in Package.

Package no longer performs filesystem operations. Properties json_file, json_pypi_symlink, simple_directory and methods save_json_metadata, sync_release_files, gen_data_requires_python, generate_simple_page, sync_simple_page, _save_simple_page_version, _prepare_versions_path, _file_url_to_local_url, _file_url_to_local_path, download_file have all been moved into BandersnatchMirror. Package's sync has been refactored into Bandersnatch's process_package.

Package class is no longer created with an instance of Mirror

StaleMetadata exception has been moved to new errors.py file

PackageNotFound exception has been moved to new errors.py file

Bug Fixes

Fixed Fix latest_release plugin to ensure latest version is included - PR #660 - Thanks @serverwentdown

Source code(tar.gz)
Source code(zip)
4.1.1(Aug 12, 2020)
Fixed name parsing issue for allow/blocklist project filters - PR #651 - Thanks @gerrod3

Source code(tar.gz)
Source code(zip)
4.1.0(Aug 10, 2020)
New Features

bandersnatch is now 100% type annotated - PRs #546 #561 #592 #593 - Thanks @ichard26 + @rkm

Move to storage abstraction - PR #445 - Thanks @techalchemy

Can now support more than just filesystem e.g. swift

Add sync subcommand to force a sync on a particular PyPI package - PR #572 - Thanks @z4yx

Added new allowlist filter - PR #626 - Thanks @gerrod3

Make webdir/pypi/json/PKG symlinks relative - PR #637 - Thanks @indrat

Makes mirror files more portable

Add main and program name override to ArgumentParser - PR #643 - Thanks @rkm

Allow non pkg_resources install to work

Internal API Changes

Refactored the removal of releases for release_plugins to happen inside of Package PR #608 - Thanks @gerrod3

Minor refactor of Package class PR #606 - Thanks @dralley

Refactored filter loading into seperate class PR #599 - Thanks @gerrod3

Move legacy directory cleanup to mirror.py PR #586

Move verify to use Master for HTTP calls - PR #555

Move http request code for package metadata to master.py - PRs #550 - Thanks @dralley

Bug Fixes

Fixed allow/blocklist release filtering pre-releases - PR #641 - Thanks @gerrod3

Casefold (normalize per PEP503) package names in blacklist/whitelist plugins config - PR #629 - Thanks @lepaperwan

Fix passing package info to filters in verify action. PR #638 - Thanks @indrat

Fix todo file removal - PR #571

Introduce a new global-timeout config option for aiohttp coroutines - Default 5 hours - PR #540 - Thanks @techalchemy

Many doc fixes - PRs #542 #551 #557 #605 #628 #630 - Thanks @pgrimaud + @ichard26 + @hugovk

Move to setting timeout only on session + 10 * total_timeout (over sock timeouts) - PR #535

Stop using include_package_data option in setup.cfg to get config files included in more installs - PR #519

Source code(tar.gz)
Source code(zip)
4.0.3(May 7, 2020)
Change aiohttp-xmlrpc to use Master.session to ensure config shared - PR #506 - Thanks @alebourdoulous for reporting

e.g. Maintin trust of proxy server environment variables

Source code(tar.gz)
Source code(zip)
4.0.2(Apr 26, 2020)
Raise for error HTML response on all aiohttp session requests - PR #494 / #496 - Thanks @windtail

Pass str to shutil.move due to Python bug - PR #497 - Thanks @SanketDG

Some more type hints added to verify.py - PR #488 - Thanks @SanketDG

Ignore atime on stat in test test_package_sync_does_not_touch_existing_local_file comparision as it casues stat compare fail on a slower run - PR #487 - Thanks @SanketDG

Source code(tar.gz)
Source code(zip)
4.0.1(Apr 5, 2020)
Pass correct aiohttp timeout objects - PR #478

Replace pkg_resources with importlib.resources - PR #479 - Thanks @SanketDG

Source code(tar.gz)
Source code(zip)
4.0.0(Mar 29, 2020)
Replace requests with aiohttp - PR #440

Replace xmlrpc2 with aiohttp-xmlrpc - PR #404

Only store PEP503 Normalized Simple API directories - PR #465 + #455

Flag errors when KeyboardInterrupt raised during sync - PR #421

Finish Windows Support + Add CI - PRs #469 + #471 - Thanks @FaustinCarter

Autobuild Docker images with master - PR #88 - Thanks @abitrolly

Only print conf deprecations if found in config - PR #327

Add PyPI metadata and Python version plugin filters - PR #391 - Thanks @TemptorSent

Add in GitHub Actions CI for Linux (Ubuntu), MacOSX + Windows

Source code(tar.gz)
Source code(zip)
3.6.0(Sep 24, 2019)

Please refer to CHANGES.md for new fixes + features and README.md. Official Docs: https://bandersnatch.readthedocs.io/
Source code(tar.gz)
Source code(zip)
3.5.0(Sep 14, 2019)

Move version to 3.5.0 + Update CHANGES.md
Source code(tar.gz)
Source code(zip)
3.4.0(May 30, 2019)

Move version to 3.4.0 + Update CHANGES.md
Source code(tar.gz)
Source code(zip)
3.3.0(Apr 12, 2019)

Move version to 3.3.0 + Update CHANGES.md
Source code(tar.gz)
Source code(zip)
3.2.0(Jan 25, 2019)

Move version to 3.2.0 + Update CHANGES.md
Source code(tar.gz)
Source code(zip)
3.1.3(Dec 26, 2018)

Please refer to CHANGES.md for new fixes + features.
Source code(tar.gz)
Source code(zip)
3.1.2(Dec 3, 2018)

Please refer to CHANGES.md for new fixes + features.
Source code(tar.gz)
Source code(zip)
3.1.1(Nov 26, 2018)

Please refer to CHANGES.md for new fixes + features.
Source code(tar.gz)
Source code(zip)
3.1.0(Nov 26, 2018)

Please refer to CHANGES.md for new fixes + features.
Source code(tar.gz)
Source code(zip)
3.0.1(Oct 30, 2018)

V3 major release of bandersnatch
Source code(tar.gz)
Source code(zip)

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

Related tags

Overview

Installation

Docker

pip

Quickstart

Webserver

Cron jobs

Maintenance

Other Commands

Operational notes

Case-sensitive filesystem needed

Windows requires elevated prompt

Many sub-directories needed

Client Compatibility

Bandersnatch Mission

Contact

Code of Conduct

Kudos

Comments

Description

Log output

Environment configuration

v6.1.1

v6.1.0

v6.0.1

v6.0.0

v6.0.0b2

v6.0.0b1

Release 6.1.1 (released Jan 05, 2023)

Bugs fixed

Release 6.1.0 (released Jan 05, 2023)

Dependencies

Incompatible changes

Deprecated

Releases(6.1.0)

6.1.0(Jan 3, 2023)

New Features

Dropped Support

CI

6.0.1(Oct 24, 2022)

Bug Fixes

New Features

6.0.0(Sep 1, 2022)

5.3.0(Jul 29, 2022)

New Features

Other

5.2.0(May 2, 2022)

5.2.0 (2022-05-02)

New Features

Other

5.1.1(Dec 15, 2021)

Bug fixes

Other

5.1.0(Nov 3, 2021)

New Features

Bug fixes

Documentation

Dependencies

5.0.0(Apr 28, 2021)

New Features

Deprecations

Bug Fixes

4.4.0(Dec 31, 2020)

New Features

Bug Fixes

4.3.0(Aug 25, 2020)

New Features

Bug Fixes

4.2.0(Aug 21, 2020)

New Features

Internal API Changes

Bug Fixes

4.1.1(Aug 12, 2020)

4.1.0(Aug 10, 2020)

New Features

Internal API Changes

Bug Fixes

4.0.3(May 7, 2020)