Python PyPi staging server and packaging, testing, release tool

Overview

devpi: PyPI server and packaging/testing/release tool

This repository contains three packages comprising the core devpi system on the server and client side:

  • devpi-server: for serving a pypi.org consistent caching index as well as user or team based indexes which can inherit packages from each other or from the pypi.org site.
  • devpi-web: plugin for devpi-server that provides a web and search interface
  • devpi-client: command line tool with sub commands for creating users, using indexes, uploading to and installing from indexes, as well as a "test" command for invoking tox.

For getting started, more docs see https://doc.devpi.net/

Holger Krekel, Florian Schulze, April 2017 (contact us at office at merlinux.eu for support contracts and paid help)

Issues
  • Add option to devpi-server to force full pypi index refresh

    Add option to devpi-server to force full pypi index refresh

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/59
    • Originally reported by: @thomasf
    • Originally created at: 2013-10-26T19:06:39.607

    For reference: My organisations devpi server somehow failed to keep in sync with the central pypi server, the latest version of django was not available even though days had passed since its release.
    I asked about forcing an full index refresh on the devpi irc channel and hpk procided an manual solution of deleting all mirrored packages which worked just fine:

    1. stop server
    2. rm -rf ~/.devpi/server/root/pypi
    3. restart server
    opened by devpi-bot 38
  • Move to GitHub, need user names

    Move to GitHub, need user names

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/389
    • Originally reported by: @fschulze
    • Originally created at: 2017-02-23T16:17:19.055

    We are preparing the move from Bitbucket to GitHub. For that we need to have a user name mapping, so notifications on ticket changes still reach everyone involved. We mapped most of the users, but the following users we either couldn't find on GitHub, or we aren't sure it's the correct user.

    First the names that appear in open tickets and would most likely want a proper mapping to get notifications in the future:

    @adrpp @aostr123 @carlososiel @cdunklau @fabian_snovna @glurk @hoefling @hoel @hoganman @jack_murphy @jblackburnsx @jcastets @jeroen1205 @lbrack @makefu @mandshaw @notsqrt @pendletongp @sbadia @sietsebb @sradhak @troyswanson @ub_marco @ValleChiara @wcooley @zealot0630 @zirpu

    And here all users that appear only in closed issues:

    @basti @beatpanic @benoit-dunandlaisin @bjacobowski @blaffoy_cfp @boosh @brouberol @bruno @carlio @chadawagner @davedash @diegor @eeaston @efren @esn89 @frainbird @gonvaled @grzegorz_dubicki @heyvaerm @href @hurie83 @igan @jacobgeiger @jdoherty1 @jgeessel @jholloway7 @jlyheden @josemanimala @josh_kuhn @jshridha @juliankrause @justinsr20 @kenfar @kgk @kieranbrownlees @kostialopuhin @kpatter @kyle_rockman @ludwigf @lvtstx @martinfiers @martinhoefling @matt_jd @matthias87 @max_gasner @maximilien_riehl_ @mcoupland @metalseargolid @mhuber @ModZero @nolar @onlynone @pasha_savchenko @paullogston @plinioga @pquentin @private @prologic @riehlm @s0lst1c3 @shaenisch @shriyanshagnihotri @stefanocrosta @tcalmant @vacplac @vortec

    Please tell us your GitHub user name in this ticket if you care about notifications.

    I hope notifications to everyone are properly sent for this ticket.

    opened by devpi-bot 36
  • devpi client installations fail to install modules - same installation with pip works fine

    devpi client installations fail to install modules - same installation with pip works fine

    Hi.

    Environment

    • MacOS Sierra, High Sierra - latest versions
    • Python 3.6.3 and 3.6.4 from Homebrew - latest versions
    • Virtual environment - no "raw" devpi client install on the Homebrew Python
    • Six projects, 3 different programmers affected
    • devpi-server 4.3.2 on a dedicated machine on AWS (ports 80, 443); OR
    • Dockerized devpi-server 4.3.2 on localhost (port 3141)
    • No plug-ins
    • devpi-client 4.0.0 with devpi-common 3.2.0 run from Homebrew Python (/usr/local/bin) and/or in the virtual environment work space - same effect

    devpi vs pip - general use, public PyPI package

    devpi + pytest

    (bergen) [0 :) [email protected] bergen $] devpi install pytest
    -->  /Users/ciurana/meltwater/bergen$ bin/pip install -U -i http://$SERVER_NAME/blah/dev/+simple/ pytest  [PIP_PRE=1,PIP_USE_WHEEL=1]
    Collecting pytest
      Downloading http://$SERVER_NAME/root/pypi/+f/5f7/a1be1015c59c6/pytest-3.3.2-py2.py3-none-any.whl (185kB)
        100% | 194kB 5.0MB/s
    Requirement already up-to-date: pluggy<0.7,>=0.5 in ./lib/python3.6/site-packages (from pytest)
    Requirement already up-to-date: py>=1.5.0 in ./lib/python3.6/site-packages (from pytest)
    Requirement already up-to-date: attrs>=17.2.0 in ./lib/python3.6/site-packages (from pytest)
    Requirement already up-to-date: setuptools in ./lib/python3.6/site-packages (from pytest)
    Requirement already up-to-date: six>=1.10.0 in ./lib/python3.6/site-packages (from pytest)
    Installing collected packages: pytest
    Successfully installed pytest-3.3.2
    
    (bergen) [0 :) [email protected] bergen $] pip install -e . && bin/pytest -v ./test
    Obtaining file:///Users/ciurana/meltwater/bergen
    Requirement already satisfied: amqp in ./lib/python3.6/site-packages (from fhai-avtale==1.8.dev2)
    .
    .
    <snip - lots of the same>
    .
    .
    Installing collected packages: fhai-avtale
      Running setup.py develop for fhai-avtale
    Successfully installed fhai-avtale
    Traceback (most recent call last):
      File "bin/pytest", line 7, in <module>
        from pytest import main
    ModuleNotFoundError: No module named 'pytest'
    

    pip install + pytest

    (bergen) [0 :) [email protected] bergen $] pip install pytest
    Collecting pytest
      Using cached pytest-3.3.2-py2.py3-none-any.whl
    Requirement already satisfied: pluggy<0.7,>=0.5 in ./lib/python3.6/site-packages (from pytest)
    Requirement already satisfied: py>=1.5.0 in ./lib/python3.6/site-packages (from pytest)
    Requirement already satisfied: attrs>=17.2.0 in ./lib/python3.6/site-packages (from pytest)
    Requirement already satisfied: setuptools in ./lib/python3.6/site-packages (from pytest)
    Requirement already satisfied: six>=1.10.0 in ./lib/python3.6/site-packages (from pytest)
    Installing collected packages: pytest
    Successfully installed pytest-3.3.2
    
    (bergen) [0 :) [email protected] bergen $] pip install -e . && bin/pytest -v ./test
    Obtaining file:///Users/ciurana/meltwater/bergen
    Requirement already satisfied: amqp in ./lib/python3.6/site-packages (from fhai-avtale==1.8.dev2)
    <snip>
    ===== test session starts ======
    platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/ciurana/meltwater/bergen/bin/python3.6
    cachedir: .cache
    rootdir: /Users/ciurana/meltwater/bergen, inifile:
    collecting 21 items
    <snip>
    ===== 109 passed in 13.82 seconds =====
    

    devpi against private packages in devpi private repository

    For what it's worth: fhai is a namespace package.

    (doc-categorizer) [0 :) [email protected] doc-categorizer $] devpi install fhai-doccat
    Collecting fhai-doccat
      Downloading http://$SERVER_NAME/fairhair/dev/+f/1e6/491a25f1f66ab/fhai_doccat-1.0.dev0-py2.py3-none-any.whl
    <snip>
    Installing collected packages: fhai-doccat
    Successfully installed fhai-doccat-1.0.dev0
    
    # From ./bin/dokkat 
    (doc-categorizer) [0 :) [email protected] doc-categorizer $] dokkat 
    Traceback (most recent call last):
      File "/Users/ciurana/meltwater/doc-categorizer/bin/dokkat", line 7, in <module>
        from fhai.dockat.server import run
    ModuleNotFoundError: No module named 'fhai'
    

    But if we try this...

    (doc-categorizer) [146 :( [email protected] doc-categorizer $] python
    Python 3.6.4 (default, Jan  6 2018, 11:51:15) 
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from fhai.dockat.server import run
    >>> dir(run)
    ['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
    

    The package appears to be available and resolves fine, namespace package and all. There are other variations of these that we can discuss on IRC, etc.

    Thanks in advance 👽🙏🏻

    opened by ciurana 30
  • Unable to invoke 'devpi upload'

    Unable to invoke 'devpi upload'

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/367
    • Originally reported by: @hoefling
    • Originally created at: 2016-08-17T15:03:08.017

    Based on this SO question of mine. Getting the same error no matter what upload parameters I specify (tried different formats, with-docs, only-docs):

    
    (tstenv) [[email protected] workspace]# devpi upload --formats sdist
    detected devpi:upload section in /var/lib/jenkins/jobs/myproject-deploy-release/workspace/setup.cfg
    using workdir /tmp/devpi29
    pre-build: cleaning /var/lib/jenkins/jobs/myproject-deploy-release/workspace/dist
    -->  /var/lib/jenkins/jobs/myproject-deploy-release/workspace$ tstenv/bin/python setup.py sdist 
    warning: sdist: standard file not found: should have one of README, README.rst, README.txt
    
    built: /var/lib/jenkins/jobs/myproject-deploy-release/workspace/dist/myproject-1.0.0.dev0.tar.gz [SDIST] 2642.98kb
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help
    
    error: no commands supplied
    Traceback (most recent call last):
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/bin/devpi", line 11, in <module>
        sys.exit(main())
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/lib/python3.5/site-packages/devpi/main.py", line 30, in main
        return method(hub, hub.args)
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/lib/python3.5/site-packages/devpi/upload.py", line 48, in main
        name_version = exported.setup_name_and_version()
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/lib/python3.5/site-packages/devpi/upload.py", line 311, in setup_name_and_version
        report=False).splitlines()[-1].strip()
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/lib/python3.5/site-packages/devpi/main.py", line 214, in popen_output
        return check_output(args, cwd=str(cwd))
      File "/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/lib/python3.5/site-packages/devpi_common/proc.py", line 18, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '['/var/lib/jenkins/jobs/myproject-deploy-release/workspace/tstenv/bin/python', '/var/lib/jenkins/jobs/myproject-deploy-release/workspace/setup.py', '--name']' returned non-zero exit status 1
    

    Running

    #!python
    python setup.py sdist upload -r reponame
    

    succeeds with response 200, I can confirm the dist file is being uploaded on correct index.

    Running

    #!python
    python setup.py build_sphinx upload_docs --upload-dir path/to/html -r reponame
    

    fails with

    #!python
    ... # sphinx output
    build succeeded, 25 warnings.
    running upload_docs
    Submitting documentation to http://localhost:4050/USER/INDEX
    Upload failed (400): :action field not found
    
    opened by devpi-bot 28
  • how to change default mirror url pypi.python.org/simple/ with devpi-server

    how to change default mirror url pypi.python.org/simple/ with devpi-server

    i start devpi-server version 4.3.0 as cache server with cmd devpi-server --serverdir /root/packages/ --port 5555 --host=0.0.0.0 --start how can i change the default cache mirror?

    opened by robertding 27
  • Use

    Use "X-Devpi-Auth" header instead of overriding Authorize header

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/75
    • Originally reported by: @hpk42
    • Originally created at: 2013-11-23T21:33:18.458

    devpi / devpi-server should not use the "Authorize" header except for the actual login operation. This allows to use https://user:[email protected] urls and handling authentication at the "fronting" web server without interferences from devpi-client.

    opened by devpi-bot 24
  • Enable inheriting from external index (other than pypi.python.org)

    Enable inheriting from external index (other than pypi.python.org)

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/12
    • Originally reported by: @vmalloc
    • Originally created at: 2013-07-08T16:05:17.826

    Assuming one have a pypi clone running in-house, he may want to use that URL as the root of his or her setup, and not the global pypi.

    If this is already supported, please describe how it can be achieved...

    opened by devpi-bot 24
  • "devpi-server --status" does not work anymore due to waitpid() in isrunning()

    The os.waitpid() call in isrunning() (server/devpi_server/vendor/xprocess.py) causes "devpi-server --status" to not work anymore, because waitpid() requires, that the waited-for process is a child process of the current process, which is not the case, so that a ChildProcessError (Python 3, subclass of OSError) or OSError (Python 2) is raised and isrunning() returns false, although the devpi-server process is running.

    wontfix 
    opened by devpiuser 22
  • Import error when importing with devpi-server 2.2.1

    Import error when importing with devpi-server 2.2.1

    • Bitbucket: https://bitbucket.org/hpk42/devpi/issue/246
    • Originally reported by: @StephanErb
    • Originally created at: 2015-06-25T17:55:19.997

    We exported state from devpi-server 2.1.4 and are trying to import using devpi-server 2.2.1. The import fails with the following exception:

    2015-06-25 14:09:11,369 INFO  [Wtx14194] set_metadata mypackage-0.2.0.post0.dev26+g54601f7
    Traceback (most recent call last):
      File "/opt/devpi-2.2/master/venv/bin/devpi-server", line 11, in <module>
        sys.exit(main())
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/main.py", line 53, in main
        return _main(pluginmanager, argv=argv)
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/main.py", line 113, in _main
        return xom.main()
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/main.py", line 200, in main
        return do_import(args.import_, xom)
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/importexport.py", line 41, in do_import
        importer.import_all(path)
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/importexport.py", line 276, in import_all
        self.import_filedesc(stage, filedesc)
      File "/opt/devpi-2.2/master/venv/local/lib/python2.7/site-packages/devpi_server/importexport.py", line 308, in import_filedesc
        assert digest == hash_value
    AssertionError
    
    opened by devpi-bot 21
  • Fix #853: prevent mirror index duplicates in sro method.

    Fix #853: prevent mirror index duplicates in sro method.

    null

    bug devpi-server 
    opened by fschulze 0
  • Avoid attribute error when route is None

    Avoid attribute error when route is None

    I've noticed the following exception in our logs. I'm not familiar with pyramid and have no idea how to add a test for this. If you can point me in the right direction, I'm happy to add one. :)

    2021-10-06 15:25:04,653 ERROR Exception while serving /+authcheck
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/pyramid/tweens.py", line 13, in _error_handler
        response = request.invoke_exception_view(exc_info)
      File "/usr/local/lib/python3.9/site-packages/pyramid/view.py", line 786, in invoke_exception_view
        raise HTTPNotFound
    pyramid.httpexceptions.HTTPNotFound: The resource could not be found.
    ​
    During handling of the above exception, another exception occurred:
    ​
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/waitress/channel.py", line 397, in service
        task.service()
      File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 168, in service
        self.execute()
      File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 434, in execute
        app_iter = self.channel.server.application(environ, start_response)
      File "/usr/local/lib/python3.9/site-packages/devpi_server/views.py", line 166, in __call__
        return self.app(environ, start_response)
      File "/usr/local/lib/python3.9/site-packages/pyramid/router.py", line 270, in __call__
        response = self.execution_policy(environ, self)
      File "/usr/local/lib/python3.9/site-packages/pyramid/router.py", line 276, in default_execution_policy
        return router.invoke_request(request)
      File "/usr/local/lib/python3.9/site-packages/pyramid/router.py", line 245, in invoke_request
        response = handle_request(request)
      File "/usr/local/lib/python3.9/site-packages/devpi_server/views.py", line 190, in request_log_handler
        response = handler(request)
      File "/usr/local/lib/python3.9/site-packages/devpi_server/views.py", line 226, in request_tx_handler
        response = handler(request)
      File "/usr/local/lib/python3.9/site-packages/pyramid/tweens.py", line 43, in excview_tween
        response = _error_handler(request, exc)
      File "/usr/local/lib/python3.9/site-packages/pyramid/tweens.py", line 17, in _error_handler
        reraise(*exc_info)
      File "/usr/local/lib/python3.9/site-packages/pyramid/util.py", line 733, in reraise
        raise value
      File "/usr/local/lib/python3.9/site-packages/pyramid/tweens.py", line 41, in excview_tween
        response = handler(request)
      File "/usr/local/lib/python3.9/site-packages/pyramid/router.py", line 143, in handle_request
        response = _call_view(
      File "/usr/local/lib/python3.9/site-packages/pyramid/view.py", line 674, in _call_view
        response = view_callable(context, request)
      File "/usr/local/lib/python3.9/site-packages/pyramid/viewderivers.py", line 392, in viewresult_to_response
        result = view(context, request)
      File "/usr/local/lib/python3.9/site-packages/pyramid/viewderivers.py", line 113, in _class_requestonly_view
        response = getattr(inst, attr)()
      File "/usr/local/lib/python3.9/site-packages/devpi_server/views.py", line 490, in authcheck_view
        root_factory = orig_request.matched_route.factory or root_factory
    AttributeError: 'NoneType' object has no attribute 'factory'
    
    opened by michael-k 1
  • Removing devpi plug-ins might result in a devpi client's 400 Bad Request (indexconfig got unexpected keyword arguments)

    Removing devpi plug-ins might result in a devpi client's 400 Bad Request (indexconfig got unexpected keyword arguments)

    Hi,

    in the past I've created a devpi plugin which comes with an "index config default": https://github.com/k-bu/devpi-rss/blob/70e218a76c71146d8414c71b08bc384f670ea0ee/devpi_rss/init.py#L47

    Now we've updated the devpi server version and simultaneously removed the plug-in, since we don't need it any longer. But after that we now get an error when trying to modify an index via devpi-client. Thus I don't directly try to touch the obsolete config value, but the client API seems to have a problem with configs of plug-ins which aren't installed any longer.

    >devpi index release/5.0 volatile=False /release/5.0 changing volatile: False PATCH https://our.server.com/release/5.0 400 Bad Request: indexconfig got unexpected keyword arguments: rss_active=True

    For now our solution is to modify an index config via "devpi getjson" + "devpi patchjson". Although that way we also wouldn't be able to remove the obsolete index config value to get rid of the error when using the devpi client.

    Is there any way to remove an old plug-in value from an index?

    Specs:

    • Debian GNU/Linux 11 (bullseye)
    • Python 3.8.12
    • appdirs 1.4.4 argon2-cffi 21.1.0 attrs 21.2.0 backports.entry-points-selectable 1.1.0 beautifulsoup4 4.10.0 bleach 4.1.0 build 0.7.0 certifi 2021.5.30 cffi 1.14.6 Chameleon 3.9.1 charset-normalizer 2.0.6 check-manifest 0.47 cmarkgfm 0.5.3 defusedxml 0.7.1 devpi-client 5.2.2 devpi-common 3.6.0 devpi-contact 1.0.21 devpi-ldap 2.0.0 devpi-server 6.2.0 devpi-web 4.0.8 distlib 0.3.3 docutils 0.17.1 execnet 1.9.0 filelock 3.1.0 hupper 1.10.3 idna 3.2 itsdangerous 2.0.1 lazy 1.4 ldap3 2.9.1 nose 1.3.7 packaging 21.0 passlib 1.7.4 PasteDeploy 2.1.1 pep517 0.11.0 pip 21.2.4 pkginfo 1.7.1 plaster 1.0 plaster-pastedeploy 0.7 platformdirs 2.4.0 pluggy 0.13.1 py 1.10.0 py-spy 0.3.10 pyasn1 0.4.8 pycparser 2.20 Pygments 2.10.0 pyparsing 2.4.7 pyramid 2.0 pyramid-chameleon 0.3 python-dateutil 2.8.2 PyYAML 5.4.1 readme-renderer 29.0 repoze.lru 0.7 requests 2.26.0 ruamel.yaml 0.17.16 ruamel.yaml.clib 0.2.6 setuptools 57.5.0 six 1.16.0 soupsieve 2.2.1 strictyaml 1.4.4 toml 0.10.2 tomli 1.2.1 tox 3.24.4 translationstring 1.4 urllib3 1.26.7 venusian 3.0.0 virtualenv 20.8.1 waitress 2.0.0 webencodings 0.5.1 WebOb 1.8.7 wheel 0.37.0 Whoosh 2.7.4 zope.deprecation 4.4.0 zope.interface 5.4.0
    documentation 
    opened by k-bu 3
  • Complex index inheritance results in redundant pypi.org requests

    Complex index inheritance results in redundant pypi.org requests

    Hi,

    we're using an index inheritance structure like this:

    • "6.0-dev" (inherited bases: 6.0, 5.0-dev)
    • "6.0" (inherited bases: 5.0)
    • "5.0-dev" (inherited bases: 5.0, 4.0-dev)
    • "5.0" (inherited bases: 4.0)
    • "4.0-dev" (inherited bases: 4.0, 3.0-dev)
    • "4.0" (inherited bases: 3.0)
    • [...]

    And as our structure slowly grows from time to time, we noticed that we get (pip's) package download timeouts (more than 15 seconds ) more and more frequently. Looking into the Nginx logs we noticed that the devpi server fires the same pypi.org request URL (for checking the package versions in PyPI?) N times, depending on the complexness of the inheritance. Thus the request is found e.g. 6 times in the log when accessing something from "6.0-dev", 5 times on "5.0-dev" and so on. As every request to PyPI takes quite a while we started reaching the default (pip) timeout of 15 seconds. The solution was to flatten the structure like this, so we finally have only 2 redundant PyPI requests, no matter which index gets accessed:

    • "6.0-dev" (inherited bases: 6.0)
    • "6.0" (inherited bases: 5.0)
    • "5.0-dev" (inherited bases: 5.0)
    • "5.0" (inherited bases: 4.0)
    • "4.0-dev" (inherited bases: 4.0)
    • "4.0" (inherited bases: 3.0)
    • [...]

    Specs:

    • Debian GNU/Linux 11 (bullseye)
    • Python 3.8.12
    • appdirs 1.4.4 argon2-cffi 21.1.0 attrs 21.2.0 backports.entry-points-selectable 1.1.0 beautifulsoup4 4.10.0 bleach 4.1.0 build 0.7.0 certifi 2021.5.30 cffi 1.14.6 Chameleon 3.9.1 charset-normalizer 2.0.6 check-manifest 0.47 cmarkgfm 0.5.3 defusedxml 0.7.1 devpi-client 5.2.2 devpi-common 3.6.0 devpi-contact 1.0.21 devpi-ldap 2.0.0 devpi-server 6.2.0 devpi-web 4.0.8 distlib 0.3.3 docutils 0.17.1 execnet 1.9.0 filelock 3.1.0 hupper 1.10.3 idna 3.2 itsdangerous 2.0.1 lazy 1.4 ldap3 2.9.1 nose 1.3.7 packaging 21.0 passlib 1.7.4 PasteDeploy 2.1.1 pep517 0.11.0 pip 21.2.4 pkginfo 1.7.1 plaster 1.0 plaster-pastedeploy 0.7 platformdirs 2.4.0 pluggy 0.13.1 py 1.10.0 py-spy 0.3.10 pyasn1 0.4.8 pycparser 2.20 Pygments 2.10.0 pyparsing 2.4.7 pyramid 2.0 pyramid-chameleon 0.3 python-dateutil 2.8.2 PyYAML 5.4.1 readme-renderer 29.0 repoze.lru 0.7 requests 2.26.0 ruamel.yaml 0.17.16 ruamel.yaml.clib 0.2.6 setuptools 57.5.0 six 1.16.0 soupsieve 2.2.1 strictyaml 1.4.4 toml 0.10.2 tomli 1.2.1 tox 3.24.4 translationstring 1.4 urllib3 1.26.7 venusian 3.0.0 virtualenv 20.8.1 waitress 2.0.0 webencodings 0.5.1 WebOb 1.8.7 wheel 0.37.0 Whoosh 2.7.4 zope.deprecation 4.4.0 zope.interface 5.4.0
    opened by k-bu 7
  • Devpi couldn't use existing directory if switched from master to replica

    Devpi couldn't use existing directory if switched from master to replica

    My devpi version: devpi (2.2.0) devpi-client (5.2.2) devpi-common (3.6.0) devpi-server (6.2.0) devpi-web (4.0.8)

    Hi, I want to switch a replica to master, but get error like this:

    fatal: The path 'dir/pip-repo/pypi' already contains devpi-server data.

    The command I use: devpi-init --role replica --master-url http://xxx:xxx --serverdir dir/pip-repo/pypi --no-root-pypi --root-passwd [email protected]!

    Anyone know how to deal with it?

    opened by orangetreee 7
  • Missing documentation

    Missing documentation

    image

    Can we please have this?

    opened by con-f-use 0
  • Release devpi client to use latest pluggy

    Release devpi client to use latest pluggy

    Please release a new version of devpi-client to allow pluggy >= 1.0.

    The current version of devpi-client is 5.2.2 which still installs pluggy < 1.0 => Collecting pluggy<1.0,>=0.6.0

    This leads later to an error with pip's dependency resolver if the pkg already uses the current version of pluggy 1.0.0.

    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    myFancyProject 0.11.5 requires pluggy==1.0.0, but you have pluggy 0.13.1 which is incompatible.
    
    opened by zerocewl 0
  • No mention of systemd config devpi.service in docs

    No mention of systemd config devpi.service in docs

    devpi-gen-config generates a devpi.service file clearly meant for use with systemd, but the docs (1, 2) don't mention this at all, focusing mostly on configuration via supervisord or launchd The devpi.service file seems to be targeted at server rather than PC use (as it makes the entire system wait for network-online.target before considering the boot complete), but can be used as a user unit as well (see also #427) after removing the Requires, After, User and WantedBy lines (took me a while to figure each of those out). Perhaps this possibility should be mentioned in the docs and perhaps devpi-gen-config should generate another systemd service file for use as a user unit or add comments to the regular one suggesting which lines to remove for such use.

    devpi-server documentation 
    opened by smheidrich 1
  • Error 500 for multiple files of the documentation of a package

    Error 500 for multiple files of the documentation of a package

    After I replaced a package in a volatile index, I noticed that the documentation was no longer correctly serverd. A look into the logfiles showed a lot of exceptions:

    2021-08-22 00:06:54,185 INFO  [req12149] [Wtx716] set_metadata robdevsim-0.2.1
    2021-08-22 00:06:54,310 INFO  [req12149] [Wtx716] fswriter717: commited at 717
    2021-08-22 00:06:54,732 INFO  [NOTI] [Rtx717] Processed a total of 1 projects and queued 1
    2021-08-22 00:06:55,310 INFO  [IDX] Indexer queue size ~ 1
    2021-08-22 00:06:55,310 INFO  [NOTI] [Rtx717] Processed a total of 1 projects and queued 1
    2021-08-22 00:06:55,435 ERROR [NOTI] Unhandled exception in notification thread.
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\devpi_server\fileutil.py", line 14, in rename
        os.rename(source, dest)
    FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'D:\\Data\\devpi\\.event_serial-tmp' -> 'D:\\Data\\devpi\\.event_serial'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\devpi_server\keyfs.py", line 107, in thread_run
        self.tick()
      File "d:\python\lib\site-packages\devpi_server\keyfs.py", line 94, in tick
        self.write_event_serial(event_serial)
      File "d:\python\lib\site-packages\devpi_server\keyfs.py", line 82, in write_event_serial
        write_int_to_file(event_serial + 1, self.event_serial_path)
      File "d:\python\lib\site-packages\devpi_server\fileutil.py", line 114, in write_int_to_file
        rename(tmp_path, path)
      File "d:\python\lib\site-packages\devpi_server\fileutil.py", line 20, in rename
        os.remove(dest)
    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Data\\devpi\\.event_serial'
    2021-08-22 00:06:57,357 INFO  [NOTI] [Rtx717] Processed a total of 1 projects and queued 1
    2021-08-22 00:06:57,497 ERROR [NOTI] calling <bound method EventSubscribers.on_changed_file_entry of <devpi_server.model.EventSubscribers object at 0x0000025BCD1E3DD8>> failed, serial=717
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\py\_error.py", line 83, in checked_call
        cls = self._geterrnoclass(_winerrnomap[errno])
    KeyError: 41
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\devpi_web\doczip.py", line 40, in locked_unpack_path
        yield (hash_file, unpack_path)
      File "d:\python\lib\site-packages\devpi_web\doczip.py", line 165, in remove_docs
        directory.remove()
      File "d:\python\lib\site-packages\py\_path\local.py", line 217, in remove
        ignore_errors=ignore_errors)
      File "d:\python\lib\site-packages\py\_error.py", line 85, in checked_call
        raise value
      File "d:\python\lib\site-packages\py\_error.py", line 66, in checked_call
        return func(*args, **kwargs)
      File "d:\python\lib\shutil.py", line 494, in rmtree
        return _rmtree_unsafe(path, onerror)
      File "d:\python\lib\shutil.py", line 384, in _rmtree_unsafe
        _rmtree_unsafe(fullname, onerror)
      File "d:\python\lib\shutil.py", line 384, in _rmtree_unsafe
        _rmtree_unsafe(fullname, onerror)
      File "d:\python\lib\shutil.py", line 384, in _rmtree_unsafe
        _rmtree_unsafe(fullname, onerror)
      File "d:\python\lib\shutil.py", line 393, in _rmtree_unsafe
        onerror(os.rmdir, path, sys.exc_info())
      File "d:\python\lib\shutil.py", line 391, in _rmtree_unsafe
        os.rmdir(path)
    OSError: [WinError 145] The directory is not empty: 'D:\\Data\\devpi\\hamilton\\staging\\robdevsim\\0.2.1\\+doc\\_static\\css\\fonts'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\py\_error.py", line 66, in checked_call
        return func(*args, **kwargs)
    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Data\\devpi\\hamilton\\staging\\robdevsim\\0.2.1\\+doc.hash'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "d:\python\lib\site-packages\devpi_server\keyfs.py", line 206, in _execute_hooks
        sub(ev)
      File "d:\python\lib\site-packages\devpi_server\model.py", line 1700, in on_changed_file_entry
        relpath=ev.typedkey.relpath
      File "d:\python\lib\site-packages\pluggy\hooks.py", line 289, in __call__
        return self._hookexec(self, self.get_hookimpls(), kwargs)
      File "d:\python\lib\site-packages\pluggy\manager.py", line 68, in _hookexec
        return self._inner_hookexec(hook, methods, kwargs)
      File "d:\python\lib\site-packages\pluggy\manager.py", line 62, in <lambda>
        firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
      File "d:\python\lib\site-packages\pluggy\callers.py", line 208, in _multicall
        return outcome.get_result()
      File "d:\python\lib\site-packages\pluggy\callers.py", line 80, in get_result
        raise ex[1].with_traceback(ex[2])
      File "d:\python\lib\site-packages\pluggy\callers.py", line 187, in _multicall
        res = hook_impl.function(*args)
      File "d:\python\lib\site-packages\devpi_web\main.py", line 340, in devpiserver_on_remove_file
        remove_docs(stage, project, version)
      File "d:\python\lib\site-packages\devpi_web\doczip.py", line 165, in remove_docs
        directory.remove()
      File "d:\python\lib\contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "d:\python\lib\site-packages\devpi_web\doczip.py", line 47, in locked_unpack_path
        hash_path.remove()
      File "d:\python\lib\site-packages\py\_path\local.py", line 223, in remove
        py.error.checked_call(os.remove, self.strpath)
      File "d:\python\lib\site-packages\py\_error.py", line 86, in checked_call
        raise cls("%s%r" % (func.__name__, args))
    py.error.EBUSY: [Resource device]: remove('D:\\Data\\devpi\\hamilton\\staging\\robdevsim\\0.2.1\\+doc.hash',)
    2021-08-22 00:06:57,950 INFO  [IDX] Committing 18 new documents to search index.
    

    As you can see, the server is running on a windows system and it seems like there are some permission issues as on window a file can not be deleted while accessed.

    Versions: OS: Windows 10 Installed Packages:

    apipkg==1.5
    appdirs==1.4.3
    argon2-cffi==19.1.0
    attrs==20.3.0
    beautifulsoup4==4.7.1
    bleach==3.1.0
    certifi==2019.3.9
    cffi==1.12.2
    Chameleon==3.6
    chardet==3.0.4
    check-manifest==0.37
    cmarkgfm==0.4.2
    defusedxml==0.5.0
    devpi-client==5.2.2
    devpi-common==3.6.0
    devpi-server==6.2.0
    devpi-web==4.0.8
    docutils==0.14
    execnet==1.5.0
    filelock==3.0.10
    hupper==1.6.1
    idna==2.8
    itsdangerous==1.1.0
    lazy==1.4
    passlib==1.7.1
    PasteDeploy==2.0.1
    pkginfo==1.5.0.1
    plaster==1.0
    plaster-pastedeploy==0.6
    pluggy==0.9.0
    py==1.8.0
    pycparser==2.19
    Pygments==2.3.1
    pyramid==2.0
    pyramid-chameleon==0.3
    python-dateutil==2.8.0
    readme-renderer==24.0
    repoze.lru==0.7
    requests==2.21.0
    ruamel.yaml==0.15.89
    six==1.12.0
    soupsieve==1.8
    strictyaml==1.0.0
    toml==0.10.0
    tox==3.7.0
    translationstring==1.3
    urllib3==1.24.1
    venusian==1.2.0
    virtualenv==16.0.0
    waitress==1.2.1
    webencodings==0.5.1
    WebOb==1.8.5
    Whoosh==2.7.4
    zope.deprecation==4.4.0
    zope.interface==4.6.0
    
    opened by jrast 9
  • error ERROR [REP] 403 Forbidden: failed fetching http://localhost:8080/+changelog/0-?initial_fetch

    error ERROR [REP] 403 Forbidden: failed fetching http://localhost:8080/+changelog/0-?initial_fetch

    devinit --role replica --master-url http://localhost:8080 --serverdir replica devpi-server --master-url http://localhost:8080port 4000 --serverdir replica --host 0.0.0.0

    log error : 2021-07-30 17:15:24,118 ERROR [REP] 403 Forbidden: failed fetching http://localhost:8080/+changelog/0-?initial_fetch 2021-07-30 17:15:24,119 INFO [REP] fetching http://localhost:8080/+changelog/0?initial_fetch 2021-07-30 17:15:24,126 ERROR [REP] 403 Forbidden: failed fetching http://localhost:8080/+changelog/0?initial_fetch w2021-07-30 17:15:29,128 INFO [REP] fetching http://localhost:8080/+changelog/0-?initial_fetch 2021-07-30 17:15:29,138 ERROR [REP] 403 Forbidden: failed fetching http://localhost:8080/+changelog/0-?initial_fetch

    help

    opened by Felix0829 1
An installation and dependency system for Python

Pyflow Simple is better than complex - The Zen of Python Pyflow streamlines working with Python projects and files. It's an easy-to-use CLI app with a

David O'Connor 904 Oct 19, 2021
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 16.8k Oct 24, 2021
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 16.8k Oct 22, 2021
A Poetry plugin for dynamically extracting the package version.

Poetry Version Plugin A Poetry plugin for dynamically extracting the package version. It can read the version from a file __init__.py with: # __init__

Sebastián Ramírez 148 Oct 9, 2021
:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED Forks: https://github.com/orsinium/forks DepHell -- project management for Python. Why it is better than all other tools: Form

DepHell 1.7k Oct 22, 2021
pip-run - dynamic dependency loader for Python

pip-run provides on-demand temporary package installation for a single interpreter run. It replaces this series of commands (or their Windows equivale

Jason R. Coombs 61 Oct 2, 2021
The Python package installer

pip - The Python Package Installer pip is the package installer for Python. You can use pip to install packages from the Python Package Index and othe

Python Packaging Authority 7.5k Oct 23, 2021
A set of tools to keep your pinned Python dependencies fresh.

pip-tools = pip-compile + pip-sync A set of command line tools to help you keep your pip-based packages fresh, even when you've pinned them. You do pi

Jazzband 5.3k Oct 22, 2021
Install and Run Python Applications in Isolated Environments

pipx — Install and Run Python Applications in Isolated Environments Documentation: https://pipxproject.github.io/pipx/ Source Code: https://github.com

null 4.2k Oct 22, 2021
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/. bandersnatch >=4.0 supports Linux, MacOSX + Wind

Python Packaging Authority 263 Oct 17, 2021
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/. bandersnatch >=4.0 supports Linux, MacOSX + Wind

Python Packaging Authority 262 Oct 11, 2021
The Python Package Index

Warehouse Warehouse is the software that powers PyPI. See our development roadmap, documentation, and architectural overview. Getting Started You can

Python Packaging Authority 2.8k Oct 24, 2021
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 22.4k Oct 24, 2021
local pypi server (custom packages and auto-mirroring of pypi)

localshop A PyPI server which automatically proxies and mirrors PyPI packages based upon packages requested. It has support for multiple indexes and t

Michael van Tellingen 372 Aug 29, 2021
The Fast Cross-Platform Package Manager

The Fast Cross-Platform Package Manager part of mamba-org Package Manager mamba Package Server quetz Package Builder boa mamba Mamba is a reimplementa

Mamba 2.3k Oct 24, 2021
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.

Spack Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, a

Spack 2.3k Oct 15, 2021
A PDM plugin that packs your packages into a zipapp

pdm-packer A PDM plugin that packs your packages into a zipapp Requirements pdm-packer requires Python >=3.7 Installation If you have installed PDM wi

Frost Ming 10 Sep 23, 2021
pipreqs - Generate pip requirements.txt file based on imports of any project. Looking for maintainers to move this project forward.

pipreqs - Generate requirements.txt file for any project based on imports Installation pip install pipreqs Usage Usage: pipreqs [options] <path>

Vadim Kravcenko 3.7k Oct 22, 2021
Conan - The open-source C/C++ package manager

Conan Decentralized, open-source (MIT), C/C++ package manager. Homepage: https://conan.io/ Github: https://github.com/conan-io/conan Docs: https://doc

Conan.io 5.3k Oct 22, 2021