Hi there!
Looking to hopefully get some possible solutions/alternatives to the issues I'm having with my devpi setup.
I'm writing a CI system for running Python unit tests for multiple python github repos at once using pre-released versions of the software.
Say I change project A, which is used heavily by projects B and C. In the CI for A I want to run pytests for project B and C, and additionally run their tests for each interpreter version our clients use for those projects. In this scenarios I need to test 3 projects and say 3 interpreter versions each, which means I create 9 virtual environments and run pytest in them, and additionally each environment needs to include the overriden version of A. In some scenarios I want to make cross cutting changes to these repos, in which case the virtual environments would be multiple overrides of packages.
Previously before trying devpi, my strategy has been to create the virtual environments like normal, and then before running pytest in the virtual environment, to run say pip install --no-deps /path/to/A. This has been really awkward and when the code includes new dependencies, and making the decisions of which projects actually need what dependencies gets complicated when I'm changing multiple repos at once for cross cutting changes.
With devpi my current idea is to simplify the workflow by just uploading the sdist of changed repos to a devpi index and referencing that in my pip config before creating any virtual environments for the other projects. My devpi index has a has a base index which mirrors our internal pypi instance. The devpi server is to be very short lived, as I only want to keep it for the lifetime of the CI test run. In my setup I'm using Argo workflows which is similar to a docker-compose setup of sorts, where I create a devpi server daemon that is referenced in other container pip.conf's which actually create virtual environments and run the pytest commands.
My server setup looks like this:
devpi-init --no-root-pypi
devpi-server --threads 32 --host="${DEVPI_HOST}" --port="${DEVPI_PORT}"
I create the indices like this:
devpi use "http://${DEVPI_HOST}:${DEVPI_PORT}/root"
devpi login root --password=""
devpi index -c root/pypi type=mirror "mirror_url=$DEVPI_MIRROR" volatile=False
devpi index -c root/dev bases=root/pypi volatile=True
And the pip.conf looks like this:
[global]
index-url = http://${DEVPI_HOST}:${DEVPI_PORT}/root/dev/+simple/
trusted-host = ${DEVPI_HOST}
disable-pip-version-check = true
[search]
index = http://${DEVPI_HOST}:${DEVPI_PORT}/root/dev/
Everything seemed like it was going fine until the amount of virtual environments I wanted to test from increased to ~15. Then I started seeing lines in the devpi-server like this:
2022-10-28 00:09:28,000 WARNI Task queue depth is 61
2022-10-28 00:09:28,076 WARNI Task queue depth is 62
2022-10-28 00:09:28,537 WARNI Task queue depth is 63
2022-10-28 00:09:28,538 WARNI Task queue depth is 64
2022-10-28 00:09:28,549 WARNI Task queue depth is 65
2022-10-28 00:09:28,549 WARNI total open connections reached the connection limit, no longer accepting new connections
Which causes issues like this in the pip install (which doesn't happen when the traffic is lower)
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPConnection object at 0x7f4415318e80>, 'Connection to X.Y.Z.W timed out. (connect timeout=15)')': /root/dev/+simple/isort/
ERROR: Could not find a version that satisfies the requirement isort>=5 (from versions: none)
ERROR: No matching distribution found for isort>=5
I'm then also getting a lot of errors by probably stressing the heck out of the sqlite storage from too many request threads (See the bottom for full stack trace).
2022-10-28 03:55:57,391 ERROR Exception while serving /root/dev/+simple/pip/
Traceback (most recent call last):
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/tweens.py", line 13, in _error_handler
response = request.invoke_exception_view(exc_info)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/view.py", line 786, in invoke_exception_view
raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.
During handling of the above exception, another exception occurred:
...
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs_sqlite.py", line 315, in get_connection
sqlconn.execute("begin immediate")
sqlite3.OperationalError: database is locked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs_sqlite.py", line 91, in rollback
self._sqlconn.rollback()
sqlite3.ProgrammingError: Cannot operate on a closed database.
Interestingly I was getting errors very infrequently when using older versions of the library. I tried updating the libraries to see if that would help and am now I've started getting even more errors.
Also note that our PyPI server is massive. I see lines like Processed a total of +200k projects and queued +200k, and many many lines like Committing 2500 new documents to search index..
Questions:
- Can I limit the amount of caching devpi is doing of the mirrored base index?
All I really want is a way that can always pin my pip install to choosing the sdist of my per-released python package under test. I don't care about the local caching aspects of it all for the upstream logic, as after I create the few virtual environments I'm just going to throw the works of this away. I imagine a lot of the sqlite errors are from trying to cache the mirrored pypi index. Or is caching the pypi state of the mirrored index required to be able to do this replacement in the first place?
- Is there a way I can have the server do a better job of queuing incoming requests?
I'd like to be able to somehow throttle the requests rather than just refusing new connections. Its unclear to me how this decision is being made... I assume it's something with the underlying waitress library, but it seems there's only limited ways to configure it. I see a connection_limit argument in the serve method but as far as I can tell I have no way to configure it. Also maybe there's also something I could be configuring in my pip.conf to make me more tolerant to this?
- For this case, would you recommend me using the postgresql extension instead of sqlite? https://pypi.org/project/devpi-postgresql/
Imagine this could help with the db contention I'm getting writing to disk. Don't mind setting this up, but it makes it again a bit more complicated and the main goal of this whole thing was aiming to simplify the CI setup.
Other things I'm considering is creating a devpi server per every 5-10 virtual environments I want to test in the CI run, or by limiting the overall parallelism of how many virtual environments are being created at once.
Thanks so much!
Are you using the latest released version?
Yep
devpi-client==6.0.2
devpi-common==3.7.0
devpi-server==6.7.0
devpi-web==4.1.1
Previously I had these versions which gave less errors than what I have now:
devpi-client==5.2.3
devpi-common==3.6.0
devpi-server==6.5.1
devpi-web==4.0.8
Provide the output of pip list from the virtual environment you are using.
bash-4.3$ pip3.10 list | sort
--------------------------------- -------------------------
Chameleon 3.9.0
Package Version
PasteDeploy 3.0.1
Pygments 2.13.0
WebOb 1.8.7
Whoosh 2.7.4
aiohttp 3.8.3
aiosignal 1.2.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
async-timeout 4.0.2
attrs 22.1.0
backports.entry-points-selectable 1.1.0
beautifulsoup4 4.11.1
bleach 5.0.1
build 0.9.0
certifi 2022.9.24
cffi 1.15.1
charset-normalizer 2.1.1
check-manifest 0.48
cmarkgfm 2022.10.27
defusedxml 0.7.1
devpi-client 6.0.2
devpi-common 3.7.0
devpi-server 6.7.0
devpi-web 4.1.1
distlib 0.3.2
docutils 0.19
filelock 3.0.12
frozenlist 1.3.1
hupper 1.10.3
idna 3.4
itsdangerous 2.1.2
lazy 1.5
multidict 6.0.2
packaging 21.3
passlib 1.7.4
pep517 0.13.0
pip 22.2.2
pkginfo 1.8.3
plaster 1.0
plaster-pastedeploy 0.7
platformdirs 2.3.0
pluggy 1.0.0
py 1.11.0
pycparser 2.21
pyparsing 3.0.9
pyramid 2.0
pyramid-chameleon 0.3
python-dateutil 2.8.2
readme-renderer 37.2
repoze.lru 0.7
requests 2.28.1
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.7
setuptools 60.8.2
six 1.16.0
soupsieve 2.3.2.post1
strictyaml 1.6.2
tomli 2.0.1
translationstring 1.4
urllib3 1.26.12
venusian 3.0.0
virtualenv 20.7.2
waitress 2.1.2
webencodings 0.5.1
wheel 0.36.2
yarl 1.8.1
zope.deprecation 4.4.0
zope.interface 5.5.0
Provide the Python and operating system versions under which the issue occurs.
I'm on python3.10 and using RHEL 7.9 (running in a docker container).
If possible, provide a minimal example to reproduce the issue.
Can whip something up if needed. Seems to be caused when pip installing in parallel from a lot of projects.
...
db stack I was getting:
2022-10-28 03:55:57,391 ERROR Exception while serving /root/dev/+simple/pip/
Traceback (most recent call last):
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/tweens.py", line 13, in _error_handler
response = request.invoke_exception_view(exc_info)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/view.py", line 786, in invoke_exception_view
raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 389, in transaction
yield tx
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/views.py", line 226, in request_tx_handler
response = handler(request)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_web/views.py", line 106, in request_trailing_slash_redirect_handler
return handler(request)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/tweens.py", line 43, in excview_tween
response = _error_handler(request, exc)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/tweens.py", line 17, in _error_handler
reraise(*exc_info)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/util.py", line 733, in reraise
raise value
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/tweens.py", line 41, in excview_tween
response = handler(request)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/router.py", line 143, in handle_request
response = _call_view(
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/view.py", line 674, in _call_view
response = view_callable(context, request)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/viewderivers.py", line 392, in viewresult_to_response
result = view(context, request)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/viewderivers.py", line 113, in _class_requestonly_view
response = getattr(inst, attr)()
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/views.py", line 574, in simple_list_project
stage.get_simplelinks(
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/model.py", line 895, in get_simplelinks
for stage, res in self.op_sro_check_mirror_whitelist(
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/model.py", line 1065, in op_sro_check_mirror_whitelist
res = getattr(stage, opname)(**kw)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/mirror.py", line 736, in get_simplelinks_perstage
return self._update_simplelinks(project, info, links, newlinks)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/mirror.py", line 596, in _update_simplelinks
self.keyfs.restart_as_write_transaction()
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 362, in restart_as_write_transaction
tx.restart(write=True)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 691, in restart
newtx = self.__class__(self.keyfs, write=write)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 478, in __init__
self.conn
File "/path/to/my/install/lib/python3.10/site-packages/lazy/lazy.py", line 36, in __get__
inst.__dict__[name] = value = self.__func(inst)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 492, in conn
return self.keyfs.get_connection(
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 235, in get_connection
self._storage.get_connection(closing=False, write=write))
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs_sqlite.py", line 315, in get_connection
sqlconn.execute("begin immediate")
sqlite3.OperationalError: database is locked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/path/to/my/install/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
task.service()
File "/path/to/my/install/lib/python3.10/site-packages/waitress/task.py", line 168, in service
self.execute()
File "/path/to/my/install/lib/python3.10/site-packages/waitress/task.py", line 434, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/middleware.py", line 21, in __call__
return self.app(environ, start_response)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/router.py", line 270, in __call__
response = self.execution_policy(environ, self)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/router.py", line 276, in default_execution_policy
return router.invoke_request(request)
File "/path/to/my/install/lib/python3.10/site-packages/pyramid/router.py", line 245, in invoke_request
response = handle_request(request)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/views.py", line 190, in request_log_handler
response = handler(request)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/views.py", line 224, in request_tx_handler
with keyfs.transaction(write=write) as tx:
File "/opt/bb/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 391, in transaction
self.rollback_transaction_in_thread()
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 375, in rollback_transaction_in_thread
self._threadlocal.tx.rollback()
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs.py", line 679, in rollback
self.conn.rollback()
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs_sqlite_fs.py", line 73, in rollback
BaseConnection.rollback(self)
File "/path/to/my/install/lib/python3.10/site-packages/devpi_server/keyfs_sqlite.py", line 91, in rollback
self._sqlconn.rollback()
sqlite3.ProgrammingError: Cannot operate on a closed database.
23.1k Jan 1, 2023
57 Dec 30, 2022
345 Dec 28, 2022
12 Jan 6, 2023
5.8k Dec 31, 2022
1.2k Dec 23, 2022
13 Nov 22, 2022
6.5k Dec 29, 2022
1.7k Dec 30, 2022
79 Dec 14, 2022
10 Oct 21, 2021
21 Dec 29, 2022
4 Aug 28, 2022
5.1k Dec 30, 2022