The Python Package Index

Is it possible to get an update on the development roadmap about when TUF or other encryption support might be deployed? Thanks!

Also, it appears that tomorrow will be 6 years since January 5, 2013.

A scary number of people embed their PyPI username and password in their Travis config (using Travis encrypted variables), to enable automatic releases for certain branches (Travis even has a guide for it).

In addition, the packaging docs example encourages users to save their password in plaintext on disk in their .pypirc (they can of course use twine's password prompting, but I wonder how many read that far, rather than just copy the example verbatim?)

Whilst in an ideal world credentials of any form wouldn't be saved unencrypted to disk (or given to a third-party such as Travis) and instead users prompted every time - I don't think this is realistic in practice.

API keys would offer the following advantages:

1. Higher-entropy credentials that are guaranteed to have not been reused on multiple sites.
2. The ability to give the API key a smaller permissions scope than that of the owner's username/password. For example an API key would not be permitted to change a user's listed GPG key or in the future, their 2FA settings. Or an API key could be limited to a specific package.
3. Since this would be separate from the existing username/password auth, a signing based approach (eg HMAC) could be used, without breaking older clients. This would ensure that if a connection was MiTMed (eg due to a protocol or client exploit), the API key itself would still remain secure.
4. Eventually support could be dropped for the password field in .pypirc, leaving a much safer choice between password prompting every time, or creating an API key that could be saved to disk.
5. If/when support is added for 2FA, users who need to automate PyPI uploads won't have to forgo 2FA for their whole account. They could instead choose to just create a 2FA-circumventing API key for just the one package that needs uploads in automation.

Many thanks :-)

(I've filed this against warehouse since I'm presuming this is beyond the scope of maintenance-only changes being made to the old PyPI codebase)

We have an internal mirror of https://pypi.python.org that stopped working as the load testing started: https://status.python.org/incidents/0gmdf90kkt8n

I am currently investigating and collecting logs to further report but would like to create an issue in case other people come across the same problem.

If this turns-out to be my internal problem (I hope so!), my apologies in advance. I will update this issue as I have more information.

In the old pypi.python.org, we can update the description by using edit or upload a pkg-info file. But now it is not support:

Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html)

But in pypi.org, I cannot find any function to add or change the description of the package : https://pypi.org/project/lightgbm/

I also tried to update it by using python setup.py register , but it failed: Server response (410): This API is no longer supported, instead simply upload the file.

So I tried to upload the pkg-file, but sill met the error: ValueError: Unknown distribution format: 'PKG-INFO'

probably due to the load testing happening right now, this happens for trying to install any package.

  Could not fetch URL https://pypi.python.org/simple/upgrade/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:645) - skipping
  Could not find a version that satisfies the requirement upgrade (from versions: )
No matching distribution found for upgrade

I've been working styling the information panel (issue #801).

One thing that I've noticed is that a number of package long descriptions are not coming through with the correct formatting:

The first is clearly markdown, the second, I am not sure, but I have noticed a number of other packages using the same syntax.

Obviously, it would be great to render these out to HTML so the new styles will work.

Describe the bug I have a python package on PyPi that just (as of the last ~4-5 days) now fails to upload, with a new error:

durr@rwpscrape /m/S/S/WebRequest> python3 setup.py sdist upload

[snip build process]

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/
Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

Ok, that's clear enough. My readme is in markdown, let's see how to fix this.

Going to https://packaging.python.org/tutorials/packaging-projects/#description, it states you can add long_description_content_type="text/markdown", to your setup.py file, and it'll be processed as markdown.

<does edits>....
<submits to PyPi again

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/

[snip build process]

Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

What?

Expected behavior I expect if I specify the readme is markdown, as the documentation says to do so, the readme should be treated as markdown.

To Reproduce Attempt to upload a package with a readme that is valid markdown, but not valid reStructuredText.

My Platform

• Python 3.5.2 x64
• Ubuntu 16.04 LTS
• setuptools-39.2.0 (I specifically checked that it's the most current version)
• readme_renderer 20.0

Actual project in question is here. The readme.md in question is here, and the setup.py is here

What's the problem this feature will solve? To finish #996 (see #5567), we need to test MFA with real users on real packages; asking them to spin up dev environments is too hard and won't help multi-maintainer projects reason well about what MFA policies they want to set up.

Describe the solution you'd like My tentative suggestion is:

• We first roll out MFA on test.pypi.org and publicize it to project maintainers for a designated 2-week testing period, and we allot extra time for dealing with support requests during that time
• We then roll it out to pypi.org and announce it on the PyPI announcement email list, PSF blog, etc.

Additional context

• If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?
• Are there particular categories of user we need to make sure we get in the beta test? Ideas that come to mind for me: Windows, mobile, users of particular TOTP implementations and U2F implementations, people on very slow connections, people who habitually block a lot of cookies/JS, maintainers who maintain 20+ projects, people with very old and weird PyPI accounts (e.g., we do not have verified email for them, their passwords do not adhere to current policy), users with upwards of 10 MFA methods they want to add to their accounts, multi-maintainer projects, multi-owner projects, organizations where users share an auth token within a group.
• Is test.pypi.org the right place for this? Should we spin up some other instance for this particular kind of testing, since maintainers do use Test PyPI for real uploads (of packages that need testing)?

Cc @ewdurbin .

Account compromise of the owner of a popular package on PyPI is something that would have pretty dire consequences. 2FA/MFA would be a big step to helping avoid this.

This would need to be implemented in both:

• Website login
• API used for package upload/register by setuptools/pip/...

The 1st of these is presumably the easiest, and I believe will still add some value even if the 2nd isn't implemented initially - since it will still prevent things like package owner/maintainer role changes or modification of a user's listed GPG key or login details (presuming these cannot be made via the API).

Harder is what to do with the API, given it requires client changes and is also used by some in a non-interactive manner in automation. I see a few options:

1. Add 2FA support to setuptools/pip, and just document that people uploading packages in automation will just need to disable 2FA for their whole account (or ideally create a separate account just for automation).
2. Add support for API keys to both Warehouse and clients (#994) and allow those to circumvent 2FA. For the people that really must upload packages non-interactively this would still protect against a few risks (credentials re-use, weak credentials), and the API keys could also be made package-specific or tied to a particular IP range. For everyone else, they would just not create an API key and still have full 2FA protection.
3. All of the above.

In terms of implementation, there are a number of packages that support TOTP (used by Google Authenticator): https://pypi.python.org/pypi/onetimepass/1.0.1 https://pypi.python.org/pypi/pyotp https://pypi.python.org/pypi/oath https://pypi.python.org/pypi/otpauth

(I've filed this against warehouse since I'm presuming this is beyond the scope of maintenance-only changes being made to the old PyPI codebase)

@webknjaz reported:

@brainwane I've tried that on Test PyPI.

So I have a TOTP set up. I clicked on Add 2FA with security key. It prompted me to enter a Key name which I did (Yubikey Neo).

STR

After that, clicking Provision key does nothing visually. So I've opened DevTools. I can see a successful GET request to https://test.pypi.org/manage/account/webauthn-provision/options with some JSON payload in the response. It looks legit, contains my user data and a challenge. After clicking more times on that button, each of them produces an exception being logged to the JS console. The same happenes on prod PyPI, in incognito mode, with browser extensions disabled.

Runtime

Google Chrome Version 69.0.3497.81 (Official Build) (64-bit) running Gentoo Linux

Trace

[8] bind-modal-keys.js:43 Uncaught (in promise) DOMException: A request is already pending.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445
[2] bind-modal-keys.js:43 Uncaught (in promise) DOMException: The operation either timed out or was not allowed. See: https://w3c.github.io/webauthn/#sec-assertion-privacy.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445

Originally posted by @webknjaz in https://github.com/pypa/warehouse/issues/5661#issuecomment-503486463

What's the problem this feature will solve?

GitHub has a "Token Scanning" feature, which enables them to detect API keys/tokens, and notify the service provider who issued the token. The service providers can then act on this information according to their own policies (revoke, contact owner etc).

This helps avoid situations where an active API key is available publicly on the project's repository. GitHub has a Partnership program for this, with various service providers (incl. npm and more).

What's the problem this feature will solve?

It would be worthwhile to explore partnering with GitHub for Token Scanning, when the support for API Keys has been completed.

Additional context

https://developer.github.com/partnerships/token-scanning/ contains details about how to contact GitHub for this partnership, and details on how the integration works.

Bumps pydantic from 1.9.2 to 1.10.3.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps types-redis from 4.3.21.6 to 4.3.21.7.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps pyrsistent from 0.19.2 to 0.19.3.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps platformdirs from 2.6.0 to 2.6.2.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps disposable-email-domains from 0.0.85 to 0.0.86.

You can trigger a rebase of this PR by commenting @dependabot rebase.