Yet another URL library

aio-libs

Last update: Jan 3, 2023

Related tags

Overview

yarl

https://readthedocs.org/projects/yarl/badge/?version=latest

Introduction

Url is constructed from str:

>>> from yarl import URL
>>> url = URL('https://www.python.org/~guido?arg=1#frag')
>>> url
URL('https://www.python.org/~guido?arg=1#frag')

All url parts: scheme, user, password, host, port, path, query and fragment are accessible by properties:

>>> url.scheme
'https'
>>> url.host
'www.python.org'
>>> url.path
'/~guido'
>>> url.query_string
'arg=1'
>>> url.query

   
>>> url.fragment
'frag'

All url manipulations produce a new url object:

>>> url = URL('https://www.python.org')
>>> url / 'foo' / 'bar'
URL('https://www.python.org/foo/bar')
>>> url / 'foo' % {'bar': 'baz'}
URL('https://www.python.org/foo?bar=baz')

Strings passed to constructor and modification methods are automatically encoded giving canonical representation as result:

>>> url = URL('https://www.python.org/путь')
>>> url
URL('https://www.python.org/%D0%BF%D1%83%D1%82%D1%8C')

Regular properties are percent-decoded, use raw_ versions for getting encoded strings:

>>> url.path
'/путь'

>>> url.raw_path
'/%D0%BF%D1%83%D1%82%D1%8C'

Human readable representation of URL is available as .human_repr():

>>> url.human_repr()
'https://www.python.org/путь'

For full documentation please read https://yarl.readthedocs.org.

Installation

$ pip install yarl

The library is Python 3 only!

PyPI contains binary wheels for Linux, Windows and MacOS. If you want to install yarl on another operating system (like Alpine Linux, which is not manylinux-compliant because of the missing glibc and therefore, cannot be used with our wheels) the the tarball will be used to compile the library from the source code. It requires a C compiler and and Python headers installed.

To skip the compilation you must explicitly opt-in by setting the YARL_NO_EXTENSIONS environment variable to a non-empty value, e.g.:

$ YARL_NO_EXTENSIONS=1 pip install yarl

Please note that the pure-Python (uncompiled) version is much slower. However, PyPy always uses a pure-Python implementation, and, as such, it is unaffected by this variable.

Dependencies

YARL requires multidict library.

API documentation

The documentation is located at https://yarl.readthedocs.org

Why isn't boolean supported by the URL query API?

There is no standard for boolean representation of boolean values.

Some systems prefer true/false, others like yes/no, on/off, Y/N, 1/0, etc.

yarl cannot make an unambiguous decision on how to serialize bool values because it is specific to how the end-user's application is built and would be different for different apps. The library doesn't accept booleans in the API; a user should convert bools into strings using own preferred translation protocol.

Comparison with other URL libraries

furl (https://pypi.python.org/pypi/furl)

The library has rich functionality but the furl object is mutable.

I'm afraid to pass this object into foreign code: who knows if the code will modify my url in a terrible way while I just want to send URL with handy helpers for accessing URL properties.

furl has other non-obvious tricky things but the main objection is mutability.
URLObject (https://pypi.python.org/pypi/URLObject)

URLObject is immutable, that's pretty good.

Every URL change generates a new URL object.

But the library doesn't do any decode/encode transformations leaving the end user to cope with these gory details.

Source code

The project is hosted on GitHub

Please file an issue on the bug tracker if you have found a bug or have some suggestion in order to improve the library.

The library uses Azure Pipelines for Continuous Integration.

Discussion list

aio-libs google group: https://groups.google.com/forum/#!forum/aio-libs

Feel free to post your questions and ideas here.

Authors and License

The yarl package is written by Andrew Svetlov.

It's Apache 2 licensed and freely available.

Comments

File "yarl/_quoting.pyx", line 192, in yarl._quoting._Quoter.__init__

Facing this issue with any version of aiohttp starting from 3.4.0 and with recently released yarl 1.5.0. Using previous yarl==1.4.2 solves this issue (error is gone). Thought it was old bug bubbling out: https://github.com/aio-libs/aiohttp/issues/1364, but it seems that this issue is new.
bug pr-merged

opened by remort 41

`_quoting_c.c` is not compatible with Python 3.11.

Describe the bug

We'd like to test Django with Python 3.11 on Windows. Unfortunately yarl/_quoting_c.c is not compatible with Python 3.11 because longintrepr.h is missing:

 Failed to build numpy Pillow aiohttp frozenlist yarl
  "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\hostedtoolcache\windows\Python\3.11.0-alpha.6\x64\include -IC:\hostedtoolcache\windows\Python\3.11.0-alpha.6\x64\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" /Tcyarl/_quoting_c.c /Fobuild\temp.win-amd64-3.11\Release\yarl/_quoting_c.obj
  _quoting_c.c
  yarl/_quoting_c.c(198): fatal error C1083: Cannot open include file: 'longintrepr.h': No such file or directory
  error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

see logs.

Thanks for all your efforts :trophy:

To Reproduce

$ python3.11 -m venv .yarl-3.11
$ . .yarl-3.11/bin/activate
$ pip install yarl==1.7.2

Expected behavior

yarl installation works on Python 3.11.

Logs/tracebacks

Collecting yarl==1.7.2
  Downloading yarl-1.7.2.tar.gz (168 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.6/168.6 KB 1.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: idna>=2.0 in ./.yarl-3.11/lib/python3.11/site-packages (from yarl==1.7.2) (3.3)
Requirement already satisfied: multidict>=4.0 in ./.yarl-3.11/lib/python3.11/site-packages (from yarl==1.7.2) (6.0.2)
Building wheels for collected packages: yarl
  Building wheel for yarl (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for yarl (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [266 lines of output]
      **********************
      * Accelerated build *
      **********************
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.11
      creating build/lib.linux-x86_64-3.11/yarl
      copying yarl/_quoting.py -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/_quoting_py.py -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/_url.py -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/__init__.py -> build/lib.linux-x86_64-3.11/yarl
      running egg_info
      writing yarl.egg-info/PKG-INFO
      writing dependency_links to yarl.egg-info/dependency_links.txt
      writing requirements to yarl.egg-info/requires.txt
      writing top-level names to yarl.egg-info/top_level.txt
      reading manifest file 'yarl.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '*.cache' found anywhere in distribution
      warning: no previously-included files found matching 'yarl/*.html'
      warning: no previously-included files found matching 'yarl/*.so'
      warning: no previously-included files found matching 'yarl/*.pyd'
      no previously-included directories found matching 'docs/_build'
      adding license file 'LICENSE'
      writing manifest file 'yarl.egg-info/SOURCES.txt'
      copying yarl/__init__.pyi -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/_quoting_c.c -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/_quoting_c.pyi -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/_quoting_c.pyx -> build/lib.linux-x86_64-3.11/yarl
      copying yarl/py.typed -> build/lib.linux-x86_64-3.11/yarl
      running build_ext
      building 'yarl._quoting_c' extension
      creating build/temp.linux-x86_64-3.11
      creating build/temp.linux-x86_64-3.11/yarl
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/felixx/repo/yarl/.yarl-3.11/include -I/usr/local/include/python3.11 -c yarl/_quoting_c.c -o build/temp.linux-x86_64-3.11/yarl/_quoting_c.o
      yarl/_quoting_c.c: In function ‘__Pyx_InitCachedConstants’:
      yarl/_quoting_c.c:7808:259: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
       7808 |   __pyx_codeobj__15 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__14, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Quoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__15)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                                                                                                                                                                                                                                                   ^
            |                                                                                                                                                                                                                                                                   |
            |                                                                                                                                                                                                                                                                   int
      yarl/_quoting_c.c:320:72: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:7808:262: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
       7808 |   __pyx_codeobj__15 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__14, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Quoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__15)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                                                                                                                                                                                                                                                      ^~~~~~~~~~~~~~~~~
            |                                                                                                                                                                                                                                                                      |
            |                                                                                                                                                                                                                                                                      PyObject * {aka struct _object *}
      yarl/_quoting_c.c:320:79: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:320:11: error: too few arguments to function ‘PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      yarl/_quoting_c.c:7808:34: note: in expansion of macro ‘__Pyx_PyCode_New’
       7808 |   __pyx_codeobj__15 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__14, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Quoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__15)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                  ^~~~~~~~~~~~~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: declared here
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:7812:261: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
       7812 |   __pyx_codeobj__17 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__16, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Unquoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__17)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                                                                                                                                                                                                                                                     ^
            |                                                                                                                                                                                                                                                                     |
            |                                                                                                                                                                                                                                                                     int
      yarl/_quoting_c.c:320:72: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:7812:264: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
       7812 |   __pyx_codeobj__17 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__16, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Unquoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__17)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                                                                                                                                                                                                                                                        ^~~~~~~~~~~~~~~~~
            |                                                                                                                                                                                                                                                                        |
            |                                                                                                                                                                                                                                                                        PyObject * {aka struct _object *}
      yarl/_quoting_c.c:320:79: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:320:11: error: too few arguments to function ‘PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      yarl/_quoting_c.c:7812:34: note: in expansion of macro ‘__Pyx_PyCode_New’
       7812 |   __pyx_codeobj__17 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__16, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_stringsource, __pyx_n_s_pyx_unpickle__Unquoter, 1, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__17)) __PYX_ERR(1, 1, __pyx_L1_error)
            |                                  ^~~~~~~~~~~~~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: declared here
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c: In function ‘__pyx_pymod_exec__quoting_c’:
      yarl/_quoting_c.c:1088:43: error: invalid application of ‘sizeof’ to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
       1088 |     ((void)__Pyx_BUILD_ASSERT_EXPR(sizeof(PyFrameObject) == offsetof(PyFrameObject, f_localsplus) + Py_MEMBER_SIZE(PyFrameObject, f_localsplus)),\
            |                                           ^~~~~~~~~~~~~
      yarl/_quoting_c.c:1081:27: note: in definition of macro ‘__Pyx_BUILD_ASSERT_EXPR’
       1081 |     (sizeof(char [1 - 2*!(cond)]) - 1)
            |                           ^~~~
      yarl/_quoting_c.c:8079:3: note: in expansion of macro ‘__Pxy_PyFrame_Initialize_Offsets’
       8079 |   __Pxy_PyFrame_Initialize_Offsets();
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      yarl/_quoting_c.c:1088:61: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
       1088 |     ((void)__Pyx_BUILD_ASSERT_EXPR(sizeof(PyFrameObject) == offsetof(PyFrameObject, f_localsplus) + Py_MEMBER_SIZE(PyFrameObject, f_localsplus)),\
            |                                                             ^~~~~~~~
      yarl/_quoting_c.c:1081:27: note: in definition of macro ‘__Pyx_BUILD_ASSERT_EXPR’
       1081 |     (sizeof(char [1 - 2*!(cond)]) - 1)
            |                           ^~~~
      yarl/_quoting_c.c:8079:3: note: in expansion of macro ‘__Pxy_PyFrame_Initialize_Offsets’
       8079 |   __Pxy_PyFrame_Initialize_Offsets();
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/local/include/python3.11/pymacro.h:22:56: error: dereferencing pointer to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | #define Py_MEMBER_SIZE(type, member) sizeof(((type *)0)->member)
            |                                                        ^~
      yarl/_quoting_c.c:1081:27: note: in definition of macro ‘__Pyx_BUILD_ASSERT_EXPR’
       1081 |     (sizeof(char [1 - 2*!(cond)]) - 1)
            |                           ^~~~
      yarl/_quoting_c.c:1088:101: note: in expansion of macro ‘Py_MEMBER_SIZE’
       1088 |     ((void)__Pyx_BUILD_ASSERT_EXPR(sizeof(PyFrameObject) == offsetof(PyFrameObject, f_localsplus) + Py_MEMBER_SIZE(PyFrameObject, f_localsplus)),\
            |                                                                                                     ^~~~~~~~~~~~~~
      yarl/_quoting_c.c:8079:3: note: in expansion of macro ‘__Pxy_PyFrame_Initialize_Offsets’
       8079 |   __Pxy_PyFrame_Initialize_Offsets();
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      yarl/_quoting_c.c: In function ‘__Pyx_PyFunction_FastCallNoKw’:
      yarl/_quoting_c.c:9004:15: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
       9004 |     ++tstate->recursion_depth;
            |               ^~~~~~~~~~~~~~~
            |               recursion_limit
      yarl/_quoting_c.c:9006:15: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
       9006 |     --tstate->recursion_depth;
            |               ^~~~~~~~~~~~~~~
            |               recursion_limit
      yarl/_quoting_c.c: In function ‘__Pyx_PyFunction_FastCallDict’:
      yarl/_quoting_c.c:9033:60: error: ‘CO_NOFREE’ undeclared (first use in this function)
       9033 |             co->co_flags == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE)) {
            |                                                            ^~~~~~~~~
      yarl/_quoting_c.c:9033:60: note: each undeclared identifier is reported only once for each function it appears in
      yarl/_quoting_c.c: In function ‘__Pyx__GetException’:
      yarl/_quoting_c.c:9200:28: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9200 |         tmp_type = exc_info->exc_type;
            |                            ^~
      yarl/_quoting_c.c:9202:26: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9202 |         tmp_tb = exc_info->exc_traceback;
            |                          ^~
      yarl/_quoting_c.c:9203:17: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9203 |         exc_info->exc_type = local_type;
            |                 ^~
      yarl/_quoting_c.c:9205:17: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9205 |         exc_info->exc_traceback = local_tb;
            |                 ^~
      yarl/_quoting_c.c: In function ‘__Pyx__ExceptionSwap’:
      yarl/_quoting_c.c:9238:24: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9238 |     tmp_type = exc_info->exc_type;
            |                        ^~
      yarl/_quoting_c.c:9240:22: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9240 |     tmp_tb = exc_info->exc_traceback;
            |                      ^~
      yarl/_quoting_c.c:9241:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9241 |     exc_info->exc_type = *type;
            |             ^~
      yarl/_quoting_c.c:9243:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9243 |     exc_info->exc_traceback = *tb;
            |             ^~
      yarl/_quoting_c.c: In function ‘__Pyx_PyErr_GetTopmostException’:
      yarl/_quoting_c.c:9273:21: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9273 |     while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
            |                     ^~
      yarl/_quoting_c.c:9273:51: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9273 |     while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
            |                                                   ^~
      yarl/_quoting_c.c: In function ‘__Pyx__ExceptionSave’:
      yarl/_quoting_c.c:9287:21: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9287 |     *type = exc_info->exc_type;
            |                     ^~
      yarl/_quoting_c.c:9289:19: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9289 |     *tb = exc_info->exc_traceback;
            |                   ^~
      yarl/_quoting_c.c: In function ‘__Pyx__ExceptionReset’:
      yarl/_quoting_c.c:9303:24: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9303 |     tmp_type = exc_info->exc_type;
            |                        ^~
      yarl/_quoting_c.c:9305:22: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9305 |     tmp_tb = exc_info->exc_traceback;
            |                      ^~
      yarl/_quoting_c.c:9306:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9306 |     exc_info->exc_type = type;
            |             ^~
      yarl/_quoting_c.c:9308:13: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9308 |     exc_info->exc_traceback = tb;
            |             ^~
      yarl/_quoting_c.c: In function ‘__Pyx_ReraiseException’:
      yarl/_quoting_c.c:9349:20: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_type’
       9349 |     type = exc_info->exc_type;
            |                    ^~
      yarl/_quoting_c.c:9351:18: error: ‘_PyErr_StackItem’ {aka ‘struct _err_stackitem’} has no member named ‘exc_traceback’
       9351 |     tb = exc_info->exc_traceback;
            |                  ^~
      yarl/_quoting_c.c: In function ‘__Pyx_CreateCodeObjectForTraceback’:
      yarl/_quoting_c.c:10264:9: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
      10264 |         py_line,
            |         ^~~~~~~
            |         |
            |         int
      yarl/_quoting_c.c:320:72: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:10265:9: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
      10265 |         __pyx_empty_bytes  /*PyObject *lnotab*/
            |         ^~~~~~~~~~~~~~~~~
            |         |
            |         PyObject * {aka struct _object *}
      yarl/_quoting_c.c:320:79: note: in definition of macro ‘__Pyx_PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      yarl/_quoting_c.c:320:11: error: too few arguments to function ‘PyCode_New’
        320 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      yarl/_quoting_c.c:10250:15: note: in expansion of macro ‘__Pyx_PyCode_New’
      10250 |     py_code = __Pyx_PyCode_New(
            |               ^~~~~~~~~~~~~~~~
      In file included from /usr/local/include/python3.11/code.h:11,
                       from /usr/local/include/python3.11/Python.h:71,
                       from yarl/_quoting_c.c:6:
      /usr/local/include/python3.11/cpython/code.h:153:28: note: declared here
        153 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for yarl
Failed to build yarl
ERROR: Could not build wheels for yarl, which is required to install pyproject.toml-based projects

Python Version

$ python --version
Python 3.11.0a6

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/felixx/repo/yarl/.yarl-3.11/lib/python3.11/site-packages
Requires: 
Required-by:

yarl Version

$ python -m pip show yarl

OS

Windows, Linux

Additional context

No response

Code of Conduct

[X] I agree to follow the aio-libs Code of Conduct

bug

opened by felixxm 28

build classmethod

kwargs constructor for create a new url instance.

u = URL.build(
    scheme='http',
    host='127.0.0.1',
    user='foo',
    password='bar',
    port=8000,
    path='/index.html',
    query_string="arg=value1",
    fragment="top"
)

assert str(u) == 'http://foo:[email protected]:8000/index.html?arg=value1#top'

scheme and host is required in this case

new version of #57

opened by mosquito 25

Support quoting list values in mappings
What do these changes do?

These changes resolve aio-libs/aiohttp#4714.

Are there changes in behavior for the user?

Yes, users can now use lists or tuples when quoting mappings.

Checklist

[x] I think the code is well written

[x] Unit tests for the changes exist

[x] Documentation reflects the changes

[x] Add a new news fragment into the CHANGES folder

name it <issue_id>.<type> (e.g. 588.bugfix)

if you don't have an issue_id change it to the pr id after creating the PR

ensure type is one of the following:

.feature: Signifying a new feature.

.bugfix: Signifying a bug fix.

.doc: Signifying a documentation improvement.

.removal: Signifying a deprecation or removal of public API.

.misc: A ticket has been closed, but it is not of interest to users.

Make sure to use full sentences with correct case and punctuation, for example: Fix issue with non-ascii contents in doctest text files.
opened by Lonami 15

URL.build doesn't url encode credentials

EDIT the bug described here is just a consequence of the bug described below.

Currently square brackets in the username or the password lead to the URL being interpreted as an IPv6 address:

>>> yarl.URL("https://x:[@localhost")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hynek/.virtualenvs/ansible-tools/lib/python3.6/site-packages/yarl/__init__.py", line 149, in __init__
    val = urlsplit(val)
  File "/Users/hynek/.pyenv/versions/3.6.3/lib/python3.6/urllib/parse.py", line 439, in urlsplit
    raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL
Invalid IPv6 URL

>>> yarl.URL("https://[:x@localhost")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hynek/.virtualenvs/ansible-tools/lib/python3.6/site-packages/yarl/__init__.py", line 149, in __init__
    val = urlsplit(val)
  File "/Users/hynek/.pyenv/versions/3.6.3/lib/python3.6/urllib/parse.py", line 439, in urlsplit
    raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL
Invalid IPv6 URL

Sadly, the error seems to come from the stdlib tho:

>>> p.urlsplit("https://x:[@localhost")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hynek/.pyenv/versions/3.6.3/lib/python3.6/urllib/parse.py", line 439, in urlsplit
    raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL
Invalid IPv6 URL

opened by hynek 15

Consider making cython a build time dependency

The sdist currently on pypi will not build with py310 due to some changes to the c-api (https://github.com/python/cpython/pull/20429). Cython has already been updated to account for this (https://github.com/cython/cython/pull/3639), however because the yarl sdist includes the output of cython against an older combination of cpython / cython they are not guaranteed to work with future versions.

Because yarl does publish wheels, in most cases users will not need to have cython install and it will "just work" out of the box, however in cases where there are not wheels it can be broken in ways that are impossible to fix via the pip cli. If users are in a situation where they can not (or do not want to) use the wheels then they will also need a compiler (which is much harder to get installed than cython these days).

The datrie projects is a pretty clean example of how to set this up: https://github.com/pytries/datrie/blob/master/setup.py
enhancement question

opened by tacaswell 13

Yarl should ship manulinux1 wheels for Python 3.7

$ ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 апр 16  2018 /usr/bin/python -> python2.7

Collecting aiohttp==3.5.4
  Using cached https://files.pythonhosted.org/packages/33/60/c21acb7002110699761d3dec36fad3f5087c2752a4eb495444f877db6553/aiohttp-3.5.4-cp37-cp37m-manylinux1_x86_64.whl
Collecting attrs>=17.3.0 (from aiohttp==3.5.4)
  Using cached https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Collecting yarl<2.0,>=1.0 (from aiohttp==3.5.4)
  Using cached https://files.pythonhosted.org/packages/fb/84/6d82f6be218c50b547aa29d0315e430cf8a23c52064c92d0a8377d7b7357/yarl-1.3.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-w0tfguog/yarl/setup.py", line 25, in <module>
        extensions = cythonize(extensions)
      File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 899, in cythonize
        ctx = c_options.create_context()
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Main.py", line 590, in create_context
        self.cplus, self.language_level, options=self)
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Main.py", line 75, in __init__
        from . import Builtin, CythonScope
      File "/usr/lib/python3/dist-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
        from .UtilityCode import CythonUtilityCode
      File "/usr/lib/python3/dist-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
        from .TreeFragment import parse_from_strings, StringParseContext
      File "/usr/lib/python3/dist-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
        from .Visitor import VisitorTransform
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Visitor.py", line 15, in <module>
        from . import ExprNodes
      File "/usr/lib/python3/dist-packages/Cython/Compiler/ExprNodes.py", line 2844
        await = None
              ^
    SyntaxError: invalid syntax
    Error in sys.excepthook:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
        from apport.fileutils import likely_packaged, get_recent_crashes
      File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
        from apport.report import Report
      File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
        import apport.fileutils
      File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
        from apport.packaging_impl import impl as packaging
      File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
        import apt
      File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
        import apt_pkg
    ModuleNotFoundError: No module named 'apt_pkg'
    
    Original exception was:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-w0tfguog/yarl/setup.py", line 25, in <module>
        extensions = cythonize(extensions)
      File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 899, in cythonize
        ctx = c_options.create_context()
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Main.py", line 590, in create_context
        self.cplus, self.language_level, options=self)
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Main.py", line 75, in __init__
        from . import Builtin, CythonScope
      File "/usr/lib/python3/dist-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
        from .UtilityCode import CythonUtilityCode
      File "/usr/lib/python3/dist-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
        from .TreeFragment import parse_from_strings, StringParseContext
      File "/usr/lib/python3/dist-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
        from .Visitor import VisitorTransform
      File "/usr/lib/python3/dist-packages/Cython/Compiler/Visitor.py", line 15, in <module>
        from . import ExprNodes
      File "/usr/lib/python3/dist-packages/Cython/Compiler/ExprNodes.py", line 2844
        await = None
              ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-w0tfguog/yarl/

enhancement Hacktoberfest

opened by socketpair 13

Germanic problems -- IDNA does not round-trip

import aiohttp
import asyncio
import sys

hostname = None

async def fetch(session, url):
        async with session.get(url) as response:
            return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, hostname)
        print(html)

hostname = sys.argv[1]
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

$ python ./bug.py http://xn--einla-pqa/robots.txt
...
UnicodeError: ('IDNA does not round-trip', b'xn--einla-pqa', b'einlass')

opened by wumpus 12

Cannot install v0.9.3+ because of UnicodeDecodeError

Today I wanted to upgrade home assistant to the newest version in my virtualenv on my RPi3. Unfortunately this was not possible as the build of yarl failed. Turns out that the root of this problem is somewhere between v0.9.2 and 0.9.3, see the following error

Installing collected packages: yarl
  Found existing installation: yarl 0.9.2
    Uninstalling yarl-0.9.2:
      Successfully uninstalled yarl-0.9.2
  Running setup.py install for yarl ... error
  Rolling back uninstall of yarl
Exception:
Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/compat/__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 28: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/req/req_install.py", line 878, in install
    spinner=spinner,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/utils/__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/compat/__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 28: invalid start byte

opened by sibbl 12

yarl doesn't unquote on Python 3.7

On python 3.6.5 I can do this with yarl 1.3.0

>>> import yarl
>>> yarl.__version__
'1.3.0'
>>> url = yarl.URL('https://www.python.org/путь%2B')
>>> url.path
'/путь+'
>>> url.raw_path
'/%D0%BF%D1%83%D1%82%D1%8C%2B'

One python 3.7.2 and yarl 1.3.0 I get:

Python 3.7.2 (default, Feb 17 2019, 09:45:52) 
[GCC 9.0.1 20190209 (Red Hat 9.0.1-0.4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yarl
>>> yarl.__version__
'1.3.0'
>>> url = yarl.URL('https://www.python.org/путь%2B')
>>> url.path
'/путь%2B'
>>> url.raw_path
'/%D0%BF%D1%83%D1%82%D1%8C%2B'

All bits came from pypi. See how on 3.7.2 it doesn't unquote the %2B to the + like it does on 3.7.2. It is expected to be the + in path so the 3.7+ is the broken version. I've also reproduced this against Python 3.7.0 as well.

opened by bmbouter 10

Add extra chars to avoid unquoting

In some cases when we have already quoted chars: &(%26),(%2C);(%3B) into path, yarl unquote it to standard representation: ``&,;` this behavior is wrong because server send redirection message with quoted url, but we again unquote it

Fixes #223

opened by amarynets 10
Bump sphinx from 5.3.0 to 6.1.1
Bumps sphinx from 5.3.0 to 6.1.1.

Release notes

Sourced from sphinx's releases.

v6.1.1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.1.0

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b2

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

Changelog

Sourced from sphinx's changelog.

Release 6.1.1 (released Jan 05, 2023)

Bugs fixed

#11091: Fix util.nodes.apply_source_workaround for literal_block nodes with no source information in the node or the node's parents.

Release 6.1.0 (released Jan 05, 2023)

Dependencies

Adopted the Ruff_ code linter.

.. _Ruff: https://github.com/charliermarsh/ruff

Incompatible changes

#10979: gettext: Removed support for pluralisation in get_translation. This was unused and complicated other changes to sphinx.locale.

Deprecated

sphinx.util functions:

Renamed sphinx.util.typing.stringify() to sphinx.util.typing.stringify_annotation()

Moved sphinx.util.xmlname_checker() to sphinx.builders.epub3._XML_NAME_PATTERN

Moved to sphinx.util.display:

sphinx.util.status_iterator

sphinx.util.display_chunk

sphinx.util.SkipProgressMessage

sphinx.util.progress_message

Moved to sphinx.util.http_date:

sphinx.util.epoch_to_rfc1123

sphinx.util.rfc1123_to_epoch

Moved to sphinx.util.exceptions:

sphinx.util.save_traceback

... (truncated)

Commits

77aaa86 Bump to 6.1.1 final

476c115 Suppress ValueError in apply_source_workaround (#11092)

c80d656 Bump version

4e1004a Bump to 6.1.0 final

a2176d4 Fix deprecation warnings

2c104e9 Merge branch '6.0.x'

a27d262 Bump to 6.0.1 final

821569e Add note for Pygments

222d366 imgmath: Fix relative file path (#10965)

c499f66 Add SIM113 lint (#11057)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1

test_no_scheme2 fails with Python 3.11

Describe the bug

test_no_scheme2 fails with Python 3.11. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026634 (or your Actions log).

> =================================== FAILURES ===================================
> ________________________ TestScheme.test_not_a_scheme2 _________________________
> 
> self = <test_url_parsing.TestScheme object at 0x7f6efadeb150>
> 
>     def test_not_a_scheme2(self):
>         u = URL("37signals:book")
> >       assert u.scheme == "37signals"
> E       AssertionError: assert '' == '37signals'
> E         - 37signals
> 
> tests/test_url_parsing.py:77: AssertionError

Python 3.10 or lower:

Python 3.10.9 (main, Dec  7 2022, 13:47:07) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import urlsplit
>>> urlsplit('37signals:book')
SplitResult(scheme='37signals', netloc='', path='book', query='', fragment='')
>>>

Python 3.11:

Python 3.11.1 (main, Dec  7 2022, 08:49:13) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import urlsplit
>>> urlsplit('37signals:book')
SplitResult(scheme='', netloc='', path='37signals:book', query='', fragment='')
>>>

Kind Regards

To Reproduce

Just run the test with Python 3.11