Fast HTTP parser

Related tags

HTTP Clients httptools
Overview

Tests

httptools is a Python binding for the nodejs HTTP parser.

The package is available on PyPI: pip install httptools.

APIs

httptools contains two classes httptools.HttpRequestParser, httptools.HttpResponseParser and a function for parsing URLs httptools.parse_url. See unittests for examples.

class HttpRequestParser:

    def __init__(self, protocol):
        """HttpRequestParser

        protocol -- a Python object with the following methods
        (all optional):

          - on_message_begin()
          - on_url(url: bytes)
          - on_header(name: bytes, value: bytes)
          - on_headers_complete()
          - on_body(body: bytes)
          - on_message_complete()
          - on_chunk_header()
          - on_chunk_complete()
          - on_status(status: bytes)
        """

    def get_http_version(self) -> str:
        """Return an HTTP protocol version."""

    def should_keep_alive(self) -> bool:
        """Return ``True`` if keep-alive mode is preferred."""

    def should_upgrade(self) -> bool:
        """Return ``True`` if the parsed request is a valid Upgrade request.
	The method exposes a flag set just before on_headers_complete.
	Calling this method earlier will only yield `False`.
	"""

    def feed_data(self, data: bytes):
        """Feed data to the parser.

        Will eventually trigger callbacks on the ``protocol``
        object.

        On HTTP upgrade, this method will raise an
        ``HttpParserUpgrade`` exception, with its sole argument
        set to the offset of the non-HTTP data in ``data``.
        """

    def get_method(self) -> bytes:
        """Return HTTP request method (GET, HEAD, etc)"""


class HttpResponseParser:

    """Has all methods except ``get_method()`` that
    HttpRequestParser has."""

    def get_status_code(self) -> int:
        """Return the status code of the HTTP response"""


def parse_url(url: bytes):
    """Parse URL strings into a structured Python object.

    Returns an instance of ``httptools.URL`` class with the
    following attributes:

      - schema: bytes
      - host: bytes
      - port: int
      - path: bytes
      - query: bytes
      - fragment: bytes
      - userinfo: bytes
    """

Development

  1. Clone this repository with git clone --recursive [email protected]:MagicStack/httptools.git

  2. Create a virtual environment with Python 3.5: python3.5 -m venv envname

  3. Activate the environment with source envname/bin/activate

  4. Install Cython with pip install cython

  5. Run make and make test.

License

MIT.

Comments
  • Swap http-parse to llhttp

    Swap http-parse to llhttp

    Hi!

    First of all, I'm pretty new to Cython, so I'm sorry if I made any dumb mistakes here (pretty sure I might have messed up setup.py).

    Some observations:

    • ~~The tests are failing due to some import error, and I'm pretty sure it is because of the changes I made to setup.py~~ Tests are now passing
    • I changed the upgrade behavior and don't know if it is the right approach, would love some feedback
    • The upgrade tests used offset that I couldn't reproduce with llhttp so I might have missed something

    @elprans I would love some feedback here to get this going!

    Also tagging @indutny in case you have any 2cents or some warnings about something we might miss on this migration, or that we need extra care :D

    opened by victoraugustolls 23
  • Failed building wheel for httptools

    Failed building wheel for httptools

    Any idea on possible workaround:

    After pip3 install httptools:

    Collecting httptools
  Using cached https://files.pythonhosted.org/packages/1b/03/215969db11abe8741e9c266a4cbe803a372bd86dd35fa0084c4df6d4bd00/httptools-0.0.13.tar.gz
Building wheels for collected packages: httptools
  Building wheel for httptools (setup.py) ... error
  Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-lq_cgux2 --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/httptools
  copying httptools/__init__.py -> build/lib.linux-x86_64-3.6/httptools
  creating build/lib.linux-x86_64-3.6/httptools/parser
  copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.6/httptools/parser
  copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.6/httptools/parser
  running egg_info
  writing httptools.egg-info/PKG-INFO
  writing dependency_links to httptools.egg-info/dependency_links.txt
  writing top-level names to httptools.egg-info/top_level.txt
  reading manifest file 'httptools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'httptools.egg-info/SOURCES.txt'
  copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.6/httptools/parser
  running build_ext
  building 'httptools.parser.parser' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/httptools
  creating build/temp.linux-x86_64-3.6/httptools/parser
  creating build/temp.linux-x86_64-3.6/vendor
  creating build/temp.linux-x86_64-3.6/vendor/http-parser
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.6/httptools/parser/parser.o -O2
  httptools/parser/parser.c:4:20: fatal error: Python.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for httptools
  Running setup.py clean for httptools
Failed to build httptools
Installing collected packages: httptools
  Running setup.py install for httptools ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-xaxh1zu8/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/httptools
    copying httptools/__init__.py -> build/lib.linux-x86_64-3.6/httptools
    creating build/lib.linux-x86_64-3.6/httptools/parser
    copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.6/httptools/parser
    copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.6/httptools/parser
    running egg_info
    writing httptools.egg-info/PKG-INFO
    writing dependency_links to httptools.egg-info/dependency_links.txt
    writing top-level names to httptools.egg-info/top_level.txt
    reading manifest file 'httptools.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'httptools.egg-info/SOURCES.txt'
    copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.6/httptools/parser
    running build_ext
    building 'httptools.parser.parser' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/httptools
    creating build/temp.linux-x86_64-3.6/httptools/parser
    creating build/temp.linux-x86_64-3.6/vendor
    creating build/temp.linux-x86_64-3.6/vendor/http-parser
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.6/httptools/parser/parser.o -O2
    httptools/parser/parser.c:4:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-xaxh1zu8/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-ffzualnz/httptools/

    Ubuntu 16.04 LTS

    opened by dejanbatanjac 21
  • Example with aiohttp

    Example with aiohttp

    In your uvloop docs you talk about running aiohttp with this parser instead of the built in one. Do you have any documentation on how that would work?

    opened by nhumrich 13
  • Chunked request

    Chunked request

    i am working on httptools integration with aiohttp, some tests are failing with chunking and compression. if i modify httptools tests, it does not pass as well.

    
CHUNKED_REQUEST1_3 = b'''POST /test.php?a=b+c HTTP/1.2
User-Agent: Fooo
Host: bar
Transfer-Encoding: chunked

b\r\n+\xce\xcfM\xb5MI,I\x04\x00\r\n0\r\n\r\n'''
    opened by fafhrd91 10
  • provide wheel for python3.9

    provide wheel for python3.9

    Would it be possible to add wheels for python3.9 with a new patch release? :) Currently it is only possible to compile the package on the fly in a virtualized environment on python3.9.

    opened by dvzrv 9
  • Upgrade response

    Upgrade response

    here test code. also i am not sure how to get data after end of message back. i am about \x89\x04data

    
import httptools

import unittest
from unittest import mock

UPGRADE_RESPONSE = b'''HTTP/1.1 101 Switching Protocols
UPGRADE: websocket
SEC-WEBSOCKET-ACCEPT: rVg+XakFNFOxk3ZH0lzrZBmg0aU=
TRANSFER-ENCODING: chunked
CONNECTION: upgrade
DATE: Sat, 07 May 2016 23:44:32 GMT
SERVER: Python/3.4 aiohttp/1.0.3
\r\n\x89\x04data'''


class TestResponseParser(unittest.TestCase):

    def test_parser_upgrade_response(self):
        m = mock.Mock()

        headers = {}
        m.on_header.side_effect = headers.__setitem__

        p = httptools.HttpResponseParser(m)
        try:
            p.feed_data(UPGRADE_RESPONSE)
        except:
            pass

        self.assertEqual(p.get_http_version(), '1.1')
        self.assertEqual(p.get_status_code(), 101)

        m.on_status.assert_called_once_with(b'Switching Protocols')

        m.on_headers_complete.assert_called_once_with()
        self.assertEqual(m.on_header.call_count, 6)
        self.assertEqual(len(headers), 6)

        m.on_message_complete.assert_called_once_with()
    opened by fafhrd91 9
  • HttpParserUpgrade raised after event hooks are called.

    HttpParserUpgrade raised after event hooks are called.

    The HttpParserUpgrade exception appears to get called after the event hooks have been called into (at least with small requests that fit into a single data_received call)

    This makes it not terribly useful, as we may have already called into a handler function / issued a response / etc... by the time the upgrade is raised.

    Sketch of protocol class:

    def __init__(self, ...):
    self.request_parser = httptools.HttpRequestParser(self)
    ...

    # The asyncio.Protocol hooks...
    ...

    def data_received(self, data):
        print('data_received()')
        try:
            self.request_parser.feed_data(data)
        except httptools.HttpParserUpgrade:
            print('upgrade exception')

    # Event hooks called back into by HttpRequestParser...
    ...

    def on_headers_complete(self):
        print('on_headers_complete()')

    def on_message_complete(self):
        print('on_message_complete()')

    Output:

    data_received()
on_headers_complete()
on_message_complete()
upgrade exception

    On first sight this looks awkward to resolve, so it could just be considered a constraint of the implementation, and deal with it in python-land instead (which would be acceptable from my POV)

    opened by tomchristie 5
  • Python 2 support

    Python 2 support

    Would you accept a Python 2 support PR?

    It looks like not a lot of code changes are required.

    I played around a little, here are my findings:

    • the Cython code just compiles and seems to work on Python 2 unchanged

    • the __all__ in the __init__.py fails, because I think __all__ in Cython is a unicode object, something Python 2 doesn't like. I'd advocate changing the __init__.py to explicitly import everything needed instead of doing a from .. import *. This way you are in full control of what's exposed in the public API.

    • the test suite relies on some features from Python 3:

      • in particular unitest.mock, which is available as mock on PyPI so that's easy
      • assertRaisesRegex which is missing. It'd need to be backported, or the test rewritten as the regexes used aren't complicated.
      • subTest, either backported or the test rewritten.

      I'm a fan of pytest myself so alternatively we can port the tests over over to that; it's compatible with Python 2 and Python 3. I'd be happier doing that instead.

    • I recommend a tool like tox so we can easily run the tests on both Pythons.

    • the Makefile is geared towards Python 3, but doesn't do all that much. I just called the build_ext command myself with Python 2. We could cruftify the makefile to support Python 2. Alternatively we could also just simply document the few standard Python commands needed; tox should care of building stuff and running stuff anyway and that's a simple command.

    opened by faassen 5
  • llhttp critical CVE's

    llhttp critical CVE's

    Hi! (Again),

    Thanks for fixing the previous CVE's which i reported here last year. Today i noticed 3 CVE's (9.1 score) which are a bit obscurely reported by the nodeJS release notes, and which have been fixed in llhttp 6.0.7. [https://www.opencve.io/cve?vendor=llhttp]

    This morning I tried to prepare a merge-request to bump llhttp to 6.0.9, but I came to the conclusion it's best to get some advice. It seems there's some controversie about llhttp dropping support for LF delimiting headers and requiring CR delimiters.

    I'm not sure how httptools and other projects using httptools potentially could get impacted by such a change, and if there are better ways to get arround this.. or if it's not an issue.

    So i stopped the process of refactoring the test-cases, to get them to just pass, and ask for help.. (I'd hoped to be of use, but this is a too big of a question for me / I'll likely hold up the process of getting this fixed properly and timely)

    notes:

    1. Add the release branch to .gitmodules: [submodule "vendor/llhttp"] path = vendor/llhttp url = https://github.com/nodejs/llhttp.git branch = release

    2. Use a recent git version to run: git submodule update --remote (or use some other methods if you prefer not to update your git client).

    3. test_parser.py contains 'HTTP/1.2' in some of the validations causing tests to fail with an exception about invalid version .. change the version to HTTP/1.1 everywhere

    4. then all the remaining exceptions are about the CR 's that are required File "httptools\parser\parser.pyx", line 212, in httptools.parser.parser.HttpParser.feed_data httptools.parser.errors.HttpParserError: Missing expected CR after header value

    Thanks again!

    opened by nlsj1985 4
  • Doesn't build against Python 3.11

    Doesn't build against Python 3.11 

            running build_ext
        building 'httptools.parser.parser' extension
        creating build/temp.linux-x86_64-3.11
        creating build/temp.linux-x86_64-3.11/httptools
        creating build/temp.linux-x86_64-3.11/httptools/parser
        creating build/temp.linux-x86_64-3.11/vendor
        creating build/temp.linux-x86_64-3.11/vendor/llhttp
        creating build/temp.linux-x86_64-3.11/vendor/llhttp/src
        x86_64-linux-gnu-gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/tmp/pip-req-build-cyh4gi6y/vendor/llhttp/include -I/tmp/pip-req-build-cyh4gi6y/vendor/llhttp/src -I/home/dima/.cache/pypoetry/virtualenvs/silly-chat-9D3wOO4G-py3.11/include -I/usr/include/python3.11 -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.11/httptools/parser/parser.o -O2
        httptools/parser/parser.c: In function ‘__pyx_pf_9httptools_6parser_6parser_10HttpParser_10feed_data’:
        httptools/parser/parser.c:3452:23: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         3452 |       __pyx_v_err_pos = llhttp_get_error_pos(__pyx_v_self->_cparser);
              |                       ^
        httptools/parser/parser.c: In function ‘__Pyx_AddTraceback’:
        httptools/parser/parser.c:454:62: error: dereferencing pointer to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
          454 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
              |                                                              ^~
        httptools/parser/parser.c:9421:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
         9421 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
              |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
        error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
        [end of output]
    opened by dimaqq 3
  • Deprecation warnings in Python 3.10

    Deprecation warnings in Python 3.10

    I am not sure if its due to the deprecation warnings but there are failures while installing it on Python 3.10. Error log as below :

    Probably related to https://github.com/python/cpython/pull/20878

        Running setup.py install for httptools ... error
    ERROR: Command errored out with exit status 1:
     command: /root/dev-denv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ldf_awgo/install-record.txt --single-version-externally-managed --compile --install-headers /root/dev-denv/include/site/python3.10/httptools
         cwd: /tmp/pip-install-3qm507hv/httptools/
    Complete output (155 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.10
    creating build/lib.linux-x86_64-3.10/httptools
    copying httptools/_version.py -> build/lib.linux-x86_64-3.10/httptools
    copying httptools/__init__.py -> build/lib.linux-x86_64-3.10/httptools
    creating build/lib.linux-x86_64-3.10/httptools/parser
    copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.10/httptools/parser
    copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.10/httptools/parser
    running egg_info
    writing httptools.egg-info/PKG-INFO
    writing dependency_links to httptools.egg-info/dependency_links.txt
    writing requirements to httptools.egg-info/requires.txt
    writing top-level names to httptools.egg-info/top_level.txt
    reading manifest file 'httptools.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'httptools.egg-info/SOURCES.txt'
    copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.10/httptools/parser
    running build_ext
    building 'httptools.parser.parser' extension
    creating build/temp.linux-x86_64-3.10
    creating build/temp.linux-x86_64-3.10/httptools
    creating build/temp.linux-x86_64-3.10/httptools/parser
    creating build/temp.linux-x86_64-3.10/vendor
    creating build/temp.linux-x86_64-3.10/vendor/http-parser
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/root/dev-denv/include -I/root/cpython/Include -I/root/cpython -I/tmp/pip-install-3qm507hv/httptools/vendor/http-parser -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.10/httptools/parser/parser.o -O2
    httptools/parser/parser.c: In function ‘__pyx_tp_dealloc_9httptools_6parser_6parser_HttpParser’:
    httptools/parser/parser.c:8187:5: error: lvalue required as increment operand
         ++Py_REFCNT(o);
         ^~
    httptools/parser/parser.c:8189:5: error: lvalue required as decrement operand
         --Py_REFCNT(o);
         ^~
    httptools/parser/parser.c: In function ‘__Pyx_ParseOptionalKeywords’:
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c: In function ‘__Pyx_decode_c_bytes’:
    httptools/parser/parser.c:10379:9: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
             return PyUnicode_FromUnicode(NULL, 0);
             ^~~~~~
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:551:42: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
                                              ^~~~~~~~~~~~~~~~~~~~~
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/dev-denv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ldf_awgo/install-record.txt --single-version-externally-managed --compile --install-headers /root/dev-denv/include/site/python3.10/httptools Check the logs for full command output.
    opened by tirkarthi 3
  • How to do an early stop?

    How to do an early stop?

    Hi there :wave:

    I'm trying to implement a flag --limit-request-header-count on uvicorn, which I implemented changing the value of an attribute too_many_headers and raising an exception from on_header callback, see https://github.com/encode/uvicorn/pull/1683/files.

    My question is: is my implementation as intended? Should the HttpParserCallbackError send more information when an exception happens from one of the callbacks?

    opened by Kludex 2
  • bump to llhttp v6.0.10

    bump to llhttp v6.0.10

    Please bump httptools version. It seems that there was some parsing issue remaining with regard to the CVE's llhttp v6.0.10 changelog: http: disable chunked encoding when OBS fold is used

    Fixes: https://hackerone.com/reports/1630336 Fixes: https://hackerone.com/reports/1665156 Fixes: https://hackerone.com/reports/1675191

    opened by nlsj1985 4
  • PROXY protocol v1 / v2 support

    PROXY protocol v1 / v2 support

    Behind a reverse-proxy or a load balancer, you often need to get the original client IP. A lot of load balancer provide these information with the PROXY protocol - HAproxy for exemple. I have a load balancer that provide only this method to get the client info, and I would not be the only one.

    Here a doc of the protocol: http://www.haproxy.org/download/2.4/doc/proxy-protocol.txt

    In a nutshell, it's one line added on top of the body, here for the v1 protocol:

        PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
    GET / HTTP/1.1\r\n
    Host: 192.168.0.11\r\n
    \r\n

    This protocol is implemented in Gunicorn, and can be activated with --proxy-protocol: https://docs.gunicorn.org/en/stable/settings.html#proxy-protocol

    edit: A NodeJS parser: https://github.com/racker/node-proxy-protocol/blob/master/index.js

    opened by elrik75 0
  • Response parser requires support for HEAD responses without body

    Response parser requires support for HEAD responses without body

    According to the specification of HTTP/1.1 ...

    Responses to the HEAD request method never include a message body because the associated response header fields (e.g., Transfer-Encoding, Content-Length, etc.), if present, indicate only what their values would have been if the request method had been GET.

    Source: RFC 7230 (Section 3.3)

    ... responses to HEAD requests do not have a body. Even if e.g. the Content-Length header is present. However, the response parser cannot know based on the data it receives whether the corresponding request was using the HEAD or another method and whether to expect a body or to just ignore any header flags about the content.

    Suggestion:

    Add an additional boolean flag to HttpResponseParser.feed_data(self, data: bytes) to tell the parser that it should not expect a body (or even throw a parser exception if there is one present).

    Because HTTP/1.1 also allows pipelining, this additional parameter should also rather be a list of "body presence" flags. Here's why: If you send multiple requests to a server without waiting for the responses (= pipelining), you are going to receive back a bunch of responses together. Without them being parsed yet, one cannot identify the message boundaries and therefore they cannot be fed to the parser one by one. So the parser has to support that the whole stack of responses is fed to it and it needs a list of the "expected body presence". Example: If the pipelined requests were something like {HEAD, GET, HEAD, GET}, the "body presence" flags need to alternate as well. For convenience, it may be good to have a parameter that can be set using both types, a single flag and a list of flags.

    opened by mxscho 1
Releases(v0.5.0)

  • v0.5.0(Sep 13, 2022)

    Changes

    • Bump bundled llhttp to 6.0.9 fixes CVE-2022-32213, CVE-2022-32214, CVE-2022-32215 (by @nlsj1985 in 56d6a163 for #83)

    • Test and build against Python 3.11 (by @elprans in 509cd149 for #84)

    Source code(tar.gz)
    Source code(zip)

  • v0.4.0(Feb 22, 2022)

  • v0.3.0(Aug 10, 2021)

    This release has no functional changes, only packaging: Python 3.5 is EOL, so wheels are no longer built, and Python 3.10 has been added to the roster along with aarch64 wheels on Linux and universal2 wheels on macOS.

    Changes:

    • Use cibuildwheel to build release wheels (by @elprans in 2f57b6b7)
    Source code(tar.gz)
    Source code(zip)

  • v0.1.2(Apr 26, 2021)

    Bug Fixes

    • Fix httptools.__all__ (by @elprans in 9340d321 for #52)

    Build

    • Add Python 3.9 in the build/test matrix (by @b0g3r in e2d1a464 for #62)
    Source code(tar.gz)
    Source code(zip)

  • v0.2.0(Apr 23, 2021)

    New Features

    • Swap http-parse to llhttp (by @victoraugustolls and @fantix in 63b5de2b for #56)

    Bug Fixes

    • Fix httptools.__all__ (by @elprans in 9340d321 for #52)

    Build

    • Add Python 3.9 in the build/test matrix (by @b0g3r in e2d1a464 for #62)
    Source code(tar.gz)
    Source code(zip)

  • v0.1.1(Apr 26, 2021)

  • v0.1.0(Feb 5, 2020)

Owner
magicstack
Any sufficiently advanced software is indistinguishable from magic.
magicstack
GitHub
HTTP request/response parser for python in C

http-parser HTTP request/response parser for Python compatible with Python 2.x (>=2.7), Python 3 and Pypy. If possible a C parser based on http-parser

Benoit Chesneau 334 Dec 24, 2022
Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

null 457 Dec 10, 2022
A next generation HTTP client for Python. 🦋

HTTPX - A next-generation HTTP client for Python. HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support

Encode 9.8k Jan 5, 2023
A simple, yet elegant HTTP library.

Requests Requests is a simple, yet elegant HTTP library. >>> import requests >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass')

Python Software Foundation 48.8k Jan 5, 2023
Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

Twisted Matrix Labs 553 Dec 18, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Dec 29, 2022
Asynchronous HTTP client/server framework for asyncio and Python

Async http client/server framework Key Features Supports both client and server side of HTTP protocol. Supports both client and server Web-Sockets out

aio-libs 13.1k Jan 1, 2023
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Jan 2, 2023
As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

HTTPie 25.4k Jan 1, 2023
A minimal HTTP client. ⚙️

HTTP Core Do one thing, and do it well. The HTTP Core package provides a minimal low-level HTTP client, which does one thing only. Sending HTTP reques

Encode 306 Dec 27, 2022
Asynchronous Python HTTP Requests for Humans using Futures

Asynchronous Python HTTP Requests for Humans Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the ba

Ross McFarland 2k Dec 30, 2022
HTTP/2 for Python.

Hyper: HTTP/2 Client for Python This project is no longer maintained! Please use an alternative, such as HTTPX or others. We will not publish further

Hyper 1k Dec 23, 2022
An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie

HTTP Prompt HTTP Prompt is an interactive command-line HTTP client featuring autocomplete and syntax highlighting, built on HTTPie and prompt_toolkit.

HTTPie 8.6k Dec 31, 2022
HTTP Request Smuggling Detection Tool

HTTP Request Smuggling Detection Tool HTTP request smuggling is a high severity vulnerability which is a technique where an attacker smuggles an ambig

Anshuman Pattnaik 282 Jan 3, 2023
Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once

pathprober Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once. Purpose Brute-forcing webs

NFA 41 Jul 6, 2022
🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

?? ?? Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

Hackers and Slackers 15 Dec 12, 2022
A Python obfuscator using HTTP Requests and Hastebin.

?? Jawbreaker ?? Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

Billy 50 Sep 28, 2022
suite de mocks http em json

Ritchie Formula Repo Documentation Contribute to the Ritchie community This repository contains rit formulas which can be executed by the ritchie-cli.

Kaio Fábio Prates Prudêncio 1 Nov 1, 2021
Script to automate PUT HTTP method exploitation to get shell.

Script to automate PUT HTTP method exploitation to get shell.

devploit 116 Nov 10, 2022