Single-file replacement for python-requests

Related tags

HTTP Clients mureq
Overview

mureq

mureq is a single-file, zero-dependency replacement for python-requests, intended to be vendored in-tree by Linux systems software and other lightweight applications. It is released under the 0BSD license to facilitate this.

>>> mureq.get('https://clients3.google.com/generate_204')
Response(status_code=204)
>>> response = _; response.status_code
204
>>> response.headers['date']
'Sun, 26 Dec 2021 01:56:04 GMT'
>>> response.body
b''
>>> params={'snap': 'certbot', 'interface': 'content'}
>>> response = mureq.get('http://snapd/v2/connections', params=params, unix_socket='/run/snapd.socket')
>>> response.status_code
200
>>> response.headers['Content-type']
'application/json'
>>> response.body
b'{"type":"sync","status-code":200,"status":"OK","result":{"established":[],"plugs":[],"slots":[]}}'

Why?

In short: performance (memory consumption), security (resilience to supply-chain attacks), and simplicity.

Performance

python-requests is extremely memory-hungry, mainly due to large transitive dependencies like chardet that are not needed by typical consumers. Here's a simple benchmark using Python 3.9.7, as packaged by Ubuntu 21.10 for amd64:

user@impish:~$ python3 -c "import os; os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:      7404 kB
user@impish:~$ python3 -c "import os, mureq; os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     13304 kB
user@impish:~$ python3 -c "import os, mureq; mureq.get('https://www.google.com'); os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     15872 kB
user@impish:~$ python3 -c "import os, requests; os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     21488 kB
user@impish:~$ python3 -c "import os, requests; requests.get('https://www.google.com'); os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     24352 kB

In terms of the time cost of HTTP requests, any differences between mureq and python-requests should be negligible, except in the case of workloads that use the connection pooling functionality of python-requests. Since mureq opens and closes a new connection for each request, migrating such a workload will incur a performance penalty. Note, however, that the normal python-requests API (requests.request, requests.get, etc.) also disables connection pooling, instead closing the socket immediately to prevent accidental resource leaks. In order to use connection pooling, you must explicitly create and manage a requests.Session object.

It's unclear to me whether connection pooling even makes sense in the typical Python context (single-threaded synchronous I/O, where there's no guarantee that the thread of control will re-enter the connection pool). It is much easier to implement this correctly in Go.

Security

Together with its transitive dependencies, python-requests is tens of thousands of lines of third-party code that cannot feasibly be audited. The most common way of distributing python-requests and its dependencies is pypi.org, which has relatively weak security properties: as of late 2021 it supports hash pinning, but not code signing. Typical Python deployments with third-party dependencies are vulnerable to supply-chain attacks against pypi.org, i.e., compromises of user credentials on pypi.org (or of pypi.org itself) that allow the introduction of malicious code into their dependencies.

In contrast, mureq is approximately 350 lines of code that can be audited easily and included directly in a project. Since mureq's functionality is limited in scope, you should be able to "install" it and forget about it.

Simplicity

python-requests was an essential addition to the ecosystem when it was created in 2011, but that time is past, and now in many cases the additional complexity it introduces is no longer justified:

  1. The standard library has caught up to python-requests in many respects. The most important change is PEP 476, which began validating TLS certificates by default against the system trust store. This change has landed in every version of Python that still receives security updates.
  2. Large portions of python-requests are now taken up with compatibility shims that cover EOL versions of Python, or that preserve compatibility with deprecated versions of the library itself.
  3. python-requests and urllib3 have never actually handled the low-level HTTP mechanics specified in RFC 7230 and its predecessors; this has always been deferred to the standard library (http.client in Python 3, httplib in Python 2). This is why it's so easy to reimplement the core functionality of python-requests in a small amount of code.

However, the API design of python-requests is excellent and in my opinion still considerably superior to that of urllib.request --- hence the case for a lightweight third-party library with a requests-like API.

How?

How do I install mureq?

mureq supports Python 3.6 and higher. Copy mureq.py into a suitable directory of your project, then import as you would any other internal module, e.g. import .mureq or import bar.baz.mureq.

Supply-chain attacks are considerably mitigated simply by vendoring mureq (i.e. copying it into your tree). If you are also concerned about future attacks on this GitHub account (or GitHub itself), tagged releases of mureq will be signed with the GPG key 0x740FC947B135E7627D4D00F21996B89DF018DCAB (expires 2025-07-28), or some future key in a chain of trust from it.

How do I use mureq?

The core API (mureq.get, mureq.post, mureq.request, etc.) is similar to python-requests, with a few differences. For now, see the docstrings in mureq.py itself for documentation. HTML documentation will be released later if there's a demand for it.

If you're switching from python-requests, there are a few things to keep in mind:

  1. mureq.get, mureq.post, and mureq.request mostly work like the analogous python-requests calls.
  2. The response type is mureq.HTTPResponse, which exposes fewer methods and properties than requests.Response. In particular, it does not have text (since mureq doesn't do any encoding detection) or json (since mureq doesn't depend on the json package). Instead, the response body is in the body member, which is always of type bytes. (For the sake of compatibility, the content property is provided as an alias for body.)
  3. The default way to send a POST body is with the body kwarg, which only accepts bytes.
  4. The json kwarg in mureq is not compatible with the corresponding kwarg in python-requests. In python-requests, json takes an arbitrary object, which the library then serializes to JSON on your behalf. In mureq, it takes an already-serialized str or bytes, e.g. the output of json.dumps(obj). (This is primarily to avoid pulling in the json package by default.) Unlike the aforementioned body kwarg, json will encode its argument as UTF-8 if necessary and add the usual Content-Type: application/json header.
  5. To send a form-encoded POST body, use the form kwarg. This accepts a dictionary of key-value pairs, or any object that can be serialized by urllib.parse.urlencode. It will add the usual Content-Type: application/x-www-form-urlencoded header.
  6. To make a request without reading the entire body at once, use with mureq.yield_response(url, method, **kwargs). This yields a http.client.HTTPResponse. Exiting the contextmanager automatically closes the socket.
  7. mureq does not follow HTTP redirections by default. To enable them, use the kwarg max_redirects, which takes an integer number of redirects to allow, e.g. max_redirects=2.
  8. All mureq calls should throw a subclass of mureq.HTTPException (which is actually just http.client.HTTPException for any runtime I/O error (including invalid HTTP responses, connection failures, timeouts, and exceeding the redirection limit). It may throw other exceptions (in particular ValueError) for programming errors, such as invalid or inconsistent arguments.
  9. mureq supports two ways of making HTTP requests over a Unix domain stream socket:
    • The unix_socket kwarg, which overrides the hostname in the URL, e.g. mureq.get('http://snapd/', unix_socket='/run/snapd.socket')
    • The http+unix URL scheme, which take the percent-encoded path as the hostname, e.g. http+unix://%2Frun%2Fsnapd.socket/ to connect to /run/snapd.socket.

Who?

If I were you, I would be asking: given that python-requests is used successfully on millions of systems, who is this person touting a replacement?

I'm nobody special --- not a security expert, not an HTTP protocol expert --- just someone who has been dealing with problems in this ecosystem for years. That's just the thing: HTTP isn't that hard! HTTP is already safe for humans.

Comments
  • Add missing response methods

    Add missing response methods

    Some notable (and simple) Response object methods are missing:

    • Reponse.json()
    • Reponse.raise_for_status()

    This PR adds those methods to the Response class.

    opened by mikeckennedy 6
  • Imports don't repeat, docs fixed

    Imports don't repeat, docs fixed

    I found this cool project through PythonBytes (https://pythonbytes.fm/episodes/show/268/wait-you-can-google-that).

    Just one little thing: There is no need to test if json has been imported already. Imports always only happen once. Just to be sure I made a little test:

    If you have a module module.py:

    print("module being imported")
    

    and a test file:

    def f1():
        import module as m
        print("inside f1")
    
    def f2():
        import module as m
        print("inside f2")
    
    f1()
    f2()
    

    the output is:

    module being imported
    inside f1
    inside f2
    

    And the readme needed updating :-)

    opened by jammon 4
  • Add support for auth parameter

    Add support for auth parameter

    python-requests supports passing an auth object, example:

    r = request(method, url, auth=("user", "password"))
    

    It detects basic authentication and builds the Basic {b64encoded} Authentication header for you It would be nice if mureq also supported this, to more easily replace requests in scripts that make use of auth

    opened by dlopes7 2
  • Add type hints

    Add type hints

    Would be really nice to have type hints for this project.

    I don't know if you want to target Python 2 or python < 3.5 but in that case you can use type comments.

    opened by RobertCraigie 4
  • support a callback for creating the connection

    support a callback for creating the connection

    Right now, _prepare_request initializes (but doesn't connect) either a HTTPConnection, an HTTPSConnection, or an UnixHTTPConnection depending on the URL and other parameters. If we added a optional callback arg that could construct the connection, we could potentially support a number of tricky use cases (similar to the "adapter" functionality in Requests): SO_REUSEPORT, SOCKS proxies, and others.

    opened by slingamn 0
  • add qa-exempting comments to mureq.py

    add qa-exempting comments to mureq.py

    1. Add # flake8: noqa. This is blocked on Ubuntu 22.04, which will contain the fix for https://github.com/pycqa/flake8/issues/318
    2. Add analogous noqa comments for other linters / IDEs
    opened by slingamn 0
Releases(v0.2.0)
  • v0.2.0(Feb 4, 2022)

    v0.2.0 is a new release of mureq, incorporating feedback from the Python community and patches from new contributors.

    Many thanks to @jammon, @mercutiodesign, and @mikeckennedy for contributing patches, and to everyone who contributed feedback.

    API breaks

    • json kwarg now accepts an arbitrary object to be serialized as JSON (matching the Requests behavior), instead of already-serialized JSON

    Added

    • Added raise_for_status() and json() methods to the Response type, increasing API compatibility with Requests
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jan 17, 2022)

Owner
Shivaram Lingamneni
Shivaram Lingamneni
Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

Twisted Matrix Labs 553 Dec 18, 2022
A modern/fast python SOAP client based on lxml / requests

Zeep: Python SOAP client A fast and modern Python SOAP client Highlights: Compatible with Python 3.6, 3.7, 3.8 and PyPy Build on top of lxml and reque

Michael van Tellingen 1.7k Jan 1, 2023
A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

null 892 Jan 6, 2023
Asynchronous Python HTTP Requests for Humans using Futures

Asynchronous Python HTTP Requests for Humans Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the ba

Ross McFarland 2k Dec 30, 2022
🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

?? ?? Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

Hackers and Slackers 15 Dec 12, 2022
A Python obfuscator using HTTP Requests and Hastebin.

?? Jawbreaker ?? Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

Billy 50 Sep 28, 2022
r - a small subset of Python Requests

r a small subset of Python Requests a few years ago, when I was first learning Python and looking for http functionality, i found the batteries-includ

Gabriel Sroka 4 Dec 15, 2022
Requests + Gevent = <3

GRequests: Asynchronous Requests GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. Note: You should probably

Spencer Phillip Young 4.2k Dec 30, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Dec 29, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Jan 2, 2023
Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

null 457 Dec 10, 2022
A next generation HTTP client for Python. 🦋

HTTPX - A next-generation HTTP client for Python. HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support

Encode 9.8k Jan 5, 2023
Asynchronous HTTP client/server framework for asyncio and Python

Async http client/server framework Key Features Supports both client and server side of HTTP protocol. Supports both client and server Web-Sockets out

aio-libs 13.1k Jan 1, 2023
PycURL - Python interface to libcurl

PycURL -- A Python Interface To The cURL library PycURL is a Python interface to libcurl, the multiprotocol file transfer library. Similarly to the ur

PycURL 933 Jan 9, 2023
HTTP/2 for Python.

Hyper: HTTP/2 Client for Python This project is no longer maintained! Please use an alternative, such as HTTPX or others. We will not publish further

Hyper 1k Dec 23, 2022
HTTP request/response parser for python in C

http-parser HTTP request/response parser for Python compatible with Python 2.x (>=2.7), Python 3 and Pypy. If possible a C parser based on http-parser

Benoit Chesneau 334 Dec 24, 2022
Python Simple SOAP Library

PySimpleSOAP / soap2py Python simple and lightweight SOAP library for client and server webservices interfaces, aimed to be as small and easy as possi

PySimpleSOAP 369 Jan 2, 2023
Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.

Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.

Paweł Piotr Przeradowski 8.6k Jan 4, 2023
Aiosonic - lightweight Python asyncio http client

aiosonic - lightweight Python asyncio http client Very fast, lightweight Python asyncio http client Here is some documentation. There is a performance

Johanderson Mogollon 93 Jan 6, 2023