T-Reqs: A grammar-based HTTP Fuzzer

Overview

T-Reqs HTTP Fuzzer

T-Reqs (Two Requests) is a grammar-based HTTP Fuzzer written as a part of the paper titled "T-Reqs: HTTP Request Smuggling with Differential Fuzzing" which was presented at ACM CCS 2021.

BibTeX of the paper:

@inproceedings{ccs2021treqs,
  title={T-Reqs: HTTP Request Smuggling with Differential Fuzzing},
  author={Jabiyev, Bahruz and Sprecher, Steven and Onarlioglu, Kaan and Kirda, Engin},
  booktitle={Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security},
  pages={1805--1820},
  year={2021}
}

About

T-Reqs is for fuzzing HTTP servers by sending mutated HTTP requests with versions 1.1 and earlier. It has three main components: 1) generating inputs, 2) mutating generated inputs and 3) delivering them to the target server(s).

Generating Inputs

A CFG grammar fed into the fuzzer is used to generate HTTP requests. As the example grammar shown below is tailored for request line fuzzing, every request line component and possible values for each of them are explicitly specified. This allows us to generate valid requests with various forms of request line and also to treat each request line component as a separate unit from the mutation perspective.

 '
   
    ':
     ['
    
     '],
 '
     
      ':
     ['
      
       
       
        '], '
        
         ': ['
         
          
           
            
             
              
               
                
                 '], '
                 
                  ': ['GET', 'HEAD', 'POST', 'PUT', 'DELETE', 'CONNECT', 'OPTIONS', 'TRACE', 'PATCH'], '
                  
                   ': [' '], '
                   
                    ': ['/_URI_'], '
                    
                     ': ['HTTP'], '
                     
                      ': ['/'], '
                      
                       ': ['0.9', '1.0', '1.1'], '
                       
                        ': ['\r\n'], '
                        ': ['Host: _HOST_\r\nConnection:close\r\nX-Request-ID: _REQUEST_ID_\r\n'], '
                        
                         ': ['Content-Length: 5\r\n\r\nBBBBBBBBBB'], 
                        
                       
                      
                     
                    
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
     
    
   

Mutating Inputs

Each component can be marked in two ways: string mutable and tree mutable (see the example configuration). If a component is string mutable, then a random character can be deleted, replaced, or inserted at a random position. In the example shown below (left side), the last character in the protocol version (1) is deleted, the third letter in the method name (S) is replaced with R, and a forward slash is inserted at the beginning of the URI. Whereas, if a component is tree mutable, then a random component can be deleted, replaced, or inserted at a random position under that component. The example below (right side) shows three tree mutations applied on the request line component: 1) method is replaced by protocol, 2) an extra URI is inserted after the current URI, and 3) the existing proto is deleted.

Mutation Types

Usage

Configuration

The fuzzer should be informed about the user preferences about the generation and mutation of inputs. More specifically, the input grammar, the mutable components, mutation preferences among other things should be specified in the configuration file (see an example configuration).

Running modes

To be able to reproduce the inputs generated and mutated in each iteration, a seed number is used. In fact, this seed number serves as a seed for random number generations during the formation and mutation of an input. Depending on how these seeds are fed into the fuzzer, it runs in one of these two modes: individual and collective. In the individual mode, inputs are generated and mutated based on the seeds specified by a user. In the command below, a single seed (i.e., 505) is specified. Alternatively, a list of seeds could be specified with -f option (see help page for more).

python3 main.py -i -c config -s 505

Whereas, in the collective mode (which is default), it starts from zero as the seed value and increments it in each iteration until the end number is reached. The beginning and end numbers can be customized.

python3 main.py -c config

Using for Finding HRS discrepancies

HTTP Request Smuggling relies on different body parsing behaviors between servers where one server uses Transfer-Encoding header while the other prefers Content-Length header to decide the boundaries of a request body, or one server ignores a request body, whereas the other one processes it.

To analyze the body parsing of servers in response to various mutations in various forms of an HTTP request, we need to have a feedback mechanism installed on those servers to tell us about the body parsing behavior. One way of installing a feedback mechanism on a server, is to run the server in the reverse-proxy mode and have it forward requests to a "feedback provider" script running as a service. This service measures the length of the body in received requests and saves it for comparing it later with other servers.

An example "feedback provider" script is available in this repository. However, this script sends the body length information back in a response assuming that this information is stored on the client side.

License

T-Reqs is licensed under MIT license.

You might also like...
Fast HTTP parser

httptools is a Python binding for the nodejs HTTP parser. The package is available on PyPI: pip install httptools. APIs httptools contains two classes

HTTP/2 for Python.
HTTP/2 for Python.

Hyper: HTTP/2 Client for Python This project is no longer maintained! Please use an alternative, such as HTTPX or others. We will not publish further

An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie
An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie

HTTP Prompt HTTP Prompt is an interactive command-line HTTP client featuring autocomplete and syntax highlighting, built on HTTPie and prompt_toolkit.

HTTP request/response parser for python in C

http-parser HTTP request/response parser for Python compatible with Python 2.x (=2.7), Python 3 and Pypy. If possible a C parser based on http-parser

HTTP Request Smuggling Detection Tool
HTTP Request Smuggling Detection Tool

HTTP Request Smuggling Detection Tool HTTP request smuggling is a high severity vulnerability which is a technique where an attacker smuggles an ambig

Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once
Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once

pathprober Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once. Purpose Brute-forcing webs

🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

A Python obfuscator using HTTP Requests and Hastebin.
A Python obfuscator using HTTP Requests and Hastebin.

🔨 Jawbreaker 🔨 Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

suite de mocks http em json
suite de mocks http em json

Ritchie Formula Repo Documentation Contribute to the Ritchie community This repository contains rit formulas which can be executed by the ritchie-cli.

Comments
  • Original Config Files

    Original Config Files

    Hi @bahruzjabiyev

    First of all, thanks for the amazing paper! I was able to run t-reqs (client and feedback server) and get the output (batch0.out) without any problem using the default config (https://github.com/bahruzjabiyev/t-reqs/blob/main/config). For the default config I can see in my batch0.out file that all 200 OK responses contains the body {body: 'BBBBB', body_length: 5} which, from my understanding, is the expected result based on the request test. Now I'm having some difficult to generate a properly config file to perform the same tests described on the paper. If I got this right for the tests described on the paper there were 3 different config files (request line, request headers, and request body). Could you please provide these 3 config files so I can check what I'm doing wrong? Thanks!

    Best regards, Ricardo Iramar

    opened by riramar 2
  • TypeError: cannot pickle '_thread.lock' object

    TypeError: cannot pickle '_thread.lock' object

    Hi, i'm getting this error when I try to run with the following command: python main.py -c ../config -i -s 505

    Traceback (most recent call last):
      File "/Users/cjensen/projects/T-Reqs-HTTP-Fuzzer/code/main.py", line 165, in <module>
        fuzzer.blackbox_fuzz_individual(fuzzer.seedfile, [fuzzer.seed])
      File "/Users/cjensen/projects/T-Reqs-HTTP-Fuzzer/code/main.py", line 117, in blackbox_fuzz_individual
        proc.start()
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
        return Popen(process_obj)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
        super().__init__(process_obj)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
        reduction.dump(process_obj, fp)
      File "/Users/cjensen/.pyenv/versions/3.9.11/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    TypeError: cannot pickle '_thread.lock' object
    

    Any idea what I can do?

    opened by ruevaughn 1
Owner
Bahruz Jabiyev
Bahruz Jabiyev
Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

null 457 Dec 10, 2022
A next generation HTTP client for Python. 🦋

HTTPX - A next-generation HTTP client for Python. HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support

Encode 9.8k Jan 5, 2023
A simple, yet elegant HTTP library.

Requests Requests is a simple, yet elegant HTTP library. >>> import requests >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass')

Python Software Foundation 48.8k Jan 5, 2023
Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

Twisted Matrix Labs 553 Dec 18, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Dec 29, 2022
Asynchronous HTTP client/server framework for asyncio and Python

Async http client/server framework Key Features Supports both client and server side of HTTP protocol. Supports both client and server Web-Sockets out

aio-libs 13.1k Jan 1, 2023
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Jan 2, 2023
As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

HTTPie 25.4k Jan 1, 2023
A minimal HTTP client. ⚙️

HTTP Core Do one thing, and do it well. The HTTP Core package provides a minimal low-level HTTP client, which does one thing only. Sending HTTP reques

Encode 306 Dec 27, 2022
Asynchronous Python HTTP Requests for Humans using Futures

Asynchronous Python HTTP Requests for Humans Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the ba

Ross McFarland 2k Dec 30, 2022