A Python pickling decompiler and static analyzer

Related tags

fickling
Overview

Fickling

Fickling is a decompiler, static analyzer, and bytecode rewriter for Python pickle object serializations.

Pickled Python objects are in fact bytecode that is interpreted by a stack-based virtual machine built into Python called the "Pickle Machine". Fickling can take pickled data streams and decompile them into human-readable Python code that, when executed, will deserialize to the original serialized object.

The authors do not prescribe any meaning to the “F” in Fickling; it could stand for “fickle,” … or something else. Divining its meaning is a personal journey in discretion and is left as an exercise to the reader.

Installation

Fickling has been tested on Python 3.6 through Python 3.9 and has very few dependencies. It can be installed through pip:

pip3 install fickling

This installs both the library and the command line utility.

Usage

Fickling can be run programmatically:

>>> import ast
>>> import pickle
>>> from fickling.pickle import Pickled
>>> print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4))
Module(
    body=[
        Assign(
            targets=[
                Name(id='result', ctx=Store())],
            value=List(
                elts=[
                    Constant(value=1),
                    Constant(value=2),
                    Constant(value=3),
                    Constant(value=4)],
                ctx=Load()))])

Fickling can also be run as a command line utility:

$ fickling pickled.data
result = [1, 2, 3, 4]

This is of course a simple example. However, Python pickle bytecode can run arbitrary Python commands (such as exec or os.system) so it is a security risk to unpickle untrusted data. You can test for common patterns of malicious pickle files with the --check-safety option:

$ fickling --check-safety pickled.data
Warning: Fickling failed to detect any overtly unsafe code, but the pickle file may still be unsafe.
Do not unpickle this file if it is from an untrusted source!

You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the --trace option.

Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the --inject option.

License

This utility was developed by Trail of Bits. It is licensed under the GNU Lesser General Public License v3.0. Contact us if you're looking for an exception to the terms. © 2021, Trail of Bits.

Issues
  • NotImplementedError: TODO: Add support for Opcode BINFLOAT

    NotImplementedError: TODO: Add support for Opcode BINFLOAT

    I was trying out something sophisticated with a simple model pre-trained on MNIST. But I git this error.

    Traceback (most recent call last):
      File ".\pytorch_poc.py", line 147, in <module>
        exfil_model.pickled.insert_python_exec(
      File ".\pytorch_poc.py", line 58, in pickled
        self._pickled = Pickled.load(pickle_file)
      File "C:\Python38\lib\site-packages\fickling\pickle.py", line 343, in load
        opcodes.append(Opcode(info=info, argument=arg, data=data, position=pos))
      File "C:\Python38\lib\site-packages\fickling\pickle.py", line 105, in __new__
        raise NotImplementedError(f"TODO: Add support for Opcode {info.name}")
    NotImplementedError: TODO: Add support for Opcode BINFLOAT
    

    I guess, the project still needs work to allow to make a full-fledged ML-based attack. Any plans for when this will be completed?

    opened by shreyansh26 2
  • Add EMPTY_SET Opcode

    Add EMPTY_SET Opcode

    Thanks for this great tool! This Opcode appears to be necessary to fickle some uses of PyTorch modules such as torch.nn.Linear, which I'd love to have support for.

    Here is an example which triggers the following error:

    # example.py
    import pickle
    from fickling.pickle import Pickled
    from torch import nn
    
    filename = "model.pt"
    model = nn.Linear(2, 1)
    
    with open(filename, "wb") as model_file:
        pickle.dump(model, model_file)
    
    with open(filename, "rb") as model_file:
        pickled = Pickled.load(model_file)
    
    $ python example.py
    Traceback (most recent call last):
      File "~/ml-attacks/src/pickle_deserialization/example.py", line 12, in <module>
        pickled = Pickled.load(model_file)
      File "~/ml-attacks/.venv/lib/python3.9/site-packages/fickling/pickle.py", line 343, in load
        opcodes.append(Opcode(info=info, argument=arg, data=data, position=pos))
      File "~/ml-attacks/.venv/lib/python3.9/site-packages/fickling/pickle.py", line 105, in __new__
        raise NotImplementedError(f"TODO: Add support for Opcode {info.name}")
    NotImplementedError: TODO: Add support for Opcode EMPTY_SET
    

    Is anything else needed?

    opened by willclarktech 1
  • Add DICT and INT opcodes

    Add DICT and INT opcodes

    Two very simple additions: The INT opcode is used to declare constant integer values. The DICT opcode reads values from the stack until it reaches a MARK opcode, alternating between keys and values.

    Take the following script:

    import ast
    import pickletools
    from fickling.pickle import Pickled
    
    if __name__ == '__main__':
    	
    	pickled = b"(I1\nI2\nd."
    
    	for op, arg, pos in pickletools.genops(pickled):
    		print(f"{pos}: {op.name} {arg}")
    
    	ast_data = Pickled.load(pickled).ast
    	print(ast.dump(ast_data))
    

    Before, it would output:

    0: MARK None
    1: INT 1
    4: INT 2
    7: DICT None
    8: STOP None
    Traceback (most recent call last):
      File "test.py", line 12, in <module>
        ast_data = Pickled.load(pickled).ast
      File "/home/carlos/.local/lib/python3.7/site-packages/fickling/pickle.py", line 343, in load
        opcodes.append(Opcode(info=info, argument=arg, data=data, position=pos))
      File "/home/carlos/.local/lib/python3.7/site-packages/fickling/pickle.py", line 105, in __new__
        raise NotImplementedError(f"TODO: Add support for Opcode {info.name}")
    NotImplementedError: TODO: Add support for Opcode INT
    

    Now it outputs:

    0: MARK None
    1: INT 1
    4: INT 2
    7: DICT None
    8: STOP None
    Module(body=[Assign(targets=[Name(id='result', ctx=Store())], value=Dict(keys=[Constant(value=1)], values=[Constant(value=2)]))])
    
    opened by 00xc 1
  • Add NewObj_EX opcode

    Add NewObj_EX opcode

    Adds the NEWOBJ_EX opcode from pickle protocol 4

    2106 in pickletools.py

    opened by carsonharmon 0
  • Add TUPLE1, TUPLE2 and TUPLE3

    Add TUPLE1, TUPLE2 and TUPLE3

    Can you check these?

    opened by carsonharmon 0
  • Add example showing the creation of a hello world Trojan on PyTorch e…

    Add example showing the creation of a hello world Trojan on PyTorch e…

    Example code showing how tutorial code from PyTorch to load/save models can be backdoored with Fickle

    opened by carsonharmon 0
  • Carson/more poc

    Carson/more poc

    This adds an exec instead of an eval in pickle, and updates the PoC to just list current items in the directory. These could just be sent over the network anywhere.

    Next up is spAcy then Keras, then back to polytracker

    opened by carsonharmon 0
  • Adds support for the LONG_BINGET opcode

    Adds support for the LONG_BINGET opcode

    Also fixes a bug where empty sets would cause an error for some versions of Python that require ast.Set to always take an argument.

    enhancement 
    opened by ESultanik 0
  • Add new fickling examples

    Add new fickling examples

    This PR adds new examples for fickling and the APPEND opcode (required by PyTorchFi).

    opened by suhacker1 0
  • NotImplementedError: TODO: Add support for Opcode LONG1

    NotImplementedError: TODO: Add support for Opcode LONG1

    When attempting to use fickling on PyTorch models I get this error. I believe these models were just the weights. So i'm currious if this is hard to fix, and if you don't have time to fix it, any guidance you can give about the code base to help me attempt to patch it.

    opened by coldwaterq 0
Releases(v0.0.2)
Owner
Trail of Bits
More code: binary lifters @lifting-bits, blockchain @crytic
Trail of Bits
EML analyzer is an application to analyze the EML file

EML analyzer EML analyzer is an application to analyze the EML file which can: Analyze headers. Analyze bodies. Extract IOCs (URLs, domains, IP addres

Manabu Niseki 108 Oct 16, 2021
TODO aplication made with Python's FastAPI framework and Hexagonal Architecture

FastAPI Todolist Description Todolist aplication made with Python's FastAPI framework and Hexagonal Architecture. This is a test repository for the pu

Giovanni Armane 40 Oct 14, 2021
Social Distancing Detector using deep learning and capable to run on edge AI devices such as NVIDIA Jetson, Google Coral, and more.

Smart Social Distancing Smart Social Distancing Introduction Getting Started Prerequisites Usage Processor Optional Parameters Configuring AWS credent

Neuralet 99 Oct 21, 2021
Monitor Python applications using Spring Boot Admin

Pyctuator Monitor Python web apps using Spring Boot Admin. Pyctuator supports Flask, FastAPI, aiohttp and Tornado. Django support is planned as well.

SolarEdge Technologies 97 Oct 22, 2021
a lightweight web framework based on fastapi

start-fastapi Version 2021, based on FastAPI, an easy-to-use web app developed upon Starlette Framework Version 2020 中文文档 Requirements python 3.6+ (fo

HiKari 61 Sep 29, 2021
Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells & whistles to make life easier.

WitnessMe WitnessMe is primarily a Web Inventory tool inspired by Eyewitness, its also written to be extensible allowing you to create custom function

byt3bl33d3r 525 Oct 22, 2021
Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions

Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions To learn more about this project: medium blog post The goal of this proje

Ahmed BESBES 26 Sep 13, 2021
row level security for FastAPI framework

Row Level Permissions for FastAPI While trying out the excellent FastApi framework there was one peace missing for me: an easy, declarative way to def

Holger Frey 187 Oct 17, 2021
The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

Bruno Rocha 56 Oct 17, 2021
Admin Panel for GinoORM - ready to up & run (just add your models)

Gino-Admin Docs (state: in process): Gino-Admin docs Play with Demo (current master 0.2.3) >>>> Gino-Admin demo <<<< (login: admin, pass: 1234) Admin

Iuliia Volkova 38 Jun 16, 2021
Analytics service that is part of iter8. Robust analytics and control to unleash cloud-native continuous experimentation.

iter8-analytics iter8 enables statistically robust continuous experimentation of microservices in your CI/CD pipelines. For in-depth information about

null 16 Oct 14, 2021
A rate limiter for Starlette and FastAPI

SlowApi A rate limiting library for Starlette and FastAPI adapted from flask-limiter. Note: this is alpha quality code still, the API may change, and

Laurent Savaete 308 Oct 21, 2021
Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.

Introduction Apiestas is a project composed of a backend powered by the awesome framework FastAPI and a crawler powered by Scrapy. This project has fo

Fran Lozano 43 Oct 9, 2021
python template private service

Template for private python service This is a cookiecutter template for an internal REST API service, written in Python, inspired by layout-golang. Th

UrvanovCompany 12 Nov 24, 2020
A Prometheus Python client library for asyncio-based applications

aioprometheus aioprometheus is a Prometheus Python client library for asyncio-based applications. It provides metrics collection and serving capabilit

null 90 Oct 20, 2021
Web Version of avatarify to democratize even further

Web-avatarify for image animations This is the code base for this website and its backend. This aims to bring technology closer to everyone, just by a

Carlos Andrés Álvarez Restrepo 59 Sep 9, 2021
Minimal example utilizing fastapi and celery with RabbitMQ for task queue, Redis for celery backend and flower for monitoring the celery tasks.

FastAPI with Celery Minimal example utilizing FastAPI and Celery with RabbitMQ for task queue, Redis for Celery backend and flower for monitoring the

Grega Vrbančič 244 Oct 15, 2021
Install multiple versions of r2 and its plugins via Pip on any system!

r2env This repository contains the tool available via pip to install and manage multiple versions of radare2 and its plugins. r2-tools doesn't conflic

radare org 10 Sep 27, 2021
FastAPI Skeleton App to serve machine learning models production-ready.

FastAPI Model Server Skeleton Serving machine learning models production-ready, fast, easy and secure powered by the great FastAPI by Sebastián Ramíre

null 208 Oct 20, 2021