ZFS, in Python, without reading the original C.

Related tags

Storage zfsp
Overview

ZFSp

What?

ZFS, in Python, without reading the original C.

What?!

That's right.

How?

Many hours spent staring at hexdumps, and asking friends to search the internet for explanations of various features.

Why?

Why not?

It seemed like it might be a fun project.

Installation

The Pipfile lists the dependencies; there aren't many.

Python 3.5+ is required, but 3.6 is not, as PyPy runs this code much, much faster (around 4x on the test suite) and didn't support 3.6 until recently.

pipenv install -e .

N.B.: -e gets you an "editable" install; changes to the source tree will affect the installed package's behavior.

Test Suite.

Running the test suite requires one-time access to a system with ZFS, to generate the test pools.

Run tests/fixtures.sh on such a system and make the resulting directory tests/fixtures (this is where it will end up if you run fixtures.sh directly from the tests directory).

The tests themselves can be run with py.test. Most of the tests pass, but there are some known failures. Feel free to try fixing them.

The tests are heavily parameterized and attempt to run all tests against all relevant pool variations.

Usage

zexplore is the main command line interface. It's reasonably well documented (I hope).

There's a subcommand for some limited FUSE support, which depends on fusepy (developed with 2.0.4). This is not installed by default in the Pipfile.

Example commands:

$ zexplore label -p tests/fixtures/feature_large_blocks
{b'errata': 0,
 b'features_for_read': {},
 b'guid': 6168868809305637343,
 b'hostid': 8323329,
 b'hostname': b'ubuntu-zesty',
 b'name': b'feature_large_blocks',
 b'pool_guid': 16637796155898928459,
 b'state': 1,
 b'top_guid': 6168868809305637343,
 b'txg': 16,
 b'vdev_children': 1,
 b'vdev_tree': {b'ashift': 9,
                b'asize': 62390272,
                b'create_txg': 4,
                b'guid': 6168868809305637343,
                b'id': 0,
                b'is_log': 0,
                b'metaslab_array': 33,
                b'metaslab_shift': 24,
                b'path': b'/vagrant/fixtures/feature_large_blocks',
                b'type': b'file'},
 b'version': 5000}

$ zexplore objset -p tests/fixtures/nested_datasets -P 1
{'config': 27,
 'creation_version': 5000,
 'deflate': 1,
 'feature_descriptions': 30,
 'features_for_read': 28,
 'features_for_write': 29,
 'free_bpobj': 11,
 'history': 32,
 'root_dataset': 2,
 'sync_bplist': 31}

$ zexplore ls -p tests/fixtures/nested_datasets /n1/n2/n3
x
y

$ zexplore cat -p tests/fixtures/nested_datasets /gzip-9/gzip-9 | head -n 5
A
a
aa
aal
aalii

Caveats

Lots of things won't work, including:

  • raidz with more than 4 disks will probably fail with an inscrutable error. ZFS uses a magic allocation function that's a non-linear black box. I haven't figured it out. raidz where the disks are larger than a few GB will probably also fail, but I haven't tested that at all.
  • Some new feature flags have come out since I last worked on this seriously, and those features are (obviously) not supported. That's things like (incomplete list):
    • SHA-512/256
    • Skein
    • spacemap_v2
  • Some feature flags that did exist when I was actively working on this are not implemented, either because I failed to figure out how they work, or because I didn't get around to trying.
  • pyndata is the struct description library used to read all the on-disk structures. I wrote it first, and it's bad. If I was doing this again today, I'd probably build something with dataclasses.
  • There's a Rust implementation of LZJB which is not currently included, but decompression is not the bottleneck.
Comments
  • dict object has no attribute 'bp'

    dict object has no attribute 'bp'

    I have been attempting to get this to work with a minimal zpool created clean. when I try to run the 'ls' command with the argument -p (pool file) I get an exception on line 26 in datasets.py objset_block = self.pool.read_block(self.dsl_dataset.bp) debug

    opened by cinterloper 2
  • click_explore.py syntax error

    click_explore.py syntax error

      File "/Users/alexander/Code/python/zfsp/click_explore.py", line 54
        except Exception, e:
                        ^
    SyntaxError: invalid syntax
    
    opened by ghost 1
  • Use feature detection instead of version detection

    Use feature detection instead of version detection

    opened by cclauss 0
  • This is too cool.

    This is too cool.

    Might want to add a few corporate-PHB-inspired types of c—kups and dependencies, you know, in the true spirit of Oracle ;) and make it so complicated that you can charge for support.

    Thanks, Larry

    [💯 joking. I mean, about it being an issue. It actually is too cool. But just leave it that way. Take that Oracle! And my name's not Larry ]

    opened by paulyc 0
  • Hint for RAIDZ2/3

    Hint for RAIDZ2/3

    RAIDZ2/3 uses pre-calculated Galois Field array (GF(8)) and Reed-Solomon error correction.

    pow2[256] = {
    	0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80,
    	0x1d, 0x3a, 0x74, 0xe8, 0xcd, 0x87, 0x13, 0x26,
    	0x4c, 0x98, 0x2d, 0x5a, 0xb4, 0x75, 0xea, 0xc9,
    	0x8f, 0x03, 0x06, 0x0c, 0x18, 0x30, 0x60, 0xc0,
    	0x9d, 0x27, 0x4e, 0x9c, 0x25, 0x4a, 0x94, 0x35,
    	0x6a, 0xd4, 0xb5, 0x77, 0xee, 0xc1, 0x9f, 0x23,
    	0x46, 0x8c, 0x05, 0x0a, 0x14, 0x28, 0x50, 0xa0,
    	0x5d, 0xba, 0x69, 0xd2, 0xb9, 0x6f, 0xde, 0xa1,
    	0x5f, 0xbe, 0x61, 0xc2, 0x99, 0x2f, 0x5e, 0xbc,
    	0x65, 0xca, 0x89, 0x0f, 0x1e, 0x3c, 0x78, 0xf0,
    	0xfd, 0xe7, 0xd3, 0xbb, 0x6b, 0xd6, 0xb1, 0x7f,
    	0xfe, 0xe1, 0xdf, 0xa3, 0x5b, 0xb6, 0x71, 0xe2,
    	0xd9, 0xaf, 0x43, 0x86, 0x11, 0x22, 0x44, 0x88,
    	0x0d, 0x1a, 0x34, 0x68, 0xd0, 0xbd, 0x67, 0xce,
    	0x81, 0x1f, 0x3e, 0x7c, 0xf8, 0xed, 0xc7, 0x93,
    	0x3b, 0x76, 0xec, 0xc5, 0x97, 0x33, 0x66, 0xcc,
    	0x85, 0x17, 0x2e, 0x5c, 0xb8, 0x6d, 0xda, 0xa9,
    	0x4f, 0x9e, 0x21, 0x42, 0x84, 0x15, 0x2a, 0x54,
    	0xa8, 0x4d, 0x9a, 0x29, 0x52, 0xa4, 0x55, 0xaa,
    	0x49, 0x92, 0x39, 0x72, 0xe4, 0xd5, 0xb7, 0x73,
    	0xe6, 0xd1, 0xbf, 0x63, 0xc6, 0x91, 0x3f, 0x7e,
    	0xfc, 0xe5, 0xd7, 0xb3, 0x7b, 0xf6, 0xf1, 0xff,
    	0xe3, 0xdb, 0xab, 0x4b, 0x96, 0x31, 0x62, 0xc4,
    	0x95, 0x37, 0x6e, 0xdc, 0xa5, 0x57, 0xae, 0x41,
    	0x82, 0x19, 0x32, 0x64, 0xc8, 0x8d, 0x07, 0x0e,
    	0x1c, 0x38, 0x70, 0xe0, 0xdd, 0xa7, 0x53, 0xa6,
    	0x51, 0xa2, 0x59, 0xb2, 0x79, 0xf2, 0xf9, 0xef,
    	0xc3, 0x9b, 0x2b, 0x56, 0xac, 0x45, 0x8a, 0x09,
    	0x12, 0x24, 0x48, 0x90, 0x3d, 0x7a, 0xf4, 0xf5,
    	0xf7, 0xf3, 0xfb, 0xeb, 0xcb, 0x8b, 0x0b, 0x16,
    	0x2c, 0x58, 0xb0, 0x7d, 0xfa, 0xe9, 0xcf, 0x83,
    	0x1b, 0x36, 0x6c, 0xd8, 0xad, 0x47, 0x8e, 0x01
    };
    
    log2[256] = {
    	0x00, 0x00, 0x01, 0x19, 0x02, 0x32, 0x1a, 0xc6,
    	0x03, 0xdf, 0x33, 0xee, 0x1b, 0x68, 0xc7, 0x4b,
    	0x04, 0x64, 0xe0, 0x0e, 0x34, 0x8d, 0xef, 0x81,
    	0x1c, 0xc1, 0x69, 0xf8, 0xc8, 0x08, 0x4c, 0x71,
    	0x05, 0x8a, 0x65, 0x2f, 0xe1, 0x24, 0x0f, 0x21,
    	0x35, 0x93, 0x8e, 0xda, 0xf0, 0x12, 0x82, 0x45,
    	0x1d, 0xb5, 0xc2, 0x7d, 0x6a, 0x27, 0xf9, 0xb9,
    	0xc9, 0x9a, 0x09, 0x78, 0x4d, 0xe4, 0x72, 0xa6,
    	0x06, 0xbf, 0x8b, 0x62, 0x66, 0xdd, 0x30, 0xfd,
    	0xe2, 0x98, 0x25, 0xb3, 0x10, 0x91, 0x22, 0x88,
    	0x36, 0xd0, 0x94, 0xce, 0x8f, 0x96, 0xdb, 0xbd,
    	0xf1, 0xd2, 0x13, 0x5c, 0x83, 0x38, 0x46, 0x40,
    	0x1e, 0x42, 0xb6, 0xa3, 0xc3, 0x48, 0x7e, 0x6e,
    	0x6b, 0x3a, 0x28, 0x54, 0xfa, 0x85, 0xba, 0x3d,
    	0xca, 0x5e, 0x9b, 0x9f, 0x0a, 0x15, 0x79, 0x2b,
    	0x4e, 0xd4, 0xe5, 0xac, 0x73, 0xf3, 0xa7, 0x57,
    	0x07, 0x70, 0xc0, 0xf7, 0x8c, 0x80, 0x63, 0x0d,
    	0x67, 0x4a, 0xde, 0xed, 0x31, 0xc5, 0xfe, 0x18,
    	0xe3, 0xa5, 0x99, 0x77, 0x26, 0xb8, 0xb4, 0x7c,
    	0x11, 0x44, 0x92, 0xd9, 0x23, 0x20, 0x89, 0x2e,
    	0x37, 0x3f, 0xd1, 0x5b, 0x95, 0xbc, 0xcf, 0xcd,
    	0x90, 0x87, 0x97, 0xb2, 0xdc, 0xfc, 0xbe, 0x61,
    	0xf2, 0x56, 0xd3, 0xab, 0x14, 0x2a, 0x5d, 0x9e,
    	0x84, 0x3c, 0x39, 0x53, 0x47, 0x6d, 0x41, 0xa2,
    	0x1f, 0x2d, 0x43, 0xd8, 0xb7, 0x7b, 0xa4, 0x76,
    	0xc4, 0x17, 0x49, 0xec, 0x7f, 0x0c, 0x6f, 0xf6,
    	0x6c, 0xa1, 0x3b, 0x52, 0x29, 0x9d, 0x55, 0xaa,
    	0xfb, 0x60, 0x86, 0xb1, 0xbb, 0xcc, 0x3e, 0x5a,
    	0xcb, 0x59, 0x5f, 0xb0, 0x9c, 0xa9, 0xa0, 0x51,
    	0x0b, 0xf5, 0x16, 0xeb, 0x7a, 0x75, 0x2c, 0xd7,
    	0x4f, 0xae, 0xd5, 0xe9, 0xe6, 0xe7, 0xad, 0xe8,
    	0x74, 0xd6, 0xf4, 0xea, 0xa8, 0x50, 0x58, 0xaf,
    };
    
    opened by raspi 0
Owner
Colin Valliant
Colin Valliant
An ZFS administration tool inspired on Midnight commander

ZC - ZFS Commander An ZFS administration tool inspired on Midnight commander Work in Progress Description ZFS Commander is a simple front-end for the

null 34 Dec 7, 2022
Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of images as "pixels"

picinpics Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of

RodrigoCMoraes 1 Oct 24, 2021
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

Institut für Kraftfahrzeuge, RWTH Aachen, ika 12 Sep 1, 2022
This repository contains a streaming Dataflow pipeline written in Python with Apache Beam, reading data from PubSub.

Sample streaming Dataflow pipeline written in Python This repository contains a streaming Dataflow pipeline written in Python with Apache Beam, readin

Israel Herraiz 9 Mar 18, 2022
borb is a library for reading, creating and manipulating PDF files in python.

borb is a library for reading, creating and manipulating PDF files in python.

Joris Schellekens 2.9k Jan 1, 2023
Small exercises to get you used to reading and writing Python code!

Pythonlings Welcome to Pythonlings, an automated Python tutorial program (inspired by Rustlings and Haskellings). WIP This program is still working in

鹤翔万里 5 Sep 23, 2022
Arxiv2Kindle is a simple script written in python that converts LaTeX source downloaded from Arxiv and recompiles it to better fit a Kindle or other similar reading devices.

Arxiv2Kindle is a simple script written in python that converts LaTeX source downloaded from Arxiv and recompiles it to better fit a read

Soumik Rakshit 8 Jul 9, 2022
Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in Python

AICSImageIO Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in Pure Python Features Supports reading metadata and imaging

Allen Institute for Cell Science - Modeling 137 Dec 14, 2022
pikepdf is a Python library for reading and writing PDF files.

A Python library for reading and writing PDF, powered by qpdf

null 1.6k Jan 3, 2023
Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

Lawrence Livermore National Laboratory 1 Nov 12, 2021
MetPy is a collection of tools in Python for reading, visualizing and performing calculations with weather data.

MetPy MetPy is a collection of tools in Python for reading, visualizing and performing calculations with weather data. MetPy follows semantic versioni

Unidata 971 Dec 25, 2022
🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.

A hyper-fast, safe Python module to read and write JSON data. Works as a drop-in replacement for Python's built-in json module. This is alpha software

Matthias 479 Jan 1, 2023
Uproot is a library for reading and writing ROOT files in pure Python and NumPy.

Uproot is a library for reading and writing ROOT files in pure Python and NumPy. Unlike the standard C++ ROOT implementation, Uproot is only an I/O li

Scikit-HEP Project 164 Dec 31, 2022
A modern pure-Python library for reading PDF files

pdf A modern pure-Python library for reading PDF files. The goal is to have a modern interface to handle PDF files which is consistent with itself and

null 6 Apr 6, 2022
Python package that mirrors the original Nodejs ReplAPI-It.

Python-ReplAPI-It Python package that mirrors the original Nodejs ReplAPI-It. Contributing First fork the repo: $ git clone https://github.com/ReplAPI

The ReplAPI.it Project 10 Jun 5, 2022
pyWhisker is a Python equivalent of the original Whisker made by Elad Shamir and written in C#.

PyWhisker pyWhisker is a Python equivalent of the original Whisker made by Elad Shamir and written in C#. This tool allows users to manipulate the msD

Shutdown 325 Jan 8, 2023
pyForgeCert is a Python equivalent of the original ForgeCert written in C#.

pyForgeCert is a Python equivalent of the original ForgeCert written in C#.

Evi1cg 47 Oct 8, 2022
MusicBot is the original Discord music bot written for Python 3.5+, using the discord.py library

The original MusicBot for Discord (formerly SexualRhinoceros/MusicBot)

Just Some Bots 2.9k Jan 2, 2023