ZFS, in Python, without reading the original C.

Related tags

zfsp
Overview

ZFSp

What?

ZFS, in Python, without reading the original C.

What?!

That's right.

How?

Many hours spent staring at hexdumps, and asking friends to search the internet for explanations of various features.

Why?

Why not?

It seemed like it might be a fun project.

Installation

The Pipfile lists the dependencies; there aren't many.

Python 3.5+ is required, but 3.6 is not, as PyPy runs this code much, much faster (around 4x on the test suite) and didn't support 3.6 until recently.

pipenv install -e .

N.B.: -e gets you an "editable" install; changes to the source tree will affect the installed package's behavior.

Test Suite.

Running the test suite requires one-time access to a system with ZFS, to generate the test pools.

Run tests/fixtures.sh on such a system and make the resulting directory tests/fixtures (this is where it will end up if you run fixtures.sh directly from the tests directory).

The tests themselves can be run with py.test. Most of the tests pass, but there are some known failures. Feel free to try fixing them.

The tests are heavily parameterized and attempt to run all tests against all relevant pool variations.

Usage

zexplore is the main command line interface. It's reasonably well documented (I hope).

There's a subcommand for some limited FUSE support, which depends on fusepy (developed with 2.0.4). This is not installed by default in the Pipfile.

Example commands:

$ zexplore label -p tests/fixtures/feature_large_blocks
{b'errata': 0,
 b'features_for_read': {},
 b'guid': 6168868809305637343,
 b'hostid': 8323329,
 b'hostname': b'ubuntu-zesty',
 b'name': b'feature_large_blocks',
 b'pool_guid': 16637796155898928459,
 b'state': 1,
 b'top_guid': 6168868809305637343,
 b'txg': 16,
 b'vdev_children': 1,
 b'vdev_tree': {b'ashift': 9,
                b'asize': 62390272,
                b'create_txg': 4,
                b'guid': 6168868809305637343,
                b'id': 0,
                b'is_log': 0,
                b'metaslab_array': 33,
                b'metaslab_shift': 24,
                b'path': b'/vagrant/fixtures/feature_large_blocks',
                b'type': b'file'},
 b'version': 5000}

$ zexplore objset -p tests/fixtures/nested_datasets -P 1
{'config': 27,
 'creation_version': 5000,
 'deflate': 1,
 'feature_descriptions': 30,
 'features_for_read': 28,
 'features_for_write': 29,
 'free_bpobj': 11,
 'history': 32,
 'root_dataset': 2,
 'sync_bplist': 31}

$ zexplore ls -p tests/fixtures/nested_datasets /n1/n2/n3
x
y

$ zexplore cat -p tests/fixtures/nested_datasets /gzip-9/gzip-9 | head -n 5
A
a
aa
aal
aalii

Caveats

Lots of things won't work, including:

  • raidz with more than 4 disks will probably fail with an inscrutable error. ZFS uses a magic allocation function that's a non-linear black box. I haven't figured it out. raidz where the disks are larger than a few GB will probably also fail, but I haven't tested that at all.
  • Some new feature flags have come out since I last worked on this seriously, and those features are (obviously) not supported. That's things like (incomplete list):
    • SHA-512/256
    • Skein
    • spacemap_v2
  • Some feature flags that did exist when I was actively working on this are not implemented, either because I failed to figure out how they work, or because I didn't get around to trying.
  • pyndata is the struct description library used to read all the on-disk structures. I wrote it first, and it's bad. If I was doing this again today, I'd probably build something with dataclasses.
  • There's a Rust implementation of LZJB which is not currently included, but decompression is not the bottleneck.
Issues
  • dict object has no attribute 'bp'

    dict object has no attribute 'bp'

    I have been attempting to get this to work with a minimal zpool created clean. when I try to run the 'ls' command with the argument -p (pool file) I get an exception on line 26 in datasets.py objset_block = self.pool.read_block(self.dsl_dataset.bp) debug

    opened by cinterloper 2
  • click_explore.py syntax error

    click_explore.py syntax error

      File "/Users/alexander/Code/python/zfsp/click_explore.py", line 54
        except Exception, e:
                        ^
    SyntaxError: invalid syntax
    
    opened by ghost 1
  • This is too cool.

    This is too cool.

    Might want to add a few corporate-PHB-inspired types of c—kups and dependencies, you know, in the true spirit of Oracle ;) and make it so complicated that you can charge for support.

    Thanks, Larry

    [💯 joking. I mean, about it being an issue. It actually is too cool. But just leave it that way. Take that Oracle! And my name's not Larry ]

    opened by paulyc 0
  • Use feature detection instead of version detection

    Use feature detection instead of version detection

    opened by cclauss 0
  • Benchmarks

    Benchmarks

    How does it perform against a native version?

    Also, other python interpreters like PyPy can add benefit in performance?

    opened by corporatepiyush 0
  • Hint for RAIDZ2/3

    Hint for RAIDZ2/3

    RAIDZ2/3 uses pre-calculated Galois Field array (GF(8)) and Reed-Solomon error correction.

    pow2[256] = {
    	0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80,
    	0x1d, 0x3a, 0x74, 0xe8, 0xcd, 0x87, 0x13, 0x26,
    	0x4c, 0x98, 0x2d, 0x5a, 0xb4, 0x75, 0xea, 0xc9,
    	0x8f, 0x03, 0x06, 0x0c, 0x18, 0x30, 0x60, 0xc0,
    	0x9d, 0x27, 0x4e, 0x9c, 0x25, 0x4a, 0x94, 0x35,
    	0x6a, 0xd4, 0xb5, 0x77, 0xee, 0xc1, 0x9f, 0x23,
    	0x46, 0x8c, 0x05, 0x0a, 0x14, 0x28, 0x50, 0xa0,
    	0x5d, 0xba, 0x69, 0xd2, 0xb9, 0x6f, 0xde, 0xa1,
    	0x5f, 0xbe, 0x61, 0xc2, 0x99, 0x2f, 0x5e, 0xbc,
    	0x65, 0xca, 0x89, 0x0f, 0x1e, 0x3c, 0x78, 0xf0,
    	0xfd, 0xe7, 0xd3, 0xbb, 0x6b, 0xd6, 0xb1, 0x7f,
    	0xfe, 0xe1, 0xdf, 0xa3, 0x5b, 0xb6, 0x71, 0xe2,
    	0xd9, 0xaf, 0x43, 0x86, 0x11, 0x22, 0x44, 0x88,
    	0x0d, 0x1a, 0x34, 0x68, 0xd0, 0xbd, 0x67, 0xce,
    	0x81, 0x1f, 0x3e, 0x7c, 0xf8, 0xed, 0xc7, 0x93,
    	0x3b, 0x76, 0xec, 0xc5, 0x97, 0x33, 0x66, 0xcc,
    	0x85, 0x17, 0x2e, 0x5c, 0xb8, 0x6d, 0xda, 0xa9,
    	0x4f, 0x9e, 0x21, 0x42, 0x84, 0x15, 0x2a, 0x54,
    	0xa8, 0x4d, 0x9a, 0x29, 0x52, 0xa4, 0x55, 0xaa,
    	0x49, 0x92, 0x39, 0x72, 0xe4, 0xd5, 0xb7, 0x73,
    	0xe6, 0xd1, 0xbf, 0x63, 0xc6, 0x91, 0x3f, 0x7e,
    	0xfc, 0xe5, 0xd7, 0xb3, 0x7b, 0xf6, 0xf1, 0xff,
    	0xe3, 0xdb, 0xab, 0x4b, 0x96, 0x31, 0x62, 0xc4,
    	0x95, 0x37, 0x6e, 0xdc, 0xa5, 0x57, 0xae, 0x41,
    	0x82, 0x19, 0x32, 0x64, 0xc8, 0x8d, 0x07, 0x0e,
    	0x1c, 0x38, 0x70, 0xe0, 0xdd, 0xa7, 0x53, 0xa6,
    	0x51, 0xa2, 0x59, 0xb2, 0x79, 0xf2, 0xf9, 0xef,
    	0xc3, 0x9b, 0x2b, 0x56, 0xac, 0x45, 0x8a, 0x09,
    	0x12, 0x24, 0x48, 0x90, 0x3d, 0x7a, 0xf4, 0xf5,
    	0xf7, 0xf3, 0xfb, 0xeb, 0xcb, 0x8b, 0x0b, 0x16,
    	0x2c, 0x58, 0xb0, 0x7d, 0xfa, 0xe9, 0xcf, 0x83,
    	0x1b, 0x36, 0x6c, 0xd8, 0xad, 0x47, 0x8e, 0x01
    };
    
    log2[256] = {
    	0x00, 0x00, 0x01, 0x19, 0x02, 0x32, 0x1a, 0xc6,
    	0x03, 0xdf, 0x33, 0xee, 0x1b, 0x68, 0xc7, 0x4b,
    	0x04, 0x64, 0xe0, 0x0e, 0x34, 0x8d, 0xef, 0x81,
    	0x1c, 0xc1, 0x69, 0xf8, 0xc8, 0x08, 0x4c, 0x71,
    	0x05, 0x8a, 0x65, 0x2f, 0xe1, 0x24, 0x0f, 0x21,
    	0x35, 0x93, 0x8e, 0xda, 0xf0, 0x12, 0x82, 0x45,
    	0x1d, 0xb5, 0xc2, 0x7d, 0x6a, 0x27, 0xf9, 0xb9,
    	0xc9, 0x9a, 0x09, 0x78, 0x4d, 0xe4, 0x72, 0xa6,
    	0x06, 0xbf, 0x8b, 0x62, 0x66, 0xdd, 0x30, 0xfd,
    	0xe2, 0x98, 0x25, 0xb3, 0x10, 0x91, 0x22, 0x88,
    	0x36, 0xd0, 0x94, 0xce, 0x8f, 0x96, 0xdb, 0xbd,
    	0xf1, 0xd2, 0x13, 0x5c, 0x83, 0x38, 0x46, 0x40,
    	0x1e, 0x42, 0xb6, 0xa3, 0xc3, 0x48, 0x7e, 0x6e,
    	0x6b, 0x3a, 0x28, 0x54, 0xfa, 0x85, 0xba, 0x3d,
    	0xca, 0x5e, 0x9b, 0x9f, 0x0a, 0x15, 0x79, 0x2b,
    	0x4e, 0xd4, 0xe5, 0xac, 0x73, 0xf3, 0xa7, 0x57,
    	0x07, 0x70, 0xc0, 0xf7, 0x8c, 0x80, 0x63, 0x0d,
    	0x67, 0x4a, 0xde, 0xed, 0x31, 0xc5, 0xfe, 0x18,
    	0xe3, 0xa5, 0x99, 0x77, 0x26, 0xb8, 0xb4, 0x7c,
    	0x11, 0x44, 0x92, 0xd9, 0x23, 0x20, 0x89, 0x2e,
    	0x37, 0x3f, 0xd1, 0x5b, 0x95, 0xbc, 0xcf, 0xcd,
    	0x90, 0x87, 0x97, 0xb2, 0xdc, 0xfc, 0xbe, 0x61,
    	0xf2, 0x56, 0xd3, 0xab, 0x14, 0x2a, 0x5d, 0x9e,
    	0x84, 0x3c, 0x39, 0x53, 0x47, 0x6d, 0x41, 0xa2,
    	0x1f, 0x2d, 0x43, 0xd8, 0xb7, 0x7b, 0xa4, 0x76,
    	0xc4, 0x17, 0x49, 0xec, 0x7f, 0x0c, 0x6f, 0xf6,
    	0x6c, 0xa1, 0x3b, 0x52, 0x29, 0x9d, 0x55, 0xaa,
    	0xfb, 0x60, 0x86, 0xb1, 0xbb, 0xcc, 0x3e, 0x5a,
    	0xcb, 0x59, 0x5f, 0xb0, 0x9c, 0xa9, 0xa0, 0x51,
    	0x0b, 0xf5, 0x16, 0xeb, 0x7a, 0x75, 0x2c, 0xd7,
    	0x4f, 0xae, 0xd5, 0xe9, 0xe6, 0xe7, 0xad, 0xe8,
    	0x74, 0xd6, 0xf4, 0xea, 0xa8, 0x50, 0x58, 0xaf,
    };
    
    opened by raspi 0
Owner
Colin Valliant
Colin Valliant
TrueNAS CORE/Enterprise/SCALE Middleware Git Repository

TrueNAS CORE/Enterprise/SCALE main source repo Want to contribute or collaborate? Join our Slack instance. IMPORTANT NOTE: This is the master branch o

TrueNAS 1.8k Jul 24, 2021
Synchronize local directories with Tahoe-LAFS storage grids

Gridsync Gridsync aims to provide a cross-platform, graphical user interface for Tahoe-LAFS, the Least Authority File Store. It is intended to simplif

null 150 Jul 21, 2021
a full featured file system for online data storage

S3QL S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provi

null 777 Jul 17, 2021
A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting.

mycli A command line client for MySQL that can do auto-completion and syntax highlighting. HomePage: http://mycli.net Documentation: http://mycli.net/

dbcli 9.7k Jul 19, 2021
Continuous Archiving for Postgres

WAL-E Continuous archiving for Postgres WAL-E is a program designed to perform continuous archiving of PostgreSQL WAL files and base backups. To corre

null 3.3k Jul 23, 2021
Postgres CLI with autocompletion and syntax highlighting

A REPL for Postgres This is a postgres client that does auto-completion and syntax highlighting. Home Page: http://pgcli.com MySQL Equivalent: http://

dbcli 9.7k Jul 23, 2021
Barman - Backup and Recovery Manager for PostgreSQL

Barman, Backup and Recovery Manager for PostgreSQL Barman (Backup and Recovery Manager) is an open-source administration tool for disaster recovery of

EDB 1.1k Jul 23, 2021
Automatic SQL injection and database takeover tool

sqlmap sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of

sqlmapproject 20.6k Jul 24, 2021
The command-line tool that gives easy access to all of the capabilities of B2 Cloud Storage

B2 Command Line Tool The command-line tool that gives easy access to all of the capabilities of B2 Cloud Storage. This program provides command-line a

Backblaze 395 Jul 22, 2021
An open source multi-tool for exploring and publishing data

Datasette An open source multi-tool for exploring and publishing data Datasette is a tool for exploring and publishing data. It helps people take data

Simon Willison 5.3k Jul 25, 2021
A generic JSON document store with sharing and synchronisation capabilities.

Kinto Kinto is a minimalist JSON storage service with synchronisation and sharing abilities. Online documentation Tutorial Issue tracker Contributing

Kinto 4.1k Jul 16, 2021
The web end of seafile server.

Introduction Seahub is the web frontend for Seafile. Preparation Build and deploy Seafile server from source. See http://manual.seafile.com/build_seaf

null 440 Jul 25, 2021
Nerd-Storage is a simple web server for sharing files on the local network.

Nerd-Storage is a simple web server for sharing files on the local network. It supports the download of files and directories, the upload of multiple files at once, making a directory, updates and deletions.

ハル 62 Jul 16, 2021