ZFS, in Python, without reading the original C.

Colin Valliant

Last update: Oct 28, 2022

Related tags

Storage zfsp

Overview

ZFSp

What?

ZFS, in Python, without reading the original C.

What?!

That's right.

How?

Many hours spent staring at hexdumps, and asking friends to search the internet for explanations of various features.

Why?

Why not?

It seemed like it might be a fun project.

Installation

The Pipfile lists the dependencies; there aren't many.

Python 3.5+ is required, but 3.6 is not, as PyPy runs this code much, much faster (around 4x on the test suite) and didn't support 3.6 until recently.

pipenv install -e .

N.B.: -e gets you an "editable" install; changes to the source tree will affect the installed package's behavior.

Test Suite.

Running the test suite requires one-time access to a system with ZFS, to generate the test pools.

Run tests/fixtures.sh on such a system and make the resulting directory tests/fixtures (this is where it will end up if you run fixtures.sh directly from the tests directory).

The tests themselves can be run with py.test. Most of the tests pass, but there are some known failures. Feel free to try fixing them.

The tests are heavily parameterized and attempt to run all tests against all relevant pool variations.

Usage

zexplore is the main command line interface. It's reasonably well documented (I hope).

There's a subcommand for some limited FUSE support, which depends on fusepy (developed with 2.0.4). This is not installed by default in the Pipfile.

Example commands:

$ zexplore label -p tests/fixtures/feature_large_blocks
{b'errata': 0,
 b'features_for_read': {},
 b'guid': 6168868809305637343,
 b'hostid': 8323329,
 b'hostname': b'ubuntu-zesty',
 b'name': b'feature_large_blocks',
 b'pool_guid': 16637796155898928459,
 b'state': 1,
 b'top_guid': 6168868809305637343,
 b'txg': 16,
 b'vdev_children': 1,
 b'vdev_tree': {b'ashift': 9,
                b'asize': 62390272,
                b'create_txg': 4,
                b'guid': 6168868809305637343,
                b'id': 0,
                b'is_log': 0,
                b'metaslab_array': 33,
                b'metaslab_shift': 24,
                b'path': b'/vagrant/fixtures/feature_large_blocks',
                b'type': b'file'},
 b'version': 5000}

$ zexplore objset -p tests/fixtures/nested_datasets -P 1
{'config': 27,
 'creation_version': 5000,
 'deflate': 1,
 'feature_descriptions': 30,
 'features_for_read': 28,
 'features_for_write': 29,
 'free_bpobj': 11,
 'history': 32,
 'root_dataset': 2,
 'sync_bplist': 31}

$ zexplore ls -p tests/fixtures/nested_datasets /n1/n2/n3
x
y

$ zexplore cat -p tests/fixtures/nested_datasets /gzip-9/gzip-9 | head -n 5
A
a
aa
aal
aalii

Caveats

Lots of things won't work, including:

raidz with more than 4 disks will probably fail with an inscrutable error. ZFS uses a magic allocation function that's a non-linear black box. I haven't figured it out. raidz where the disks are larger than a few GB will probably also fail, but I haven't tested that at all.
Some new feature flags have come out since I last worked on this seriously, and those features are (obviously) not supported. That's things like (incomplete list):
- SHA-512/256
- Skein
- spacemap_v2
Some feature flags that did exist when I was actively working on this are not implemented, either because I failed to figure out how they work, or because I didn't get around to trying.
pyndata is the struct description library used to read all the on-disk structures. I wrote it first, and it's bad. If I was doing this again today, I'd probably build something with dataclasses.
There's a Rust implementation of LZJB which is not currently included, but decompression is not the bottleneck.

Comments

dict object has no attribute 'bp'

I have been attempting to get this to work with a minimal zpool created clean. when I try to run the 'ls' command with the argument -p (pool file) I get an exception on line 26 in datasets.py objset_block = self.pool.read_block(self.dsl_dataset.bp)

opened by cinterloper 2

click_explore.py syntax error

  File "/Users/alexander/Code/python/zfsp/click_explore.py", line 54
    except Exception, e:
                    ^
SyntaxError: invalid syntax

opened by ghost 1

Use feature detection instead of version detection

Python porting best practice use feature detection instead of version detection.

unicode was removed in Python 3 because all str default to Unicode utf-8.

opened by cclauss 0
This is too cool.

Might want to add a few corporate-PHB-inspired types of c—kups and dependencies, you know, in the true spirit of Oracle ;) and make it so complicated that you can charge for support.

Thanks, Larry

[💯 joking. I mean, about it being an issue. It actually is too cool. But just leave it that way. Take that Oracle! And my name's not Larry ]

opened by paulyc 0

Hint for RAIDZ2/3

RAIDZ2/3 uses pre-calculated Galois Field array (GF(8)) and Reed-Solomon error correction.

pow2[256] = {
	0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80,
	0x1d, 0x3a, 0x74, 0xe8, 0xcd, 0x87, 0x13, 0x26,
	0x4c, 0x98, 0x2d, 0x5a, 0xb4, 0x75, 0xea, 0xc9,
	0x8f, 0x03, 0x06, 0x0c, 0x18, 0x30, 0x60, 0xc0,
	0x9d, 0x27, 0x4e, 0x9c, 0x25, 0x4a, 0x94, 0x35,
	0x6a, 0xd4, 0xb5, 0x77, 0xee, 0xc1, 0x9f, 0x23,
	0x46, 0x8c, 0x05, 0x0a, 0x14, 0x28, 0x50, 0xa0,
	0x5d, 0xba, 0x69, 0xd2, 0xb9, 0x6f, 0xde, 0xa1,
	0x5f, 0xbe, 0x61, 0xc2, 0x99, 0x2f, 0x5e, 0xbc,
	0x65, 0xca, 0x89, 0x0f, 0x1e, 0x3c, 0x78, 0xf0,
	0xfd, 0xe7, 0xd3, 0xbb, 0x6b, 0xd6, 0xb1, 0x7f,
	0xfe, 0xe1, 0xdf, 0xa3, 0x5b, 0xb6, 0x71, 0xe2,
	0xd9, 0xaf, 0x43, 0x86, 0x11, 0x22, 0x44, 0x88,
	0x0d, 0x1a, 0x34, 0x68, 0xd0, 0xbd, 0x67, 0xce,
	0x81, 0x1f, 0x3e, 0x7c, 0xf8, 0xed, 0xc7, 0x93,
	0x3b, 0x76, 0xec, 0xc5, 0x97, 0x33, 0x66, 0xcc,
	0x85, 0x17, 0x2e, 0x5c, 0xb8, 0x6d, 0xda, 0xa9,
	0x4f, 0x9e, 0x21, 0x42, 0x84, 0x15, 0x2a, 0x54,
	0xa8, 0x4d, 0x9a, 0x29, 0x52, 0xa4, 0x55, 0xaa,
	0x49, 0x92, 0x39, 0x72, 0xe4, 0xd5, 0xb7, 0x73,
	0xe6, 0xd1, 0xbf, 0x63, 0xc6, 0x91, 0x3f, 0x7e,
	0xfc, 0xe5, 0xd7, 0xb3, 0x7b, 0xf6, 0xf1, 0xff,
	0xe3, 0xdb, 0xab, 0x4b, 0x96, 0x31, 0x62, 0xc4,
	0x95, 0x37, 0x6e, 0xdc, 0xa5, 0x57, 0xae, 0x41,
	0x82, 0x19, 0x32, 0x64, 0xc8, 0x8d, 0x07, 0x0e,
	0x1c, 0x38, 0x70, 0xe0, 0xdd, 0xa7, 0x53, 0xa6,
	0x51, 0xa2, 0x59, 0xb2, 0x79, 0xf2, 0xf9, 0xef,
	0xc3, 0x9b, 0x2b, 0x56, 0xac, 0x45, 0x8a, 0x09,
	0x12, 0x24, 0x48, 0x90, 0x3d, 0x7a, 0xf4, 0xf5,
	0xf7, 0xf3, 0xfb, 0xeb, 0xcb, 0x8b, 0x0b, 0x16,
	0x2c, 0x58, 0xb0, 0x7d, 0xfa, 0xe9, 0xcf, 0x83,
	0x1b, 0x36, 0x6c, 0xd8, 0xad, 0x47, 0x8e, 0x01
};

log2[256] = {
	0x00, 0x00, 0x01, 0x19, 0x02, 0x32, 0x1a, 0xc6,
	0x03, 0xdf, 0x33, 0xee, 0x1b, 0x68, 0xc7, 0x4b,
	0x04, 0x64, 0xe0, 0x0e, 0x34, 0x8d, 0xef, 0x81,
	0x1c, 0xc1, 0x69, 0xf8, 0xc8, 0x08, 0x4c, 0x71,
	0x05, 0x8a, 0x65, 0x2f, 0xe1, 0x24, 0x0f, 0x21,
	0x35, 0x93, 0x8e, 0xda, 0xf0, 0x12, 0x82, 0x45,
	0x1d, 0xb5, 0xc2, 0x7d, 0x6a, 0x27, 0xf9, 0xb9,
	0xc9, 0x9a, 0x09, 0x78, 0x4d, 0xe4, 0x72, 0xa6,
	0x06, 0xbf, 0x8b, 0x62, 0x66, 0xdd, 0x30, 0xfd,
	0xe2, 0x98, 0x25, 0xb3, 0x10, 0x91, 0x22, 0x88,
	0x36, 0xd0, 0x94, 0xce, 0x8f, 0x96, 0xdb, 0xbd,
	0xf1, 0xd2, 0x13, 0x5c, 0x83, 0x38, 0x46, 0x40,
	0x1e, 0x42, 0xb6, 0xa3, 0xc3, 0x48, 0x7e, 0x6e,
	0x6b, 0x3a, 0x28, 0x54, 0xfa, 0x85, 0xba, 0x3d,
	0xca, 0x5e, 0x9b, 0x9f, 0x0a, 0x15, 0x79, 0x2b,
	0x4e, 0xd4, 0xe5, 0xac, 0x73, 0xf3, 0xa7, 0x57,
	0x07, 0x70, 0xc0, 0xf7, 0x8c, 0x80, 0x63, 0x0d,
	0x67, 0x4a, 0xde, 0xed, 0x31, 0xc5, 0xfe, 0x18,
	0xe3, 0xa5, 0x99, 0x77, 0x26, 0xb8, 0xb4, 0x7c,
	0x11, 0x44, 0x92, 0xd9, 0x23, 0x20, 0x89, 0x2e,
	0x37, 0x3f, 0xd1, 0x5b, 0x95, 0xbc, 0xcf, 0xcd,
	0x90, 0x87, 0x97, 0xb2, 0xdc, 0xfc, 0xbe, 0x61,
	0xf2, 0x56, 0xd3, 0xab, 0x14, 0x2a, 0x5d, 0x9e,
	0x84, 0x3c, 0x39, 0x53, 0x47, 0x6d, 0x41, 0xa2,
	0x1f, 0x2d, 0x43, 0xd8, 0xb7, 0x7b, 0xa4, 0x76,
	0xc4, 0x17, 0x49, 0xec, 0x7f, 0x0c, 0x6f, 0xf6,
	0x6c, 0xa1, 0x3b, 0x52, 0x29, 0x9d, 0x55, 0xaa,
	0xfb, 0x60, 0x86, 0xb1, 0xbb, 0xcc, 0x3e, 0x5a,
	0xcb, 0x59, 0x5f, 0xb0, 0x9c, 0xa9, 0xa0, 0x51,
	0x0b, 0xf5, 0x16, 0xeb, 0x7a, 0x75, 0x2c, 0xd7,
	0x4f, 0xae, 0xd5, 0xe9, 0xe6, 0xe7, 0xad, 0xe8,
	0x74, 0xd6, 0xf4, 0xea, 0xa8, 0x50, 0x58, 0xaf,
};

opened by raspi 0