Python's Filesystem abstraction layer

Overview

PyFilesystem2

Python's Filesystem abstraction layer.

PyPI version PyPI Downloads Build Status Windows Build Status Coverage Status Codacy Badge Docs

Documentation

Introduction

Think of PyFilesystem's FS objects as the next logical step to Python's file objects. In the same way that file objects abstract a single file, FS objects abstract an entire filesystem.

Let's look at a simple piece of code as an example. The following function uses the PyFilesystem API to count the number of non-blank lines of Python code in a directory. It works recursively, so it will find .py files in all sub-directories.

def count_python_loc(fs):
    """Count non-blank lines of Python code."""
    count = 0
    for path in fs.walk.files(filter=['*.py']):
        with fs.open(path) as python_file:
            count += sum(1 for line in python_file if line.strip())
    return count

We can call count_python_loc as follows:

from fs import open_fs
projects_fs = open_fs('~/projects')
print(count_python_loc(projects_fs))

The line project_fs = open_fs('~/projects') opens an FS object that maps to the projects directory in your home folder. That object is used by count_python_loc when counting lines of code.

To count the lines of Python code in a zip file, we can make the following change:

projects_fs = open_fs('zip://projects.zip')

Or to count the Python lines on an FTP server:

projects_fs = open_fs('ftp://ftp.example.org/projects')

No changes to count_python_loc are necessary, because PyFileystem provides a simple consistent interface to anything that resembles a collection of files and directories. Essentially, it allows you to write code that is independent of where and how the files are physically stored.

Contrast that with a version that purely uses the standard library:

def count_py_loc(path):
    count = 0
    for root, dirs, files in os.walk(path):
        for name in files:
            if name.endswith('.py'):
                with open(os.path.join(root, name), 'rt') as python_file:
                    count += sum(1 for line in python_file if line.strip())
    return count

This version is similar to the PyFilesystem code above, but would only work with the OS filesystem. Any other filesystem would require an entirely different API, and you would likely have to re-implement the directory walking functionality of os.walk.

Credits

The following developers have contributed code and their time to this projects:

See CONTRIBUTORS.md for a full list of contributors.

PyFilesystem2 owes a massive debt of gratitude to the following developers who contributed code and ideas to the original version.

  • Ryan Kelly
  • Andrew Scheller
  • Ben Timby

Apologies if I missed anyone, feel free to prompt me if your name is missing here.

Support

If commercial support is required, please contact Will McGugan.

Comments
  • How to get a path pointing to io.BytesIO?

    How to get a path pointing to io.BytesIO?

    I am finding lot of librarys that use the filename for opening the file. However, in my Flask app I would like to get a filename that points to IO.BytesIO without the need o using temporal files

    opened by WaterKnight1998 32
  • fs.listdir and UnicodeError

    fs.listdir and UnicodeError

    On my system for whatever reason I have a file whith wrong encoding in the / dir. Always if I want
    for item in sorted(self.fs.listdir(_sel_dir)):

    I have to encapsulate this by an exception for UnicodeDecodeError. I would prefer to not crash but just ignore this file.

    (I am still looking on why that file anyway is there)

    bug accepted 
    opened by ReimarBauer 29
  • Add copy_if_newer parameter to functions in copy.py

    Add copy_if_newer parameter to functions in copy.py

    Added copy_if_newer parameter to all functions in copy.py. Files are copied only if destination files don't exist or sources are newer then destination files. Directories are always copied. In case time of modification for files cannot be determined files are safely copied.

    Moreover I added geturl method to osfs returning "file://" + self.getsyspath(path).

    opened by gpcimino 28
  • Bad namespacing, imports breaks setuptools

    Bad namespacing, imports breaks setuptools

    fs.dropboxfs is implemented using fs as a namespace package. This causes it to misbehave when importing from fs for several reasons:

    fs declares itself as a namespace package. According to the setuptools documentation here,

    A) only namespace packages are supposed to contain the __import__('pkg_resources')... B) they must not contain executable code aside from that import.

    We violate both of those rules, thus resulting in weird exceptions like this:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/dargueta/dev/goodrx-data-scripts/airflow_dags/grx/third_party/dropbox.py", line 6, in <module>
        from fs.dropboxfs import DropboxFS
      File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/__init__.py", line 1, in <module>
        from .dropboxfs import DropboxFile
      File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/dropboxfs.py", line 12, in <module>
        from fs.base import FS
      File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/base.py", line 24, in <module>
        from . import copy, errors, fsencode, iotools, move, tools, walk, wildcard
    ImportError: cannot import name fsencode
    
    cannot import name fsencode
    

    Changing that line to these two fixes the problem:

    from . import copy, errors, iotools, move, tools, walk, wildcard
    from ._fscompat import fsencode
    

    There's a similar issue in fs.test:

    # Broken
    from fs import ResourceType, Seek
    
    # Works
    from fs.enums import ResourceType, Seek
    

    It appears these are the only two places in the repo we do this. Fixing these imports gets us a separate weird error in code using fs.dropboxfs but that may be unrelated.

    opened by dargueta 23
  • Update tests, and fix discovered bugs

    Update tests, and fix discovered bugs

    Hi Will, I updated the tests in fs.test with some missing checks, and discovered some new bugs I will fix gradually in this PR.

    New tests

    1. FS.copy, FS.move, FS.copydir and FS.movedir should raise ResourceNotFound when the source is nowhere to be found
    2. FS.copy and FS.move should probably raise FileExpected when the source is a directory (not listed in the documentation, but makes sense I guess ?...)
    3. FS.movedir and FS.copydir should raise DirectoryExpected when the source is a file
    4. If a filesystem is advertising itself as case insensitive, check that's true
    5. If a filesystem is advertising itself as supporting unicode paths, check that's true
    6. Check that file-like objects returned by FS.openbin:
      • have a seek method that returns the new absolute position when called
      • have a write method that returns the number of bytes (in binary mode) / characters (in text mode) written
      • have a truncate method that returns the new length of the file

    Discovered bugs

    • ~FTPFile.seek, FTPFile.truncate and FTPFile.write do not return anything~ fixed
    • ~FTPFS crashes on unicode paths (/földér)~ fixed, see below
    • ~MemoryFile.write and MemoryFile.truncate do not return anything~ fixed
    • ~many filesystem's move and copy do not raise FileExpected when given a directory~ fixed by fixing FS.copy and FS.move

    About FTPFS

    FTP servers supporting UTF-8 encoded paths display the feature UTF8. But the ftplib.FTP objects require to set the encoding before connecting to the server, or else the server and the client encoding will not be synchronised (client will send utf-8 paths but server will treat them as latin-1).

    To fix this, the encoding is checked when accessing the FTPFS.ftp property for the first time. A mock connection is established to check for the UTF8 feature ; then, a proper connection is established, with utf-8 as the encoding if available, and latin-1 otherwise. Then, any new ftp connection will use the previous encoding, so the overhead is limited.

    About test_ftpfs

    I rewrote the class so that a new server is created once by class instead of before each test, which was a lot longer for nothing. I also made use of pyftpdlib.test.FTPd instead of externally calling the file and then task-killing the thread, which was a lot hackier.

    I still think there is an issue with the timeout of FTPFS objects, since the test_connection_error tests take a really long time, longer than what they should take given the timeout.

    opened by althonos 22
  • backslash in windows path definition - need a forward slash

    backslash in windows path definition - need a forward slash

    Hi I ran in a problem of \ used in pathes. This was in the past different. Also the documentations shows:

    This is broadly similar to the standard os.path module but works with paths in the canonical format expected by all FS objects (that is, separated by forward slashes and with an optional leading slash).

    from fs.tempfs import TempFS
    ROOT_FS = TempFS()
    ROOT_DIR = ROOT_FS.root_path
    
    print(ROOT_DIR)
    'C:\\Users\\Local...'
    
    
    question 
    opened by ReimarBauer 21
  • Fix units in `fs.filesize`

    Fix units in `fs.filesize`

    According to Wikipedia, the binary filesize units should be KiB, MiB, ..., and the decimal units should be kB, MB, ...

    Previously, traditional would use the SI prefixes, and decimal would display bits instead of bytes.

    Also, I'm not sure about this, but once again functions should probably be raising TypeError when given an argument that cannot be coerced to a numerical value.

    opened by althonos 19
  • Migrate tests to Pytest

    Migrate tests to Pytest

    Closes #327

    Type of changes

    • [x] Bug fix
    • [x] New feature
    • [ ] Documentation / docstrings
    • [x] Tests
    • [ ] Other

    Checklist

    • [x] I've run the latest black with default args on new code.
    • [x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
    • [ ] I've added tests for new code.
    • [x] I accept that @willmcgugan may be pedantic in the code review.

    Description

    • Migrate test suite from nose to pytest
    • Randomized execution order of tests to expose interdependencies
    • Overhauled Travis config to use tox environments instead of passing them in multiple places

    Bugs / Cleanup

    • Fix a few import mock bugs that imported mock on Python 3 instead of the native module
    • Remove unused imports in a few test files
    • Remove requirements.txt, as it's out of date and no longer needed.
    • Fixed abstract class imports from collections which will break on Python 3.8
    opened by dargueta 18
  • Is there a way to check whether two paths (or filesystems) are referencing the same file?

    Is there a way to check whether two paths (or filesystems) are referencing the same file?

    I'd like to know if there is a way to check whether two pathes reference the same file. I found that not even the filesystems can be compared:

    >>> import fs
    >>> fs1 = fs.open_fs("~/Desktop")
    >>> fs2 = fs.open_fs("~/Desktop")
    >>> fs1 == fs2
    False
    

    At the moment I'm resorting to this, which is not nice and complete at all.

    from typing import NamedTuple
    
    class Resource(NamedTuple):
        fs: FS
        path: str
    
    def is_same(a: Resource, b: Resource):
        from fs.osfs import OSFS
        from fs.zipfs import ZipFS
        from fs.tarfs import TarFS
    
        if isinstance(a.fs, OSFS) and isinstance(b.fs, OSFS):
            return a.fs.getospath(a.path) == b.fs.getospath(b.path)
        elif isinstance(a.fs, ZipFS) and isinstance(b.fs, ZipFS):
            return a.fs._file == b.fs._file and a.path == b.path
        elif isinstance(a.fs, TarFS) and isinstance(b.fs, TarFS):
            return a.fs._file == b.fs._file and a.path == b.path
        return False
    

    Is there a nicer way to check this? Would you be interested in a PR that fixes the __eq__ method for some filesystems?

    question 
    opened by tfeldmann 17
  • splitext removes a forward slash from url

    splitext removes a forward slash from url

    Why does fs.path.splitext(url) remove one of the forward slashes?

    url = 's3://simonm3/_testdata/file1.pkl'
    fs.path.splitext(url), os.path.splitext(url)
    

    (('s3:/simonm3/_testdata/file1', '.pkl'), ('s3://simonm3/_testdata/file1', '.pkl'))

    Also what is the purpose of the second forward slash? The standard format of a url is that // denotes the start what urllib.urlparse calls "netloc" and includes username etc.. The pyfilesystem url instead uses // as part of the filesystem. Why not have "s3:/somefolder/somefile" rather than "s3://somefolder/somefile" ?

    https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax

    opened by simonm3 17
  • Implement TarFS.geturl and ZipFS.geturl and Fix #329, #333, #340

    Implement TarFS.geturl and ZipFS.geturl and Fix #329, #333, #340

    Type of changes

    • [x] Bug fix
    • [x] New feature
    • [ ] Documentation / docstrings
    • [x] Tests
    • [ ] Other

    Checklist

    • [x] I've run the latest black with default args on new code.
    • [x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
    • [x] I've added tests for new code.
    • [x] I accept that @willmcgugan may be pedantic in the code review.

    Description

    Fix #329, #333, #340

    And deliver geturl for ZipFS and TarFS, and my use case is to get url for a file inside a zip, deposit the url, and read it later.

            with file_system.open_fs('zip://test.zip') as fs_handle:
                    template_file_exists = fs_handle.exists(
                        template
                    ) and fs_handle.isfile(template)
    
                    if template_file_exists:
                        return fs_handle.geturl(template)
    

    The existing code in ZipFS and TarFS did not imagine an download url is really needed for a file inside a zip or tar. Hence, I would like to present my use case and get it supported.

    opened by chfw 17
  • Fix file move time preservation (#558)

    Fix file move time preservation (#558)

    Type of changes

    • Bug fix

    Checklist

    • [x] I've run the latest black with default args on new code.
    • [x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
    • [x] I've added tests for new code.
    • [ ] I've updated the Documentation.
    • [x] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

    Description

    This PR fixes a bug in the methods FS.move and MemoryFS.move, where preserve_time=True resulted in an ResourceNotFound error (see #558).

    The new functions to separate copy.copy_modified_time are currently residing in copy.py. One could argue that reading and updating a files meta info are more suitable for info.py, but this would introduce further changes. Additionally, we could modify the original one to utilize both new functions. This would avoid code duplication but introduce further function calls. Opinions on this matter and reviews are welcomed!

    Lastly, I would update the documentation for the new functions accordingly.

    opened by mj0nez 1
  • ResourceNotFound - moving files on the same filesystem with preserve_time=True

    ResourceNotFound - moving files on the same filesystem with preserve_time=True

    Hi, while experimenting with the FS lib, I’ve encountered an unexpected ResourceNotFound error. The error is raised if the param preserve_time=True and we try to move a file on the same OS or in-memory filesystem.

    This block represents a reproduceable example for a MemoryFS:

        mem_fs = memoryfs.MemoryFS()
    
        mem_fs.makedir("foo")
        mem_fs.writetext("foo/README.md", "Tetris clone")
    
        move_file(
            src_fs=mem_fs,
            src_path="foo/README.md",
            dst_fs=mem_fs,
            dst_path="foo/README_2.md",
            preserve_time=True,
        )
    

    The raised error message:

    ./tests/test_resource_error.py::test_move_between_same_fs_mem Failed: [undefined]fs.errors.ResourceNotFound: resource 'foo/README.md' not found
    def test_move_between_same_fs_mem():
            mem_fs = memoryfs.MemoryFS()
        
            mem_fs.makedir("foo")
            mem_fs.writetext("foo/README.md", "Tetris clone")
        
    >       move_file(
                src_fs=mem_fs,
                src_path="foo/README.md",
                dst_fs=mem_fs,
                dst_path="foo/README_2.md",
                preserve_time=True,
            )
    
    tests\test_resource_error.py:31: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    fs\move.py:68: in move_file
        _src_fs.move(
    fs\memoryfs.py:478: in move
        copy_modified_time(self, src_path, self, dst_path)
    fs\copy.py:532: in copy_modified_time
        src_meta = _src_fs.getinfo(src_path, namespaces)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = MemoryFS(), path = 'foo/README.md', namespaces = ('details',)
    
        def getinfo(self, path, namespaces=None):
            # type: (Text, Optional[Collection[Text]]) -> Info
            _path = self.validatepath(path)
            dir_entry = self._get_dir_entry(_path)
            if dir_entry is None:
    >           raise errors.ResourceNotFound(path)
    E           fs.errors.ResourceNotFound: resource 'foo/README.md' not found
    
    fs\memoryfs.py:394: ResourceNotFound
    

    I have modified the test case test_move_file_same_fs and found similar errors for OS- and Temp-filesystems. For my understanding, the current implementation wants to copy the file’s metadata, after moving it or renaming its path. Therefore, the copy_modified_time function encounters a missing resource under these conditions:

    • moving a file on the same filesystem
    • preserve_time=True

    AND either

    1. copy_modified_time is called after moving the file, see memoryfs.py

    2. or FS.move of base.py is used and the filesystem supports renaming, see base.py

    Is this a limitation of my system (Win11 x64)? I could not find similar issues or information in the docs.

    If this is a not yet supported edge case, I would suggest that we read the file’s meta data first, then move it and update the meta info afterwards.

    Any review and comments are welcome! Regards

    opened by mj0nez 2
  • #556 Fallback to alternate mtime correctly

    #556 Fallback to alternate mtime correctly

    Type of changes

    • Bug fix

    Checklist

    • [X] I've run the latest black with default args on new code.
    • [x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
    • [x] I've added tests for new code.
    • [X] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

    Description

    This addresses an issues where write_zip() assumed that the details namespace was always available. The stat namespace will be checked first, followed by details, and finally the current time will be used if st_mtime is still None.

    I'm willing to author a test against this, but I'm only aware of this being an issue with the s3 filesystem (and only when it encounters a directory) and I'm not certain how best to test that here.

    https://github.com/PyFilesystem/pyfilesystem2/issues/556

    bug 
    opened by james-emerton 2
  • Unable to use compress.write_zip() with s3fs source

    Unable to use compress.write_zip() with s3fs source

    I believe this ticket should be for this package, but the issue is caused by implementation details of the fs-s3fs package. Attempting to create an archive of files from s3 using write_zip() fails with the following error:

    Traceback (most recent call last):
      File "/Users/james/dev/spire/mount-s3-archive/test.py", line 7, in <module>
        write_zip(source, sys.argv[2])
      File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/compress.py", line 78, in write_zip
        mt = info.modified or datetime.utcnow()
      File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/info.py", line 322, in modified
        self._require_namespace("details")
      File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/info.py", line 128, in _require_namespace
        raise MissingInfoNamespace(namespace)
    fs.errors.MissingInfoNamespace: namespace 'details' is required for this attribute
    

    Code:

    import sys
    
    import fs
    from fs.compress import write_zip
    
    source = fs.open_fs(sys.argv[1])
    write_zip(source, sys.argv[2])
    

    Executing the above code as test.py s3://some-bucket/ test.zip will fail with the above error while a local source directory will compress nicely.

    The issue seems to be that S3FS directory entries don't expose a details namespace. It's unclear if that's correct or not, but the code in write_zip() seems to assume that details is always available (but makes allowance for Info.modified to be a falsy value).

    opened by james-emerton 0
  • Invalid characters in the path

    Invalid characters in the path

    I apologize if my question sounds silly but I just started using this library and can't find the answer to my question. I'm trying to copy a file from my disk system to a memory file system:

    mem_fs = fs.open_fs('mem://')
    zip_path = 'C:\\project\\test.zip"
    fs.copy.copy_file('/', zip_path.format(os.getcwd()), mem_fs, 'test.zip')
    

    I have the following exception: fs.errors.InvalidCharsInPath because the path contains ":" and "\". If I replace all these "invalid" characters with "/" the resource can't be found. The OS: Win7. How can this issue be solved? And what is the cross-platform solution (I will run my app on Linux too)? Thanks!

    opened by kwantdev 11
  • Drop usage of `pkg_resources`

    Drop usage of `pkg_resources`

    Type of changes

    • Bug fix

    Checklist

    • [ ] I've run the latest black with default args on new code.
    • [ ] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
    • [x] I've added tests for new code.
    • [x] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

    Description

    Drop pkg_resources in favor of the importlib.metadata API.

    Closes https://github.com/PyFilesystem/pyfilesystem2/issues/356

    opened by edgarrmondragon 1
Releases(2.4.16)
  • 2.4.16(May 2, 2022)

    Changed

    • Make fs.zipfs._ZipExtFile use the seeking mechanism implemented in the Python standard library in Python version 3.7 and later (#527).
    • Mark fs.zipfs.ReadZipFS as a case-sensitive filesystem (#527).
    • Optimized moving files between filesystems with syspaths. (#523).
    • Fixed fs.move.move_file to clean up the copy on the destination in case of errors.
    • fs.opener.manage_fs with writeable=True will now raise a ResourceReadOnly exception if the managed filesystem is not writeable.
    • Marked filesystems wrapped with fs.wrap.WrapReadOnly as read-only.
    Source code(tar.gz)
    Source code(zip)
  • 2.4.15(Feb 7, 2022)

    Changed

    • Support more lenient usernames and group names in FTP servers (#507). Closes #506.

    Fixed

    • Fixed MemoryFS.move and MemoryFS.movedir not updating the name of moved resources, causing MemoryFS.scandir to use the old name. (#510). Closes #509.
    • Make WrapFS.move and WrapFS.movedir use the delegate FS methods instead of fs.move functions, which was causing optimized implementation of movedir to be always skipped. (#511).
    Source code(tar.gz)
    Source code(zip)
  • v2.4.14(Nov 16, 2021)

    Added

    • Added fs.copy.copy_file_if, fs.copy.copy_dir_if, and fs.copy.copy_fs_if. Closes #458.
    • Added fs.base.FS.getmodified.

    Changed

    • FTP servers that do not support the MLST command now try to use the MDTM command to retrieve the last modification timestamp of a resource. Closes #456.

    Fixed

    • Fixed performance bugs in fs.copy.copy_dir_if_newer. Test cases were adapted to catch those bugs in the future.
    • Fixed precision bug for timestamps in fs.OSFS.setinfo.
    Source code(tar.gz)
    Source code(zip)
  • v2.4.13(Mar 27, 2021)

    Added

    • Added FTP over TLS (FTPS) support to FTPFS. Closes #437, #449.
    • PathError now supports wrapping an exception using the exc argument. Closes #453.
    • Better documentation of the writable parameter of fs.open_fs, and hint about using fs.wrap.read_only when a read-only filesystem is required. Closes #441.

    Changed

    • Make FS.upload explicit about the expected error when the parent directory of the destination does not exist. Closes #445.
    • Migrate continuous integration from Travis-CI to GitHub Actions and introduce several linters again in the build steps (#448). Closes #446.
    • Stop requiring pytest to run tests, allowing any test runner supporting unittest-style test suites.
    • FSTestCases now builds the large data required for upload and download tests only once in order to reduce the total testing time.
    • MemoryFS.move and MemoryFS.movedir will now avoid copying data. Closes #452.
    • FS.removetree("/") behaviour has been standardized in all filesystems, and is expected to clear the contents of the root folder without deleting it. Closes #471.
    • FS.getbasic is now deprecated, as it is redundant with FS.getinfo, and FS.getinfo is now explicitly expected to return the basic info namespace unconditionally. Closes #469.

    Fixed

    • Make FTPFile, MemoryFile and RawWrapper accept array.array arguments for the write and writelines methods, as expected by their base class io.RawIOBase.
    • Various documentation issues, including MemoryFS docstring not rendering properly.
    • Avoid creating a new connection on every call of FTPFS.upload. Closes #455.
    • WrapReadOnly.removetree not raising a ResourceReadOnly when called. Closes #468.
    • WrapCachedDir.isdir and WrapCachedDir.isfile raising a ResourceNotFound error on non-existing path (#470).
    • FTPFS not listing certain entries with sticky/SUID/SGID permissions set by Linux server (#473). Closes #451.
    • scandir iterator not being closed explicitly in OSFS.scandir, occasionally causing a ResourceWarning to be thrown. Closes #311.
    • Incomplete type annotations for the temp_fs parameter of WriteTarFS and WriteZipFS. Closes #410.
    Source code(tar.gz)
    Source code(zip)
  • v2.4.12(Jan 14, 2021)

    Added

    • Missing mode attribute to _MemoryFile objects returned by MemoryFS.openbin.
    • Missing readinto method for MemoryFS and FTPFS file objects. Closes #380.
    • Added compatibility if a Windows FTP server returns file information to the LIST command with 24-hour times. Closes #438.

    Changed

    • Start testing on PyPy. Due to #342 we have to treat PyPy builds specially and allow them to fail, but at least we'll be able to see if we break something aside from known issues with FTP tests.
    • Include docs in source distributions as well as the whole tests folder, ensuring conftest.py is present, fixes #364.
    • Stop patching copy with Python 3.8+ because it already uses sendfile.

    Fixed

    • Fixed crash when CPython's -OO flag is used
    • Fixed error when parsing timestamps from a FTP directory served from a WindowsNT FTP Server, fixes #395.
    • Fixed documentation of Mode.to_platform_bin. Closes #382.
    • Fixed the code example in the "Testing Filesystems" section of the "Implementing Filesystems" guide. Closes #407.
    • Fixed FTPFS.openbin not implicitly opening files in binary mode like expected from openbin. Closes #406.
    Source code(tar.gz)
    Source code(zip)
  • v2.4.11(Sep 7, 2019)

    [2.4.11] - 2019-09-07

    Added

    • Added geturl for TarFS and ZipFS for 'fs' purpose. NoURL for 'download' purpose.
    • Added helpful root path in CreateFailed exception #340
    • Added Python 3.8 support

    Fixed

    • Fixed tests leaving tmp files
    • Fixed typing issues
    • Fixed link namespace returning bytes
    • Fixed broken FSURL in windows #329
    • Fixed hidden exception at fs.close() when opening an absent zip/tar file URL #333
    • Fixed abstract class import from collections which would break on Python 3.8
    • Fixed incorrect imports of mock on Python 3
    • Removed some unused imports and unused requirements.txt file
    • Added mypy checks to Travis. Closes #332.
    • Fixed missing errno.ENOTSUP on PyPy. Closes #338.
    • Fixed bug in a decorator that would trigger an AttributeError when a class was created that implemented a deprecated method and had no docstring of its own.

    Changed

    • Entire test suite has been migrated to pytest. Closes #327.
    • Style checking is now enforced using flake8; this involved some code cleanup such as removing unused imports.
    Source code(tar.gz)
    Source code(zip)
  • v2.4.10(Jul 29, 2019)

  • v2.4.9(Jul 28, 2019)

    [2.4.9] - 2019-07-28

    Fixed

    • Restored fs.path import
    • Fixed potential race condition in makedirs. Fixes #310
    • Added missing methods to WrapFS. Fixed #294

    Changed

    • MemFS now immediately releases all memory it holds when close() is called, rather than when it gets garbage collected. Closes issue #308.
    • FTPFS now translates EOFError into RemoteConnectionError. Closes #292
    • Added automatic close for filesystems that go out of scope. Fixes #298
    Source code(tar.gz)
    Source code(zip)
  • v2.4.8(Jun 12, 2019)

  • v2.4.7(Jun 8, 2019)

  • v2.4.6(Jun 8, 2019)

    [2.4.6] - 2019-06-08

    Added

    • Implemented geturl in FTPFS @zmej-serow

    Fixed

    • Fixed FTP test suite when time is not UTC-0 @mrg0029
    • Fixed issues with paths in tarfs https://github.com/PyFilesystem/pyfilesystem2/issues/284

    Changed

    • Dropped Python3.3 support
    Source code(tar.gz)
    Source code(zip)
  • v2.4.5(May 5, 2019)

    [2.4.5] - 2019-05-05

    Fixed

    • Restored deprecated setfile method with deprecation warning to change to writefile
    • Fixed exception when a tarfile contains a path called '.' https://github.com/PyFilesystem/pyfilesystem2/issues/275
    • Made TarFS directory loading lazy

    Changed

    • Detect case insensitivity using by writing temp file
    Source code(tar.gz)
    Source code(zip)
  • v2.4.4(Feb 23, 2019)

  • v2.4.3(Feb 23, 2019)

  • v2.4.2(Feb 22, 2019)

  • v2.4.1(Feb 20, 2019)

  • v2.4.0(Feb 15, 2019)

  • v2.3.1(Feb 10, 2019)

  • v2.3.0(Jan 30, 2019)

  • v2.2.1(Jan 6, 2019)

  • v2.2.0(Jan 2, 2019)

    [2.2.0] - 2019-01-01

    A few methods have been renamed for greater clarity (but functionality remains the same).

    The old methods are now aliases and will continue to work, but will issue a deprecation warning via the warnings module. Please update your code accordingly.

    • getbytes -> readbytes
    • getfile -> download
    • gettext -> readtext
    • setbytes -> writebytes
    • setbinfile -> upload
    • settext -> writetext

    Changed

    • Changed default chunk size in copy_file_data to 1MB
    • Added chunk_size and options to FS.upload
    Source code(tar.gz)
    Source code(zip)
  • v2.1.3(Dec 24, 2018)

    [2.1.3] - 2018-12-24

    Fixed

    • Incomplete FTPFile.write when using workers @geoffjukes
    • Fixed AppFS not creating directory

    Added

    • Added load_extern switch to opener, fixes #228 @althanos
    Source code(tar.gz)
    Source code(zip)
  • v2.1.2(Nov 10, 2018)

    [2.1.2] - 20180-11-10

    Added

    • Support for Windows NT FTP servers @sspross

    Fixed

    • Root dir of MemoryFS accessible as a file
    • Packaging issues @televi
    • Deprecation warning re collections.Mapping
    Source code(tar.gz)
    Source code(zip)
  • v2.1.1(Oct 3, 2018)

    [2.1.1] - 2018-10-03

    Added

    • Added PEP 561 py.typed files
    • Use sendfile for faster copies @althonos
    • Atomic exclusive mode in Py2.7 @sqwishy

    Fixed

    • Fixed lstat @kamomil
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Aug 12, 2018)

  • v2.0.27(Aug 5, 2018)

  • v2.0.26(Jul 26, 2018)

    Fixed

    • fs.copy and fs.move disable workers if not thread-safe
    • fs.match detects case insensitivity
    • Open in exclusive mode is atomic (@squishy)
    • Exceptions can be pickleabe (@Spacerat)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.25(Jul 20, 2018)

  • v2.0.24(Jun 28, 2018)

  • v2.0.23(May 19, 2018)

    [2.0.23] - 2018-05-02

    • Fix for Markdown on PyPi, no code changes

    [2.0.22] - 2018-05-02

    Fixed

    • Handling of broken unicode on Python2.7

    Added

    • Added fs.getospath
    Source code(tar.gz)
    Source code(zip)
RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem

RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem. These files are exposed either in their original format, or as PDF files that contain your annotations. This lets you manage files in the reMarkable Cloud using the same tools you use on your local system.

Robert Schroll 82 Nov 24, 2022
Python virtual filesystem for SQLite to read from and write to S3

Python virtual filesystem for SQLite to read from and write to S3

Department for International Trade 70 Jan 4, 2023
FUSE filesystem Python scripts for Nintendo console files

ninfs (formerly fuse-3ds) is a FUSE program to extract data from Nintendo game consoles. It works by presenting a virtual filesystem with the contents of your games, NAND, or SD card contents, and you can browse and copy out just the files that you need.

Ian Burgwin 343 Jan 2, 2023
Quick and dirty FAT12 filesystem to ZIP file converter

Quick and Dirty FAT12 Filesystem Converter This is a really crappy Python script I wrote to convert a semi-compatible FAT12 filesystem from my HP150's

Tube Time 2 Feb 12, 2022
Network Automation and Programmability Abstraction Layer with Multivendor support

NAPALM NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functio

napalm-automation 2k Dec 31, 2022
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

null 440 Nov 13, 2022
An abstraction layer for mathematical optimization solvers.

MathOptInterface Documentation Build Status Social An abstraction layer for mathematical optimization solvers. Replaces MathProgBase. Citing MathOptIn

JuMP-dev 284 Jan 4, 2023
Foundation Auth Proxy is an abstraction on Foundations' authentication layer and is used to authenticate requests to Atlas's REST API.

foundations-auth-proxy Setup By default the server runs on http://0.0.0.0:5558. This can be changed via the arguments. Arguments: '-H' or '--host': ho

Dessa - Open Source 2 Jul 3, 2020
Kglab - an abstraction layer in Python for building knowledge graphs

Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, pyarrow, etc.

derwen.ai 466 Jan 9, 2023
Python library and shell utilities to monitor filesystem events.

Watchdog Python API and shell utilities to monitor file system events. Works on 3.6+. If you want to use Python 2.6, you should stick with watchdog <

Yesudeep Mangalapilly 5.6k Jan 4, 2023
RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem

RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem. These files are exposed either in their original format, or as PDF files that contain your annotations. This lets you manage files in the reMarkable Cloud using the same tools you use on your local system.

Robert Schroll 82 Nov 24, 2022
The Tahoe-LAFS decentralized secure filesystem.

Free and Open decentralized data store Tahoe-LAFS (Tahoe Least-Authority File Store) is the first free software / open-source storage technology that

Tahoe-LAFS 1.2k Jan 1, 2023
Python virtual filesystem for SQLite to read from and write to S3

Python virtual filesystem for SQLite to read from and write to S3

Department for International Trade 70 Jan 4, 2023
FUSE filesystem Python scripts for Nintendo console files

ninfs (formerly fuse-3ds) is a FUSE program to extract data from Nintendo game consoles. It works by presenting a virtual filesystem with the contents of your games, NAND, or SD card contents, and you can browse and copy out just the files that you need.

Ian Burgwin 343 Jan 2, 2023
A webdav demo using a virtual filesystem that serves a random status of whether a cat in a box is dead or alive.

A webdav demo using a virtual filesystem that serves a random status of whether a cat in a box is dead or alive.

Marshall Conover 2 Jan 12, 2022
Quick and dirty FAT12 filesystem to ZIP file converter

Quick and Dirty FAT12 Filesystem Converter This is a really crappy Python script I wrote to convert a semi-compatible FAT12 filesystem from my HP150's

Tube Time 2 Feb 12, 2022
Python utility to generate filesystem content for Obsidian.

Security Vault Generator Quickly parse, format, and output common frameworks/content for Obsidian.md. There is a strong focus on MITRE ATT&CK because

Justin Angel 73 Dec 2, 2022
This repo is a PyTorch implementation for Paper "Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds"

Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds This repository is a PyTorch implementation for paper: Uns

Kaizhi Yang 42 Dec 9, 2022
Abstraction of a Unit, includes convertions and basic operations.

Units Abstraction of a Unit, includes convertions and basic operations. ------ EXAMPLE : Free Fall (No air resistance) ------- from units_test import

null 1 Dec 23, 2021
This library provides an abstraction to perform Model Versioning using Weight & Biases.

Description This library provides an abstraction to perform Model Versioning using Weight & Biases. Features Version a new trained model Promote a mod

Hector Lopez Almazan 2 Jan 28, 2022