OnedataFS is a PyFilesystem interface to Onedata virtual file system

Overview

OnedataFS

OnedataFS is a PyFilesystem interface to Onedata virtual file system.

As a PyFilesystem concrete class, OnedataFS allows you to work with Onedata in the same way as any other supported filesystem.

Installing

You can install OnedataFS from pip as follows:

pip install fs-onedatafs

Opening a OnedataFS

Open an OnedataFS by explicitly using the constructor:

from fs.onedatafs import OnedataFS
onedata_provider_host = "..."
onedata_access_token = "..."
odfs = OnedataFS(onedata_provider_host, onedata_access_token)

Or with a FS URL:

  from fs import open_fs
  odfs = open_fs('onedatafs://HOST?token=...')

Extended attributes

Onedata FS supports in addition to standard PyFilesystem API operations on metadata via POSIX compatible extended attributes API.

Documentation

You might also like...
File-manager - A basic file manager, written in Python

File Manager A basic file manager, written in Python. Installation Install Pytho

Two scripts help you to convert csv file to md file by template

Two scripts help you to convert csv file to md file by template. One help you generate multiple md files with different filenames from the first colume of csv file. Another can generate one md file with several blocks.

A simple Python code that takes input from a csv file and makes it into a vcf file.

Contacts-Maker A simple Python code that takes input from a csv file and makes it into a vcf file. Imagine a college or a large community where each y

Object-oriented file system path manipulation

path (aka path pie, formerly path.py) implements path objects as first-class entities, allowing common operations on files to be invoked on those path

Object-oriented file system path manipulation

path (aka path pie, formerly path.py) implements path objects as first-class entities, allowing common operations on files to be invoked on those path

Import Python modules from any file system path

pathimp Import Python modules from any file system path. Installation pip3 install pathimp Usage import pathimp

Python virtual filesystem for SQLite to read from and write to S3

Python virtual filesystem for SQLite to read from and write to S3

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

An object-oriented approach to Python file/directory operations.

Unipath An object-oriented approach to file/directory operations Version: 1.1 Home page: https://github.com/mikeorr/Unipath Docs: https://github.com/m

Comments
  • Installation issues from anaconda

    Installation issues from anaconda

    Hello i'm trying to install the fs-onedatafs lib using anaconda, to install it inside a jupyter notebook

    this is my dockerfile

    FROM jupyter/minimal-notebook:ubuntu-18.04
    
    USER $NB_UID
    
    RUN conda install mamba -y --quiet -c conda-forge 
    RUN mamba install -y  -c onedata fs.onedatafs
    

    installation goes fine ( it installs fs.onedatafs 20.02.4 py37_0 )

    but when i try to import it in a notebook i got

    from fs.onedatafs import OnedataFS
    
    ImportError                               Traceback (most recent call last)
    <ipython-input-1-12d9be3ef961> in <module>
    ----> 1 from fs.onedatafs import OnedataFS
    
    /opt/conda/lib/python3.7/site-packages/fs/onedatafs/__init__.py in <module>
         10 
         11 if "pytest" not in sys.modules:
    ---> 12     from ._onedatafs import OnedataFS, OnedataSubFS # noqa
    
    /opt/conda/lib/python3.7/site-packages/fs/onedatafs/_onedatafs.py in <module>
         34 import six
         35 
    ---> 36 import onedatafs # noqa
         37 
         38 from ._util import ensure_unicode, info_to_stat, stat_to_permissions, to_ascii
    
    /opt/conda/lib/python3.7/site-packages/onedatafs/__init__.py in <module>
    ----> 1 from .onedatafs import OnedataFS, Stat, Xattr
    
    ImportError: libprotobuf.so.24: cannot open shared object file: No such file or directory
    

    i see that this version of protobuf is installed when running the installation of the lib

    libprotobuf 3.14.0 h780b84a_0 conda-forge/linux-64 2 MB

    cheers Andrea

    opened by andrea-manzi 4
  • Preferred way to install onedatafs

    Preferred way to install onedatafs

    Hi,

    we would like to use and push the usage of onedata in our community. For that, we wanted to integrate fs.onedatafs into our system. However, it is not clear what the preferred way is to install this library.

    Installing it via PyPI leads to the following error on python3.8 but also on python3.7.

    Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import fs.onedatafs
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/bag/miniconda3/envs/python37/lib/python3.7/site-packages/fs/onedatafs/__init__.py", line 12, in <module>
        from ._onedatafs import OnedataFS, OnedataSubFS # noqa
      File "/home/bag/miniconda3/envs/python37/lib/python3.7/site-packages/fs/onedatafs/_onedatafs.py", line 36, in <module>
        import onedatafs # noqa
    ModuleNotFoundError: No module named 'onedatafs'
    

    Is python>3.7 supported? The PyPI release is really outdated and the GitHub release is much newer. Is there a reason? We tried the latest github release but we got the same error.

    Next, we tried the anaconda release, but this does not work because you don't pin protobuf or libtbb so this release is also not useable out of the box. I'm part of the conda-forge and bioconda community so we could help with the correct setup of conda builds if you like.

    My general concern is that this project does not seem to be very active :( no issues, no PRs, the master branch is not updated and no CI infrastructure that runs tests. Is there any other preferred way to access OneData via python that I missed?

    Thanks for OneData we like it a lot and would more tightly integrate with it but we would need to python library for that. Bjoern

    opened by bgruening 13
  • ArgumentError when using numpy types for reading files

    ArgumentError when using numpy types for reading files

    Reading one large netCDF file using xarray results in errors due to wrong arguments to read:

    ArgumentError: Python argument types in
        OnedataFileHandle.read(OnedataFileHandle, numpy.int64, int)
    did not match C++ signature:
        read(OnedataFileHandle {lvalue}, long, unsigned long)
    

    The self.pos value was updated before by a call to seek() with a numpy.int64.

    Code to reproduce:

    import numpy as np
    
    from fs.onedatafs import OnedataFS
    onedata_provider_host = "plg-cyfronet-01.datahub.egi.eu"
    onedata_access_token = "MDA.."
    
    # Create connection to Oneprovider
    odfs = OnedataFS(onedata_provider_host, onedata_access_token)
    h = np.int64(100)
    f = odfs.open('/EGI Foundation/Untitled Folder/tas_1991_2016_NLD.csv', 'rb')
    f.seek(h)
    f.read(10)
    

    The original code was doing this:

    import xarray as xr
    f = odfs.open('/EGI Foundation/Untitled Folder/data2/ERA5_rainfall_and_temperature.nc', 'rb')
    ds = xr.open_dataset(f)
    
    opened by enolfc 3
Owner
onedata
onedata
Extract an archive file (zip file or tar file) stored on AWS S3

S3 Extract Extract an archive file (zip file or tar file) stored on AWS S3. Details Downloads archive from S3 into memory, then extract and re-upload

Evan 1 Dec 14, 2021
A python script to convert an ucompressed Gnucash XML file to a text file for Ledger and hledger.

README 1 gnucash2ledger gnucash2ledger is a Python script based on the Github Gist by nonducor (nonducor/gcash2ledger.py). This Python script will tak

Thomas Freeman 0 Jan 28, 2022
This is a file deletion program that asks you for an extension of a file (.mp3, .pdf, .docx, etc.) to delete all of the files in a dir that have that extension.

FileBulk This is a file deletion program that asks you for an extension of a file (.mp3, .pdf, .docx, etc.) to delete all of the files in a dir that h

Enoc Mena 1 Jun 26, 2022
Python package to read and display segregated file names present in a directory based on type of the file

tpyfilestructure Python package to read and display segregated file names present in a directory based on type of the file. Installation You can insta

Tharun Kumar T 2 Nov 28, 2021
Search for files under the specified directory. Extract the file name and file path and import them as data.

Search for files under the specified directory. Extract the file name and file path and import them as data. Based on that, search for the file, select it and open it.

G-jon FujiYama 2 Jan 10, 2022
Small-File-Explorer - I coded a small file explorer with several options

Petit explorateur de fichier / Small file explorer Pour la première option (création de répertoire) / For the first option (creation of a directory) e

Xerox 1 Jan 3, 2022
Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

pti-file-format Reverse engineering the Polyend Tracker instrument file format.

Jaap Roes 12 Aug 26, 2022
Generates a clean .txt file of contents of a 3 lined csv file

Generates a clean .txt file of contents of a 3 lined csv file. File contents is the .gml file of some function which stores the contents of the csv as a map.

Alex Eckardt 1 Jan 9, 2022
PaddingZip - a tool that you can craft a zip file that contains the padding characters between the file content.

PaddingZip - a tool that you can craft a zip file that contains the padding characters between the file content.

phithon 53 Nov 7, 2022
Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

laojunjun 12 Nov 21, 2022