Cached file system for online resources in Python

Overview

Minato

Actions Status Python version License pypi version

Cache & file system for online resources in Python

Features

Minato enables you to:

  • Download & cache online recsources
    • minato supports the following protocols: HTTP(S) / AWS S3 / Google Cloud Storage
    • You can manage cached files via command line interface
  • Automatically update cached files based on ETag
    • minato downloads new versions if available when you access cached files
  • Open online files super easily
    • By using minato.open, you can read/write online resources like the built-in open method

Installation

pip install minato[all]

Usage

Python

import minato

# Read / write files on online storage
with minato.open("s3://your_bucket/path/to/file", "w") as f:
    f.write("Create a new file on AWS S3!")

# Cache & manage online resources in local storage
local_filename = minato.cached_path("http://example.com/path/to/archive.zip!inner/path/to/file")

CLI

❯ poetry run minato --help
usage: minato

positional arguments:
  {cache,list,remove,update}
    cache               cache remote file and return cached local file path
    list                show list of cached files
    remove              remove cached files
    update              update cached files

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
You might also like...
Exploit Discord's cache system to remote upload payloads on Discord users machines
Exploit Discord's cache system to remote upload payloads on Discord users machines

Exploit Discord's cache system to hide payloads PoC Remote upload embedded payload from image using EOF to Discord users machines through cache. Depen

A generic system for filtering Django QuerySets based on user selections

Django Filter Django-filter is a reusable Django application allowing users to declaratively add dynamic QuerySet filtering from URL parameters. Full

A Django based shop system

django-SHOP Django-SHOP aims to be a the easy, fun and fast e-commerce counterpart to django-CMS. Here you can find the full documentation for django-

Login System Django

Login-System-Django Login System Using Django Tech Used Django Python Html Run Locally Clone project git clone https://link-to-project Get project for

A Blog Management System Built with django
A Blog Management System Built with django

Blog Management System Backend use: Django Features Enhanced Ui

This repository contains django library management system project.

Library Management System Django ** INSTALLATION** First of all install python on your system. Then run pip install -r requirements.txt to required se

Djang Referral System
Djang Referral System

Djang Referral System About | Features | Technologies | Requirements | Starting | License | Author 🎯 About I created django referral system and I wan

Advanced school management system written in Django :)
Advanced school management system written in Django :)

Advanced school management system written in Django :) ⚙️ Config the project First you should make venv for this project. So in the main root of proje

A visual indicator of what environment/system you're using in django

A visual indicator of what environment/system you're using in django

Comments
  • Bump protobuf from 4.21.5 to 4.21.6

    Bump protobuf from 4.21.5 to 4.21.6

    Bumps protobuf from 4.21.5 to 4.21.6.

    Release notes

    Sourced from protobuf's releases.

    Protocol Buffers v3.0.0-beta-4

    Version 3.0.0-beta-4

    General

    • Added a deterministic serialization API for C++. The deterministic serialization guarantees that given a binary, equal messages will be serialized to the same bytes. This allows applications like MapReduce to group equal messages based on the serialized bytes. The deterministic serialization is, however, NOT canonical across languages; it is also unstable across different builds with schema changes due to unknown fields. Users who need canonical serialization, e.g. persistent storage in a canonical form, fingerprinting, etc, should define their own canonicalization specification and implement the serializer using reflection APIs rather than relying on this API.

    • Added OneofOptions. You can now define custom options for oneof groups.

      import "google/protobuf/descriptor.proto";
      extend google.protobuf.OneofOptions {
        optional int32 my_oneof_extension = 12345;
      }
      message Foo {
        oneof oneof_group {
          (my_oneof_extension) = 54321;
          ...
        }
      }
      

    C++ (beta)

    • Introduced a deterministic serialization API in CodedOutputStream::SetSerializationDeterministic(bool). See the notes about deterministic serialization in the General section.
    • Added google::protobuf::Map::swap() to swap two map fields.
    • Fixed a memory leak when calling Reflection::ReleaseMessage() on a message allocated on arena.
    • Improved error reporting when parsing text format protos.
    • JSON
      • Added a new parser option to ignore unknown fields when parsing JSON.
      • Added convenient methods for message to/from JSON conversion.
    • Various performance optimizations.

    Java (beta)

    • File option "java_generate_equals_and_hash" is now deprecated. equals() and hashCode() methods are generated by default.
    • Added a new JSON printer option "omittingInsignificantWhitespace" to produce a more compact JSON output. The printer will pretty-print by default.
    • Updated Java runtime to be compatible with 2.5.0/2.6.1 generated protos.

    Python (beta)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump ujson from 5.3.0 to 5.4.0

    Bump ujson from 5.3.0 to 5.4.0

    Bumps ujson from 5.3.0 to 5.4.0.

    Release notes

    Sourced from ujson's releases.

    5.4.0

    Added

    Fixed

    Commits
    • 9c20de0 Merge pull request from GHSA-fm67-cv37-96ff
    • b21da40 Fix double free on string decoding if realloc fails
    • 67ec071 Merge pull request #555 from JustAnotherArchivist/fix-decode-surrogates-2
    • bc7bdff Replace wchar_t string decoding implementation with a uint32_t-based one
    • cc70119 Merge pull request #548 from JustAnotherArchivist/arbitrary-ints
    • 4b5cccc Merge pull request #553 from bwoodsend/pypy-ci
    • abe26fc Merge pull request #551 from bwoodsend/bye-bye-travis
    • 3efb5cc Delete old TravisCI workflow and references.
    • 404de1a xfail test_decode_surrogate_characters() on Windows PyPy.
    • f7e66dc Switch to musl docker base images.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
Releases(v0.10.0)
  • v0.10.0(Aug 11, 2022)

    What's Changed

    • Bump ujson from 5.3.0 to 5.4.0 by @dependabot in https://github.com/altescy/minato/pull/36
    • Table supports full-width characters by @altescy in https://github.com/altescy/minato/pull/37
    • Modify auto-format settings by @altescy in https://github.com/altescy/minato/pull/38
    • Add progress & remove tqdm by @altescy in https://github.com/altescy/minato/pull/39
    • Add disable-progressbar option by @altescy in https://github.com/altescy/minato/pull/40
    • Add interactive selector by @altescy in https://github.com/altescy/minato/pull/41
    • Bump version number to v0.10.0 by @altescy in https://github.com/altescy/minato/pull/42

    New Contributors

    • @dependabot made their first contribution in https://github.com/altescy/minato/pull/36

    Full Changelog: https://github.com/altescy/minato/compare/v0.9.0...v0.10.0

    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Jul 2, 2022)

    What's Changed

    • Use options for open_file function by @altescy in https://github.com/altescy/minato/pull/33
    • Support Python 3.9 and 3.10 by @altescy in https://github.com/altescy/minato/pull/34
    • Bump version number to v0.9.0 by @altescy in https://github.com/altescy/minato/pull/35

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.2...v0.9.0

    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(Feb 24, 2022)

    What's Changed

    • Modify log level by @altescy in https://github.com/altescy/minato/pull/30
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/31
    • Bump version number to v0.8.2 by @altescy in https://github.com/altescy/minato/pull/32

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.1...v0.8.2

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Feb 11, 2022)

    What's Changed

    • Fix s3 progress by @altescy in https://github.com/altescy/minato/pull/27
    • Add all command to Makefile by @altescy in https://github.com/altescy/minato/pull/28
    • Bump version number to v0.8.1 by @altescy in https://github.com/altescy/minato/pull/29

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.0...v0.8.1

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Feb 8, 2022)

    What's Changed

    • Modify tqdm progress by @altescy in https://github.com/altescy/minato/pull/22
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/23
    • Split github workflows into ci and pypi-publish by @altescy in https://github.com/altescy/minato/pull/24
    • Modify type annotations by @altescy in https://github.com/altescy/minato/pull/25
    • Bump version number to v0.8.0 by @altescy in https://github.com/altescy/minato/pull/26

    Full Changelog: https://github.com/altescy/minato/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Nov 19, 2021)

    • Shrink table column width automatically (#16)
    • Modify S3 download (#17)
    • Move boto3-stubs to dev-dependencies (#18)
    • Add upload method to S3FileSystem (#19)
    • Add upload method to GCSFileSystem (#20)
    • Bump version number to v0.7.0 (#21)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(Oct 23, 2021)

  • v0.6.1(Oct 1, 2021)

  • v0.6.0(Sep 30, 2021)

    • Remove quotations of S3 Etags (#2)
    • Modify CI settings (#3)
    • Update dependent packages (#4)
    • Make filesystems optional (#5)
    • Add opening options (#6)
    • Add helps to each command line options (#7)
    • Add feature description to readme (#8)
    • Bump version number to v0.6.0 (#9)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jul 31, 2021)

  • v0.4.0(Jun 25, 2021)

    • Support file-level expiration
    • Enable automatic cache updating based on HTTP ETag
    • Show download progress of S3 and GCS
    • Deprecate upload/download commands
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jun 23, 2021)

Owner
Yasuhiro Yamaguchi
Research Engineer / NLP / ML
Yasuhiro Yamaguchi
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 8 Oct 28, 2022
A Django Online Library Management Project.

Why am I doing this? I started learning ?? Django few months back, and this is a practice project from MDN Web Docs that touches the aspects of Django

null 1 Nov 13, 2021
This website serves as an online database (hosted via SQLLite) for fictional businesses in the area to store contact information (name, email, phone number, etc.) for fictional customers.

Django-Online-Business-Database-Project this project is still in progress Overview of Website This website serves as an online database (hosted via SQ

null 1 Oct 30, 2021
Radically simplified static file serving for Python web apps

WhiteNoise Radically simplified static file serving for Python web apps With a couple of lines of config WhiteNoise allows your web app to serve its o

Dave Evans 2.1k Dec 15, 2022
AUES Student Management System Developed for laboratory works №9 Purpose using Python (Django).

AUES Student Management System (L M S ) AUES Student Management System Developed for laboratory works №9 Purpose using Python (Django). I've created t

ANAS NABIL 2 Dec 6, 2021
A task management system created using Django 4.0 and Python 3.8 for a hackathon.

Task Management System A task management app for Projects created using Django v4.0 and Python 3.8 for educational purpose. This project was created d

Harsh Agarwal 1 Dec 12, 2021
Automatically deletes old file for FileField and ImageField. It also deletes files on models instance deletion.

Django Cleanup Features The django-cleanup app automatically deletes files for FileField, ImageField and subclasses. When a FileField's value is chang

Ilya Shalyapin 838 Dec 30, 2022
File and Image Management Application for django

Django Filer django Filer is a file management application for django that makes handling of files and images a breeze. Contributing This is a an open

django CMS Association 1.6k Dec 28, 2022
django Filer is a file management application for django that makes handling of files and images a breeze.

django Filer is a file management application for django that makes handling of files and images a breeze.

django CMS Association 1.6k Jan 6, 2023
System checks for your project's environment.

django-version-checks System checks for your project's environment. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your te

Adam Johnson 33 Dec 22, 2022