Keepsake is a Python library that uploads files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage

Overview

Keepsake

Version control for machine learning.

Keepsake is a Python library that uploads files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get the data back out using the command-line interface or a notebook.

  • Track experiments: Automatically track code, hyperparameters, training data, weights, metrics, Python dependencies — everything.
  • Go back in time: Get back the code and weights from any checkpoint if you need to replicate your results or commit to Git after the fact.
  • Version your models: Model weights are stored on your own Amazon S3 or Google Cloud bucket, so it's really easy to feed them into production systems.

How it works

Just add two lines to your training code:

import torch
import keepsake

def train():
    # Save training code and hyperparameters
    experiment = keepsake.init(path=".", params={...})
    model = Model()

    for epoch in range(num_epochs):
        # ...

        torch.save(model, "model.pth")
        # Save model weights and metrics
        experiment.checkpoint(path="model.pth", metrics={...})

Then Keepsake will start tracking everything: code, hyperparameters, training data, weights, metrics, Python dependencies, and so on.

  • Open source & community-built: We’re trying to pull together the ML community so we can build this foundational piece of technology together.
  • You're in control of your data: All the data is stored on your own Amazon S3 or Google Cloud Storage as plain old files. There's no server to run.
  • It works with everything: Tensorflow, PyTorch, scikit-learn, XGBoost, you name it. It's just saving files and dictionaries – export however you want.

Features

Throw away your spreadsheet

Your experiments are all in one place, with filter and sort. Because the data's stored on S3, you can even see experiments that were run on other machines.

$ keepsake ls --filter "val_loss<0.2"
EXPERIMENT   HOST         STATUS    BEST CHECKPOINT
e510303      10.52.2.23   stopped   49668cb (val_loss=0.1484)
9e97e07      10.52.7.11   running   41f0c60 (val_loss=0.1989)

Analyze in a notebook

Don't like the CLI? No problem. You can retrieve, analyze, and plot your results from within a notebook. Think of it like a programmable Tensorboard.

Compare experiments

It diffs everything, all the way down to versions of dependencies, just in case that latest Tensorflow version did something weird.

$ keepsake diff 49668cb 41f0c60
Checkpoint:       49668cb     41f0c60
Experiment:       e510303     9e97e07

Params
learning_rate:    0.001       0.002

Python Packages
tensorflow:       2.3.0       2.3.1

Metrics
train_loss:       0.4626      0.8155
train_accuracy:   0.7909      0.7254
val_loss:         0.1484      0.1989
val_accuracy:     0.9607      0.9411

Commit to Git, after the fact

If you eventually want to store your code on Git, there's no need to commit everything as you go. Keepsake lets you get back to any point you called experiment.checkpoint() so, you can commit to Git once you've found something that works.

$ keepsake checkout f81069d
Copying code and weights to working directory...

# save the code to git
$ git commit -am "Use hinge loss"

Load models in production

You can use Keepsake to feed your models into production systems. Connect them back to how they were trained, who trained them, and what their metrics were.

import keepsake
model = torch.load(keepsake.experiments.get("e45a203").best().open("model.pth"))

Install

pip install -U keepsake

Get started

If you prefer training scripts and the CLI, follow the our tutorial to learn how Keepsake works.

If you prefer working in notebooks, follow our notebook tutorial on Colab.

If you like to learn concepts first, read our guide about how Keepsake works.

Get involved

Everyone uses version control for software, but it is much less common in machine learning.

Why is this? We spent a year talking to people in the ML community and this is what we found out:

  • Git doesn’t work well with machine learning. It can’t handle large files, it can’t handle key/value metadata like metrics, and it can’t commit automatically in your training script. There are some solutions for this, but they feel like band-aids.
  • It should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community.
  • It needs to be small, easy to use, and extensible. We found people struggling to integrate with “AI Platforms”. We want to make a tool that does one thing well and can be combined with other tools to produce the system you need.

We think the ML community needs a good version control system. But, version control systems are complex, and to make this a reality we need your help.

Have you strung together some shell scripts to build this for yourself? Are you interested in the problem of making machine learning reproducible?

Here are some ways you can help out:

Contributing & development environment

Take a look at our contributing instructions.

Comments
  • python/go grpc interface for async uploads

    python/go grpc interface for async uploads

    Removes duplicate logic in Python, reusing the Go implementation via a grpc API.

    Apologies for this massive PR, I couldn't see a way of splitting it up since it touches everything.

    Closes #317 Closes #344

    opened by andreasjansson 15
  • Pytorch Lightning Callback

    Pytorch Lightning Callback

    Created a Pytorch Lightning Callback to address issue #367 . I used a Callback instead of Logger as ModelCheckpoint is an existing callback in Pytorch Lightning. Using this is simple.

    replicate_callback = ReplicateCallback({
        'dataset': 'mnist',
        'precision': 16,
        'encoder_dim': 3
    }, monitor='val_loss')
    trainer = pl.Trainer(gpus=1, num_nodes=1, precision=16,
                         limit_train_batches=0.5,
                         callbacks=[replicate_callback])
    

    This doesn't have any checks about monitor and doesn't inherit from PL's ModelCheckpoint like you have done for Keras Integration. Please comment and let me know what would be a good way to progress in the PR.

    opened by gan3sh500 12
  • Remove host from list

    Remove host from list

    Pulled this out as a separate PR from #53. Discussion here: https://github.com/replicate/replicate/pull/53#discussion_r466522697.

    This is probably not the right solution, as per discussion, but gives us a place to discuss and work around it.

    Original commit message:

    This is wide and isn't that useful:

    • When it is running, you probably only need to know the host when something is broke. In which case, use show.
    • When it is stopped, you don't really have any reason to know it besides posterity.

    A valid use case I can think of is figuring out which machines are being used, but that feels like a bodge that needs a better solution.

    opened by bfirsh 6
  • Best commit in replicate ls/ps using primary metric

    Best commit in replicate ls/ps using primary metric

    Looks something like

    experiment  started             status   host      user     param-1  latest   step  label-1  best     step  label-1
    1eeeeee     10 seconds ago      running  10.1.1.1  andreas  100      3cccccc  20    0.02     2cccccc  20    0.01
    2eeeeee     about a second ago  stopped  10.1.1.2  andreas  200      4cccccc  5              N/A
    

    In order to support --storage-url and config metrics we need to fetch config from storage. This means that when you update metrics in your local replicate.yaml it won't affect replicate ls. I added a TODO to re-visit this.

    opened by andreasjansson 5
  • Make Keepsake a typed package

    Make Keepsake a typed package

    I use mypy in my day-to-day programming. I notice that the python keepsake package also has type annotations. Could you add a py.typed file to the python package as per mypy so that mypy uses the type annotations?

    I can also open a PR for this if that's easier.

    opened by samuelstevens 4
  • No module named 'replicate.keras_callback'

    No module named 'replicate.keras_callback'

    I really like the concept of replicate and would like to integrate it with my keras-training. So I used the keras-integration-tutorial. But:

    from replicate.keras_callback import ReplicateCallback
    

    just yields: ModuleNotFoundError: No module named 'replicate.keras_callback' I'm using: replicate==0.1.16 ... and as I'm writing this issue, I see that this seems to be the most recent version on windows.

    But for linux there is already 0.2.1, which indeed has the ReplicateCallback.

    So for everyone having this issue, switch to linux. ;-) Or maybe a more recent replicate-version for windows could be pushed to pypi.

    opened by stvogel 4
  • Check out individual files

    Check out individual files

    Why

    Currently, you can only check out the entire directory that you save. You should be able to check out individual files.

    Potential solution

    replicate checkout abc123 -f model.pth
    

    But what about checking out directories? Then -f is a bit weird.

    Design considerations

    Some design thinking is needed about what path things are checked out as. If you do this on a checkpoint which has data/model.pth:

    replicate checkout abc123 -o out/
    

    It will create out/data/model.pth. If you do this:

    replicate checkout abc123 -o out/ -f data/model.pth
    

    Then you might expect it to appear as out/model.pth, but that would be inconsistent with behavior when you don't pass -f. Similarly, what about -o out/somemodel.pth -f data/model.pth?

    help wanted type/enhancement 
    opened by bfirsh 4
  • basic checkout from python

    basic checkout from python

    Had to add a reference from the checkpoint to the experiment to get this working, but I think that's fine. The alternative APIs of exp.checkout(checkpoint_id, output_path) or exp.checkout(exp.best(), output_path) feel clunky. image

    opened by andreasjansson 4
  • Fully document S3 and EC2

    Fully document S3 and EC2

    This isn't actually working yet but I want to open this to confirm the approach with you before I implement it.

    Basically I'm thinking of making AWS work the same way GCP works:

    1. Run aws configure, to make ~/.aws/credentials.
    2. Use these admin creds to do all local stuff (this just works at the moment, because both Python and Go libs read ~/.aws/credentials
    3. For remote replicate run, automatically generate an IAM account with limited access and pass as environment variable to Docker container

    Does that sound sensible to you?

    Also god the AWS CLI is awful isn't it? I've tried to make it copy-paste friendly by using a stable instance name and putting EXTERNAL_IP in an environment variable but urrghh it does not make that easy. The first pass of this had loads of snark about how terrible the AWS CLI was but I removed it, lol.

    opened by bfirsh 4
  • Support .yml extension for replicate.yaml

    Support .yml extension for replicate.yaml

    We read the config replicate.yaml, but we should also support the filename replicate.yml, because it is a common alternative extension. It needs support in both Python and Go.

    good first issue help wanted type/enhancement 
    opened by bfirsh 3
  • Better `replicate ls` output

    Better `replicate ls` output

    Why

    The usual 2 column terminal table does not work so well for experiment data, because it quickly becomes too wide and doesn't fit.

    Specific issues:

    • It gets very wide
    • There's lots of blank horizontal space if you add new params
    • It's not obvious at a glance what has changed in an experiment

    How

    Some ideas:

    • Make the output responsive, so if you're on a narrower terminal it displays less information, and on a wider terminal it displays more information.
    • Add more dimensions to give us more freedom to design a better interface, so each line doesn't necessarily correspond to one experiment
    • Add formatting, which tabwriter does not let us do #67

    Strawman

    EXPERIMENT  STARTED     STATUS   PARAMS                 BEST CHECKPOINT     LATEST CHECKPOINT
    1f04f97     2020-11-04  stopped  learning_rate=0.001    6059d7b (step 17)   83cceda (step 20)
                                     num_epochs=20          accuracy=0.7366     accuracy=0.7336
                                                          
    b1de1ab     2020-11-04  stopped  learning_rate=0.001    db3b439 (step 10)   ce1358f (step 20)
                                     num_epochs=20          accuracy=0.843      accuracy=0.7492
    
    12c306a     2020-11-04  stopped  learning_rate=0.001    601ae0b (step 75)   ea78914 (step 85)   
                                     num_epochs=150         accuracy=0.8153     accuracy=0.8063
    
    3432557     2020-11-07  stopped  learning_rate=0.001    30b5116 (step 134)  2b003cc (step 150)  
                                     num_epochs=150         accuracy=0.8396     accuracy=0.839  
                                     discrimitive_rate=0.5  
    
    20ae2b5     2020-11-08  stopped  learning_rate=0.001    29c95b1 (step 101)  7f7a4d0 (step 150)  
                                     num_epochs=150         accuracy=0.8523     accuracy=0.8425
                                     widen_factor=10
                                     discrimitive_rate=1
    
    e6b67aa     2020-11-08  stopped  learning_rate=0.01     fae24ed (step 5)    fae24ed (step 5)
                                     num_epochs=150         accuracy=0.5643     accuracy=0.5643
                                     widen_factor=2
                                     discrimitive_rate=1
    
    c413e7c     2020-11-10  running  learning_rate=0.01     c19ab4e (step 112)  c19ab4e (step 112)
                                     num_epochs=150         accuracy=0.8734     accuracy=0.8734
                                     widen_factor=2
                                     discrimitive_rate=1
    
    

    Notes:

    • Pure tabular view does not scale well as you add params, so params and metrics are in single columns, solving it getting to wide. The downside is pure tabular is easier to scan. If you want that, export to CSV (or wait for a GUI!).
    • Params are only displayed if non-blank, solving problem of lots of blank space when you add more params.
    • params and metrics would be truncated to fit into terminal width

    Additional improvements:

    • Perhaps --plain (as per https://clig.dev/#output ) or --tabular reverts to current behavior for a tabular view and to pipe to line-based tools. If stdout is a TTY, perhaps this could output in a pager with the correct options such that it doesn't wrap and you can scroll right.
    • Perhaps there is useful formatting we can add -- e.g. to highlight was has changed in an experiment, or add visual structure. This is a tabwriter that works with formatting: https://github.com/juju/ansiterm #67
    • replicate ls gets more verbose in a compounding way because it displays all params used forever. Perhaps a more sensible default would be to display the last 25 experiments, or whatever, and replicate ls -a displays all experiments.
      • Showing latest 25 experiments, 1535 in total. Run replicate ls -a to display all experiments.
    • Perhaps we can add more metrics? Which ones should be added?

    Related

    • #182
    type/roadmap 
    opened by bfirsh 3
  • Bump github.com/aws/aws-sdk-go from 1.37.26 to 1.44.173 in /go

    Bump github.com/aws/aws-sdk-go from 1.37.26 to 1.44.173 in /go

    Bumps github.com/aws/aws-sdk-go from 1.37.26 to 1.44.173.

    Release notes

    Sourced from github.com/aws/aws-sdk-go's releases.

    Release v1.44.173 (2023-01-04)

    Service Client Updates

    • service/application-autoscaling: Updates service API and documentation
    • service/logs: Updates service documentation
      • Update to remove sequenceToken as a required field in PutLogEvents calls.
    • service/ssm: Updates service API and documentation
      • Adding support for QuickSetup Document Type in Systems Manager

    Release v1.44.172 (2023-01-03)

    Service Client Updates

    • service/securitylake: Updates service API and documentation

    Release v1.44.171 (2022-12-30)

    Service Client Updates

    • service/cloudfront: Updates service API and documentation
      • Extend response headers policy to support removing headers from viewer responses
    • service/iotfleetwise: Updates service documentation

    Release v1.44.170 (2022-12-29)

    Service Client Updates

    • service/apigateway: Updates service documentation
      • Documentation updates for Amazon API Gateway
    • service/elasticmapreduce: Updates service API and documentation
      • Added GetClusterSessionCredentials API to allow Amazon SageMaker Studio to connect to EMR on EC2 clusters with runtime roles and AWS Lake Formation-based access control for Apache Spark, Apache Hive, and Presto queries.
    • service/secretsmanager: Updates service API and documentation
      • Added owning service filter, include planned deletion flag, and next rotation date response parameter in ListSecrets.
    • service/wisdom: Updates service API and documentation

    Release v1.44.169 (2022-12-28)

    Service Client Updates

    • service/elasticache: Updates service API and documentation
      • This release allows you to modify the encryption in transit setting, for existing Redis clusters. You can now change the TLS configuration of your Redis clusters without the need to re-build or re-provision the clusters or impact application availability.
    • service/network-firewall: Updates service API and documentation
    • service/rds: Updates service API, documentation, waiters, paginators, and examples
      • This release adds support for Custom Engine Version (CEV) on RDS Custom SQL Server.
    • service/route53-recovery-control-config: Updates service documentation and paginators

    Release v1.44.168 (2022-12-27)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • Bump golang.org/x/tools from 0.1.0 to 0.5.0 in /go

    Bump golang.org/x/tools from 0.1.0 to 0.5.0 in /go

    Bumps golang.org/x/tools from 0.1.0 to 0.5.0.

    Release notes

    Sourced from golang.org/x/tools's releases.

    gopls/v0.5.0

    A full list of issues closed can be found in the gopls/v0.5.0 milestone.

    Memory usage

    • Rewrite of caching model, resulting in significant memory usage improvements (@​heschik).

    New features

    • Extract to function: Support for extracting code blocks that contain return statements (@​joshbaum).
    • Workspace symbols: Support for fzf-style search syntax (@​findleyr). The following syntax is supported:
      • ' for exact matching
      • ^ for prefix matching
      • $ for suffix matching

    Note: This feature does not yet work in VS Code. See golang/vscode-go#647 and microsoft/vscode#106788.

    • An experimental new code lens to view GC optimization details (@​pjweinb). Once the code lens is enabled, you will see a Toggle gc details annotation at the top of your file. Clicking it will show optimization diagnostics produced by the Go compiler, and clicking it once again will hide these diagnostics. Enable the code lens by adding the following to your settings:
      "codelens": {
      	"gc_details": true
      }
      
    • go mod tidy and go mod vendor code lenses for go.mod files (@​dandua98).
    • Support for filling in matching in-scope variables instead of just empty values in fillstruct and fillreturns (@​joshbaum).
    • Autocompletion within import statements (@​dandua98).
    • Autocompletion within package declarations (@​dandua98).

    Improvements

    • Improvements to workspace symbols ranking and fuzzy matching (@​findleyr, @​myitcv).
    • Better completion suggestions in type switch case clauses and for calls to append, function literals, and unnamed types (@​muirdm).

    Thank you

    Thank you to everyone who contributed to this release!

    @​heschik @​findleyr @​pjweinb @​joshbaum @​mcjcloud @​dandua98 @​muirdm @​leitzler @​myitcv @​matloob @​tennashi @​ainar-g @​hasheddan

    ... (truncated)

    Commits
    • 7db99dd go.mod: update golang.org/x dependencies
    • 1e0dff2 gopls/internal/regtest: avoid race in TestSwitchFromGOPATHToModuleMode
    • 0441b43 gopls/internal/lsp/cache: use specific mutexes for module data
    • 33071fb internal/robustio: move robustio
    • b01e7a4 gopls/internal/regtest/watch: don't run TestSwitchFromGOPATHToModuleMode
    • e417ea3 gopls: remove dead analysis code
    • 1a08d01 gopls/internal/lsp: update replace directives in go.mod for package renaming
    • eac36cb gopls/internal/regtest: port experimental workspace tests to go.work
    • 224a61b gopls/internal/lsp/source: delete Snapshot.WriteEnv method
    • 81e741e gopls/internal/lsp/safetoken: funnel more calls through this package
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • Bump google.golang.org/api from 0.40.0 to 0.106.0 in /go

    Bump google.golang.org/api from 0.40.0 to 0.106.0 in /go

    Bumps google.golang.org/api from 0.40.0 to 0.106.0.

    Release notes

    Sourced from google.golang.org/api's releases.

    v0.106.0

    0.106.0 (2023-01-04)

    Features

    • all: Auto-regenerate discovery clients (#1784) (a7f08e2)
    • all: Auto-regenerate discovery clients (#1788) (9fb35f5)
    • all: Auto-regenerate discovery clients (#1790) (7bd17b3)
    • all: Auto-regenerate discovery clients (#1794) (3944e86)
    • idtoken: Add support for impersonated_service_account creds type (#1792) (f6dec99), refs #873
    • option/internaloption: Add new EmbeddableAdapter option (#1787) (1569e5b)

    Bug Fixes

    • idtoken: Configure validator constructor to use no authentication (#1789) (b35900a), refs #1682

    v0.105.0

    0.105.0 (2022-12-14)

    Features

    • all: Auto-regenerate discovery clients (#1773) (37a2e41)
    • all: Auto-regenerate discovery clients (#1777) (5b02761)
    • googleapi: Add response headers to Error reported by CheckMediaResponse (#1781) (e4271df)
    • Support set null map entries for non-simple map values (#1782) (c58bf4c)

    v0.104.0

    0.104.0 (2022-12-07)

    Features

    Bug Fixes

    • idtoken: Increase MaxIdleConnsPerHost to 100 in NewClient (#1754) (629e217), refs #1744

    ... (truncated)

    Changelog

    Sourced from google.golang.org/api's changelog.

    0.106.0 (2023-01-04)

    Features

    • all: Auto-regenerate discovery clients (#1784) (a7f08e2)
    • all: Auto-regenerate discovery clients (#1788) (9fb35f5)
    • all: Auto-regenerate discovery clients (#1790) (7bd17b3)
    • all: Auto-regenerate discovery clients (#1794) (3944e86)
    • idtoken: Add support for impersonated_service_account creds type (#1792) (f6dec99), refs #873
    • option/internaloption: Add new EmbeddableAdapter option (#1787) (1569e5b)

    Bug Fixes

    • idtoken: Configure validator constructor to use no authentication (#1789) (b35900a), refs #1682

    0.105.0 (2022-12-14)

    Features

    • all: Auto-regenerate discovery clients (#1773) (37a2e41)
    • all: Auto-regenerate discovery clients (#1777) (5b02761)
    • googleapi: Add response headers to Error reported by CheckMediaResponse (#1781) (e4271df)
    • Support set null map entries for non-simple map values (#1782) (c58bf4c)

    0.104.0 (2022-12-07)

    Features

    Bug Fixes

    • idtoken: Increase MaxIdleConnsPerHost to 100 in NewClient (#1754) (629e217), refs #1744
    • transport/grpc: Separate resolution of creds and certs (#1759) (c213153)

    ... (truncated)

    Commits
    • ac7eb8f chore(main): release 0.106.0 (#1786)
    • 3944e86 feat(all): auto-regenerate discovery clients (#1794)
    • f6dec99 feat(idtoken): add support for impersonated_service_account creds type (#1792)
    • ddb5c65 test: add buffer to both sides of token expiry validation (#1797)
    • b35900a fix(idtoken): configure validator constructor to use no authentication (#1789)
    • ca86833 chore(all): update all (#1796)
    • a6b0739 chore: skip generating integrations:v1 as it fails generation (#1793)
    • 7bd17b3 feat(all): auto-regenerate discovery clients (#1790)
    • 9fb35f5 feat(all): auto-regenerate discovery clients (#1788)
    • 1569e5b feat(option/internaloption): add new EmbeddableAdapter option (#1787)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 1
  • Bump github.com/stretchr/testify from 1.7.0 to 1.8.1 in /go

    Bump github.com/stretchr/testify from 1.7.0 to 1.8.1 in /go

    Bumps github.com/stretchr/testify from 1.7.0 to 1.8.1.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • Bump github.com/golangci/golangci-lint from 1.38.0 to 1.50.1 in /go

    Bump github.com/golangci/golangci-lint from 1.38.0 to 1.50.1 in /go

    Bumps github.com/golangci/golangci-lint from 1.38.0 to 1.50.1.

    Release notes

    Sourced from github.com/golangci/golangci-lint's releases.

    v1.50.1

    Changelog

    • bd7edf31 build(deps): bump github.com/BurntSushi/toml from 1.2.0 to 1.2.1 (#3310)
    • 8926a95f build(deps): bump github.com/daixiang0/gci from 0.8.0 to 0.8.1 (#3309)
    • 98df0492 build(deps): bump github.com/kkHAIKE/contextcheck from 1.1.2 to 1.1.3 (#3296)
    • a760cb12 build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.0 (#3298)
    • 2823ec62 build(deps): bump github.com/tomarrell/wrapcheck/v2 from 2.6.2 to 2.7.0 (#3287)
    • 0793063c build(deps): bump github.com/tommy-muehle/go-mnd/v2 from 2.5.0 to 2.5.1 (#3295)
    • 046dce3a build(deps): bump golang.org/x/tools from 0.1.12 to 0.2.0 (#3312)
    • 2fa3b6f8 chore: update go.mod for gen_github_action_config script (#3300)
    • 6740559b code-climate: add default severity (#3294)
    • d03294f2 fix: remove redundant character escape '/' (#3278)
    • 8a1cf904 lll: skip imports (#3288)
    • 5ef7b413 revive: fix configuration parsing (#3284)

    v1.50.0

    Changelog

    • 890a8265 Normalize exclude-rules paths for Windows (#2387)
    • db4955a3 build(deps): bump github.com/OpenPeeDeeP/depguard from 1.1.0 to 1.1.1 (#3186)
    • 937e3d40 build(deps): bump github.com/curioswitch/go-reassign v0.1.2 to v0.2.0 (#3234)
    • 40082da3 build(deps): bump github.com/daixiang0/gci from 0.6.3 to 0.7.0 (#3190)
    • c78bc230 build(deps): bump github.com/daixiang0/gci from 0.7.0 to 0.8.0 (#3241)
    • 28d70954 build(deps): bump github.com/go-critic/go-critic from 0.6.4 to 0.6.5 (#3150)
    • ff61ea9f build(deps): bump github.com/kkHAIKE/contextcheck from 1.1.0 to 1.1.1 (#3208)
    • 0a581b3c build(deps): bump github.com/kkHAIKE/contextcheck from 1.1.1 to 1.1.2 (#3243)
    • 0a94c369 build(deps): bump github.com/mgechev/revive from 1.2.3 to 1.2.4 (#3257)
    • 33a78c92 build(deps): bump github.com/nishanths/exhaustive from 0.8.1 to 0.8.3 (#3207)
    • d5dc2a19 build(deps): bump github.com/polyfloyd/go-errorlint from 1.0.2 to 1.0.3 (#3209)
    • ac95c85f build(deps): bump github.com/polyfloyd/go-errorlint from 1.0.3 to 1.0.4 (#3226)
    • b5203c38 build(deps): bump github.com/polyfloyd/go-errorlint from 1.0.4 to 1.0.5 (#3254)
    • 4189d1cc build(deps): bump github.com/sashamelentyev/usestdlibvars from 1.13.0 to 1.14.0 (#3147)
    • 70d595e9 build(deps): bump github.com/sashamelentyev/usestdlibvars from 1.14.0 to 1.17.1 (#3188)
    • 1f155b7e build(deps): bump github.com/sashamelentyev/usestdlibvars from 1.17.1 to 1.19.0 (#3206)
    • d1dc9ae4 build(deps): bump github.com/sashamelentyev/usestdlibvars from 1.19.0 to 1.19.1 (#3242)
    • e3aa30f3 build(deps): bump github.com/sashamelentyev/usestdlibvars from 1.19.1 to 1.20.0 (#3255)
    • 24f86dfd build(deps): bump github.com/shirou/gopsutil/v3 from 3.22.7 to 3.22.8 (#3187)
    • 704109c6 build(deps): bump github.com/shirou/gopsutil/v3 from 3.22.8 to 3.22.9 (#3267)
    • 794f00c0 build(deps): bump github.com/spf13/viper from 1.12.0 to 1.13.0 (#3205)
    • 942e7e51 build(deps): bump github.com/sylvia7788/contextcheck from 1.0.6 to 1.0.8 (#3148)
    • ed47d7bf build(deps): bump github.com/sylvia7788/contextcheck from 1.0.8 to 1.0.9 (#3189)
    • 2019f766 build(deps): bump mvdan.cc/gofumpt from 0.3.1 to 0.4.0 (#3256)
    • 0d33a5ba cli: remove reference to old service (#3173)
    • 213fb565 contextcheck: change owner (#3198)
    • 8a3b754c exhaustive: add missing config (#3212)
    • febe5fdb feat: add linter dupword (#3192)
    • 281e1847 feat: add testableexamples linter (#3170)
    • 3a2ad908 feat: rename logrlint to loggercheck (#3144)
    • aba80c7f feat: update gofmt and goimports and add option "rewrite-rules" (#3174)
    • 091d2f43 interfacebloat: fix configuration loading (#3194)
    • 05c3af1a paralleltest: add tests of the ignore-missing option (#3233)

    ... (truncated)

    Changelog

    Sourced from github.com/golangci/golangci-lint's changelog.

    v1.50.1

    1. updated linters
      • contextcheck: from 1.1.2 to 1.1.3
      • go-mnd: from 2.5.0 to 2.5.1
      • wrapcheck: from 2.6.2 to 2.7.0
      • revive: fix configuration parsing
      • lll: skip imports
    2. misc.
      • windows: remove redundant character escape '/'
      • code-climate: add default severity

    v1.50.0

    1. new linters
    2. updated linters
      • contextcheck: change owner
      • contextcheck: from 1.0.6 to 1.1.2
      • depguard: from 1.1.0 to 1.1.1
      • exhaustive: add missing config
      • exhaustive: from 0.8.1 to 0.8.3
      • gci: from 0.6.3 to 0.8.0
      • go-critic: from 0.6.4 to 0.6.5
      • go-errorlint: from 1.0.2 to 1.0.5
      • go-reassign: v0.1.2 to v0.2.0
      • gofmt: add option rewrite-rules
      • gofumpt from 0.3.1 to 0.4.0
      • goimports: update to HEAD
      • interfacebloat: fix configuration loading
      • logrlint: rename logrlint to loggercheck
      • paralleltest: add tests of the ignore-missing option
      • revive: from 1.2.3 to 1.2.4
      • usestdlibvars: from 1.13.0 to 1.20.0
      • wsl: support all configs and update docs
    3. misc.
      • Normalize exclude-rules paths for Windows
      • add riscv64 to the install script
    4. Documentation
      • cli: remove reference to old service

    v1.49.0

    IMPORTANT: varcheck and deadcode has been removed of default linters.

    1. new linters

    ... (truncated)

    Commits
    • 8926a95 build(deps): bump github.com/daixiang0/gci from 0.8.0 to 0.8.1 (#3309)
    • 046dce3 build(deps): bump golang.org/x/tools from 0.1.12 to 0.2.0 (#3312)
    • bd7edf3 build(deps): bump github.com/BurntSushi/toml from 1.2.0 to 1.2.1 (#3310)
    • 6740559 code-climate: add default severity (#3294)
    • c1e24c1 doc: improved wording (#3302)
    • 9a592fb build(deps): bump parse-url and gatsby-telemetry in /docs (#3306)
    • 98df049 build(deps): bump github.com/kkHAIKE/contextcheck from 1.1.2 to 1.1.3 (#3296)
    • 0793063 build(deps): bump github.com/tommy-muehle/go-mnd/v2 from 2.5.0 to 2.5.1 (#3295)
    • 2fa3b6f chore: update go.mod for gen_github_action_config script (#3300)
    • a760cb1 build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.0 (#3298)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
Releases(v0.4.2)
  • v0.4.2(Mar 11, 2021)

  • v0.4.1(Mar 10, 2021)

    Just a few bugs squished:

    • Prefix some private Python APIs with underscores so they don't mess with IDE auto-complete. #542 (thanks @gabrielmbmb!)
    • Fix different plots being written on the same plot with incorrect axes. #553 (thanks @enochkan!)
    • Use AWS_DEFAULT_REGION environment variable to determine what region to create S3 buckets in. #561
    • Correctly throw error if path passed to checkpoint() does not exist. #556
    • Fix an issue where if only a single file was saved, then checkout would fail. #560
    • Bump dependencies. (thanks @dependabot!)

    Run pip install -U keepsake to get the new goodness.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Feb 6, 2021)

    With this release, we've renamed Replicate to Keepsake.

    We're going to use "Replicate" as the name for our broader project to make machine learning reproducible. We think that tools should do one thing well, and we want to create space for multiple tools.

    Migration is simple:

    • Use pip install keepsake to install or use keepsake in requirements.txt
    • Rename replicate to keepsake in your code, both where it is imported and where it is used
    • If you use the Keras or PyTorch Lightning callbacks, they're now both called KeepsakeCallback
    • Rename replicate.yaml to keepsake.yaml
    • Rename .replicateignore to .keepsakeignore

    If you just want to stick with the replicate package that's no problem, but you'll just miss out on new stuff in the future.

    No new features in this release. Keep on trackin'.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.2(Jan 12, 2021)

  • v0.3.1(Jan 12, 2021)

  • v0.3.0(Jan 8, 2021)

    This is the first big release since we launched Replicate, and we've got a smorgasbord of new features for you.

    In case you're landing here for the first time, Replicate is a version control system for machine learning models.

    Some highlights:

    Files are uploaded in the background

    When you pass files to replicate.init() or experiment.checkpoint(), these will now be uploaded in the background while your training continues. Using Replicate won't make your training any slower.

    This also involved a lot of behind-the-scenes improvements which have made things faster and more robust. For example, opening files from Python is now much faster.

    Easier to read experiment list

    The output of replicate ls has been redesigned so it works better with more experiments and params:

    Screen Shot 2021-01-06 at 15 02 36

    Check out individual files

    You can now check out individual files instead of everything inside a checkpoint and its experiment. For example, you can run this to just check out the file model.pth:

    replicate checkout 3b453f0 --path model.pth
    

    Thanks to @VastoLorde95 for working on this!

    PyTorch Lightning integration

    Replicate now includes a callback that makes it much easier to use with PyTorch Lightning. Full details are in the documentation.

    Thanks @gan3sh500 for working on this!

    And a few more things

    Some more features:

    • experiment.checkpoint() automatically generates the step number if you don't specify it. (Thanks @justinchuby!)
    • The Python version is now recorded when you run your experiment. One more step towards making your training environment reproducible. (Thanks @kvthr!)

    And a few bugs squashed:

    • Checkpoints are no longer counted as best if the primary metrics is NaN or None. (Thanks @ryanbloom!)
    • Fix ID generation when random seed is fixed.

    Thanks to everyone who suggested features, complained about broken things, and help build things!

    Run pip install -U replicate to upgrade. Or, add replicate~=0.3.0 to requirements.txt if you're using that.

    Source code(tar.gz)
    Source code(zip)
Owner
Replicate
Reproducible machine learning
Replicate
Bib-parser - Convenient script to parse .bib files with the ACM Digital Library like metadata

Bib Parser Convenient script to parse .bib files with the ACM Digital Library li

Mehtab Iqbal (Shahan) 1 Jan 26, 2022
Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

Simon Willison 140 Dec 4, 2022
Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images"

Reverse_Engineering_GMs Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Gener

null 100 Dec 18, 2022
A voice recognition assistant similar to amazon alexa, siri and google assistant.

kenyan-Siri Build an Artificial Assistant Full tutorial (video) To watch the tutorial, click on the image below Installation For windows users (run th

Alison Parker 3 Aug 19, 2022
Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

Antonio Piazza 39 Dec 27, 2022
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
MPRNet-Cloud-removal: Progressive cloud removal

MPRNet-Cloud-removal Progressive cloud removal Requirements 1.Pytorch >= 1.0 2.Python 3 3.NVIDIA GPU + CUDA 9.0 4.Tensorboard Installation 1.Clone the

Semi 95 Dec 18, 2022
Image Captioning on google cloud platform based on iot

Image-Captioning-on-google-cloud-platform-based-on-iot - Image Captioning on google cloud platform based on iot

Shweta_kumawat 1 Jan 20, 2022
This is a Machine Learning Based Hand Detector Project, It Uses Machine Learning Models and Modules Like Mediapipe, Developed By Google!

Machine Learning Hand Detector This is a Machine Learning Based Hand Detector Project, It Uses Machine Learning Models and Modules Like Mediapipe, Dev

Popstar Idhant 3 Feb 25, 2022
A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

null 34 Dec 28, 2022
Project Aquarium is a SUSE-sponsored open source project aiming at becoming an easy to use, rock solid storage appliance based on Ceph.

Project Aquarium Project Aquarium is a SUSE-sponsored open source project aiming at becoming an easy to use, rock solid storage appliance based on Cep

Aquarist Labs 73 Jul 21, 2022
Large dataset storage format for Pytorch

H5Record Large dataset ( > 100G, <= 1T) storage format for Pytorch (wip) Support python 3 pip install h5record Why? Writing large dataset is still a

theblackcat102 43 Oct 22, 2022
Exporter for Storage Area Network (SAN)

SAN Exporter Prometheus exporter for Storage Area Network (SAN). We all know that each SAN Storage vendor has their own glossary of terms, health/perf

vCloud 32 Dec 16, 2022
Code I use to automatically update my videos' metadata on YouTube

mCodingYouTube This repository contains the code I use to automatically update my videos' metadata on YouTube, including: titles, descriptions, tags,

James Murphy 19 Oct 7, 2022
Hierarchical Metadata-Aware Document Categorization under Weak Supervision (WSDM'21)

Hierarchical Metadata-Aware Document Categorization under Weak Supervision This project provides a weakly supervised framework for hierarchical metada

Yu Zhang 53 Sep 17, 2022
Tooling for converting STAC metadata to ODC data model

Tooling for converting STAC metadata to ODC data model.

Open Data Cube 65 Dec 20, 2022
Python wrapper to access the amazon selling partner API

PYTHON-AMAZON-SP-API Amazon Selling-Partner API If you have questions, please join on slack Contributions very welcome! Installation pip install pytho

Michael Primke 330 Jan 6, 2023