📚 Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

nteract

Last update: Jan 3, 2023

Overview

papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

Papermill lets you:

parameterize notebooks
execute notebooks

This opens up new opportunities for how notebooks can be used. For example:

Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.
Do you want to run a notebook and depending on its results, choose a particular notebook to run next? You can now programmatically execute a workflow without having to copy and paste from notebook to notebook manually.

Papermill takes an opinionated approach to notebook parameterization and execution based on our experiences using notebooks at scale in data pipelines.

Installation

From the command line:

pip install papermill

For all optional io dependencies, you can specify individual bundles like s3, or azure -- or use all. To use Black to format parameters you can add as an extra requires ['black'].

pip install papermill[all]

Python Version Support

This library currently supports Python 3.6+ versions. As minor Python versions are officially sunset by the Python org papermill will similarly drop support in the future.

Usage

Parameterizing a Notebook

To parameterize your notebook designate a cell with the tag parameters.

Papermill looks for the parameters cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with injected-parameters with input parameters in order to overwrite the values in parameters. If no cell is tagged with parameters the injected cell will be inserted at the top of the notebook.

Additionally, if you rerun notebooks through papermill and it will reuse the injected-parameters cell from the prior run. In this case Papermill will replace the old injected-parameters cell with the new run's inputs.

Executing a Notebook

The two ways to execute the notebook with parameters are: (1) through the Python API and (2) through the command line interface.

Execute via the Python API

import papermill as pm

pm.execute_notebook(
   'path/to/input.ipynb',
   'path/to/output.ipynb',
   parameters = dict(alpha=0.6, ratio=0.1)
)

Execute via CLI

Here's an example of a local notebook being executed and output to an Amazon S3 account:

$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

NOTE: If you use multiple AWS accounts, and you have properly configured your AWS credentials, then you can specify which account to use by setting the AWS_PROFILE environment variable at the command-line. For example:

$ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

In the above example, two parameters are set: alpha and l1_ratio using -p (--parameters also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters:

$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0

Using -r or --parameters_raw, users can set parameters one by one. However, unlike -p, the parameter will remain a string, even if it may be interpreted as a number or boolean.

$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml

Using -f or --parameters_file, users can provide a YAML file from which parameter values should be read.

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
alpha: 0.6
l1_ratio: 0.1"

Using -y or --parameters_yaml, users can directly provide a YAML string containing parameter values.

$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==

Using -b or --parameters_base64, users can provide a YAML string, base64-encoded, containing parameter values.

When using YAML to pass arguments, through -y, -b or -f, parameter values can be arrays or dictionaries:

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
    - 0.0
    - 1.0
    - 2.0
    - 3.0
linear_function:
    slope: 3.0
    intercept: 1.0"

Supported Name Handlers

Papermill supports the following name handlers for input and output paths during execution:

Local file system: local
HTTP, HTTPS protocol: http://, https://
Amazon Web Services: AWS S3 s3://
Azure: Azure DataLake Store, Azure Blob Store adl://, abs://
Google Cloud: Google Cloud Storage gs://

Development Guide

Read CONTRIBUTING.md for guidelines on how to setup a local development environment and make code changes back to Papermill.

For development guidelines look in the DEVELOPMENT_GUIDE.md file. This should inform you on how to make particular additions to the code base.

Documentation

We host the Papermill documentation on ReadTheDocs.

Comments

Unable to write into GCS bucket with papermill[gcs]

When running GCFS application via papermill[gcs]

papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb

I'm getting Error: HTTP 429 Rate exceeds.

Works if output notebook is written locally:

papermill gs://my-bucket/test.ipynb /tmp/test.ipynb

Local file size is: 57K

ls -alh /tmp/test.ipynb 
-rw-r--r--  1 gogasca  wheel    57K Feb 14 10:37 /tmp/test.ipynb

GCSFS reference https://github.com/dask/gcsfs/issues/130

How to reproduce?

pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb

Logs:

10
Ending Cell 6------------------------------------------
Exception gcsfs.utils.HtmlError: HtmlError(u'The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.',) in <bound method GCSFile.__del__ of <GCSFile d
pe-sandbox/test.ipynb>> ignored
Traceback (most recent call last):
  File "/usr/local/bin/papermill", line 11, in <module>
    sys.exit(papermill())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/usr/local/lib/python2.7/dist-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-150>", line 2, in close
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1548, in close
    self.flush(force=True)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-145>", line 2, in flush
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1367, in flush
    self._simple_upload()
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-148>", line 2, in _simple_upload
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1465, in _simple_upload
    validate_response(r, path)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 162, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
+ err 'Processing notebook failed'
++ date +%Y-%m-%dT%H:%M:%S%z
+ echo '[2019-02-14T18:32:58+0000]: Processing notebook failed'
[2019-02-14T18:32:58+0000]: Processing notebook failed
+ exit 1

I already defined:

export GOOGLE_APPLICATION_CREDENTIALS=/keys/my-project.json
gcloud config set account [email protected]
gcloud auth activate-service-account --key-file=/keys/my-project.json

in MacOS environment I get similar errors: (Added debugging)

papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://dpe-sandbox/test.ipynb
Input Notebook:  gs://cloud-samples-data/papermill/samples/test.ipynb
Output Notebook: gs://dpe-sandbox/test.ipynb
('dpe-cloud-mle', 'full_control', None, None, 'none', None)
  0%|                                                                                                   | 0/28 [00:00<?, ?it/s]uploading
11317
uploading
11342
uploading
11372
  4%|███▎                                                                                       | 1/28 [00:01<00:50,  1.88s/it]uploading
11397
uploading
11427
  7%|██████▌                                                                                    | 2/28 [00:02<00:41,  1.59s/it]uploading
11452
uploading
11482
 11%|█████████▊                                                                                 | 3/28 [00:03<00:33,  1.32s/it]uploading
11507
uploading
11537
 14%|█████████████                                                                              | 4/28 [00:04<00:26,  1.10s/it]uploading
11562
429

429
Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/bin/papermill", line 10, in <module>
    sys.exit(papermill())
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 302, in execute_notebook
    nb_man.notebook_complete()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 249, in notebook_complete
    self.save()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.

bug

opened by gogasca 22

Read notebook from stdin/write to stdout when path is '-'
This pull requests implements - as stdin/stdout, cf. #372.

A case when one wants to read notebooks from stdin are Jupytext notebooks (#365). Piping notebooks to stdout can also be useful in case of chained notebook conversions (to html with nbconvert for instance).

I have included two tests. What remains is probably to

[x] review the stdout option, and the tests

[ ] mention - as stdin/stdout in Papermill's documentation, or at least in the CLI

[x] increase the default value for --log-level when the output path is -

I have tested the PR with the command below, which prints the notebook (represented as an Hydrogen script) once the injected parameter cell as been added by papermill:

jupytext notebook_as_script.py --to ipynb -o - | papermill - - --prepare-only -p a 2 --log-level ERROR | jupytext --to py:percent

where notebook_as_script.py is a notebook represented as a script using Jupytext, e.g.:

# --- # jupyter: # jupytext: # text_representation: # extension: .py # format_name: percent # format_version: '1.2' # jupytext_version: 1.1.3 # kernelspec: # display_name: Python3 # language: python # name: python3 # --- # %% {"tags": ["parameters"]} a = 1 # %% print('a={}'.format(a))

As expected, the command above returns the updated representation of the notebook, with an additional cell in between:

# %% {"tags": ["injected-parameters"]} # Parameters a = 2
opened by mwouts 21

ipyleaflet map is not rendered when ruhning in papermill

I have the following cell:

from IPython.display import display
from ipyleaflet import Map, Polyline

m = Map(center=(32.2, 34.8), zoom=12)
for r in df.itertuples():
    line = Polyline(
        locations=[[[r.start_latitude, r.start_longitude], [r.end_latitude, r.end_longitude]]],
        weight=15
    )
    m.add_layer(line)

display(m)

The map is displayed correctly, when running interactively. However, when running the same notebook using papermill it renders just this:

Map(basemap={'url': 'https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', 'max_zoom': 19, 'attribution': 'Map …

opened by spektom 20

`RuntimeError: Kernel didn't respond in 60 seconds`, when trying to run papermill with python multiprocessing
Hello, I am trying to run multiple parameterized notebooks in parallel. Currently, I am using papermill inside Jupyter Notebook and if I try to use multiprocessing pool to map a list of parameters as pass them to pm.execute_notebook, I get RuntimeError: Kernel didn't respond in 60 seconds. I am running everything with Python 2.7.

This is the code I use:

import papermill as pm import multiprocessing as mp def run_nb(data): d1, d2 = data pm.execute_notebook(in_nb, out_nb, parameters = dict(d1=d1, d2=d2) ) pool = mp.Pool(4) pool.map(run_nb, zip(data1, data2)) pool.close() pool.join()

It works correctly using the standard python map.

Btw, is there a known way to produce multiple notebooks in parallel with papermill?

Thanks!
upstream
opened by franzoni315 18

Papermill raises error for id field when running jupyter-notebook 6.2 with the -p flag

When running papermill v2.2.1 in an environment where jupyter-notebook v6.2 is installed I am seeing the following error when running with parameters (-p flag)

papermill notebook.ipynb outputs/notebook.ipynb -p my_config --no-progress-bar --log-output

Output:

[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)
[2021-01-18T09:44:24.218Z] Failed validating 'additionalProperties' in code_cell:
[2021-01-18T09:44:24.218Z] On instance['cells'][0]:
[2021-01-18T09:44:24.218Z] {'cell_type': 'code',
[2021-01-18T09:44:24.218Z]  'execution_count': 1,
[2021-01-18T09:44:24.218Z]  'id': 'nasty-bearing',
[2021-01-18T09:44:24.218Z]  'metadata': {'execution': {'iopub.execute_input': '2021-01-18T09:44:22.903942Z',
[2021-01-18T09:44:24.218Z]                             'iopub.status.busy': '2021-01-18T09:44:22.903349Z',
[2021-01-18T09:44:24.218Z]                             'iopub.status.idle': '2021-01-18T09:44:22.905999Z',
[2021-01-18T09:44:24.218Z]                             'shell.execute_reply': '2021-01-18T09:44:22.905474Z'},
[2021-01-18T09:44:24.218Z]               'papermill': {'duration': 0.01294,
[2021-01-18T09:44:24.218Z]                             'end_time': '2021-01-18T09:44:22.906187',
[2021-01-18T09:44:24.218Z]                             'exception': False,
[2021-01-18T09:44:24.218Z]                             'start_time': '2021-01-18T09:44:22.893247',
[2021-01-18T09:44:24.218Z]                             'status': 'completed'},
[2021-01-18T09:44:24.218Z]               'tags': ['injected-parameters']}

I think this may be due to a change in jupyter-notebook v6.2 that has added an "id" field to the cell properties : https://github.com/jupyter/notebook/pull/5928/ .

The error does not occur when jupyter-notebook v6.1.6 or earlier is running

opened by malcolmbovey 16

Let user choose how to deal with execution result.

Hi, here I add this to let the user choose how to deal with execution result if I use it in an application, I can use variable to deal with the result, no need to read again from the output file. Thanks.

opened by jarrekk 16
Return code of executing notebooks in papermill

Hi All,

Is there a way to capture return code of executing notebooks in papermill API library or CLI methods? When any notebook cell execution encounters an error, it should return an error return code AND when all notebook cells execute successfully it should return a success return code.

Thanks!
question

opened by vinaykumar80 14
expose current output path as $PAPERMILL_OUTPUT_PATH

As a papermill user I want to expose the output path of the currently run notebook via an environment variable and an API So that the output path is available inside of the executed notebook

Addresses #231

The PR adds an interface to Engine to pass environment variables to the executing kernel. This decision was made s.t. remotely executed engines/kernel can make use of this feature, e.g., the kaggle engine from #229. Whether environment variables are the way to go is up for discussion.

opened by mindsbackyard 14
error message suddenly popped up

Hi,

I have been using papermill and NbConvert to parameterize and convert Jupyter Notebooks. It worked really well until recently I started getting the following messages:

/home/florathecat/anaconda3/lib/python3.6/site-packages/nbconvert/filters/datatypefilter.py:41: UserWarning: Your element with mimetype(s) dict_keys(['application/papermill.record+json']) is not able to be represented. mimetypes=output.keys())

The notebooks are still parameterized and converted fine. It is just the error message bugs me. I have tried to update conda, python, nbconvert, and reinstall papermill. None of these seem to work. I'll appreciate if somebody can help.

Yun

opened by florathecat 13

Some weird error messages when executing a notebook involving pytorch

I have a notebook for training a model using pytorch. The notebook runs fine if I run it from browser. But I ran into the following problem when executing it via papermill

Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/Grammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp0twtlmvs'
Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/PatternGrammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp2_jvdud_'
Executing:   0%|                                                                                   | 0/23 [00:00<?, ?cell/s]Executing notebook with kernel: pytorch_p36
Executing:  22%|████████████████▎                                                          | 5/23 [00:03<00:13,  1.38cell/s]
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/cli.py", line 256, in papermill
    execution_timeout=execution_timeout,
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 118, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 230, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-3916aaf64ab2> in <module>
----> 1 from torchvision import datasets, transforms
      2 
      3 datasets.MNIST('data', download=True, transform=transforms.Compose([
      4     transforms.ToTensor(),
      5     transforms.Normalize((0.1307,), (0.3081,))

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/__init__.py in <module>
      1 import warnings
      2 
----> 3 from torchvision import models
      4 from torchvision import datasets
      5 from torchvision import ops

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/__init__.py in <module>
     10 from .shufflenetv2 import *
     11 from . import segmentation
---> 12 from . import detection
     13 from . import video
     14 from . import quantization

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/__init__.py in <module>
----> 1 from .faster_rcnn import *
      2 from .mask_rcnn import *
      3 from .keypoint_rcnn import *

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py in <module>
     11 
     12 from .generalized_rcnn import GeneralizedRCNN
---> 13 from .rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
     14 from .roi_heads import RoIHeads
     15 from .transform import GeneralizedRCNNTransform

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/rpn.py in <module>
      9 from torchvision.ops import boxes as box_ops
     10 
---> 11 from . import _utils as det_utils
     12 from .image_list import ImageList
     13 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py in <module>
     17 
     18 @torch.jit.script
---> 19 class BalancedPositiveNegativeSampler(object):
     20     """
     21     This class samples batches, ensuring that they contain a fixed proportion of positives

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1217         if _rcb is None:
   1218             _rcb = _jit_internal.createResolutionCallback(_frames_up + 1)
-> 1219         _compile_and_register_class(obj, _rcb, qualified_name)
   1220         return obj
   1221     else:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in _compile_and_register_class(obj, rcb, qualified_name)
   1074 def _compile_and_register_class(obj, rcb, qualified_name):
   1075     ast = get_jit_class_def(obj, obj.__name__)
-> 1076     _jit_script_class_compile(qualified_name, ast, rcb)
   1077     _add_script_class(obj, qualified_name)
   1078 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in try_compile_fn(fn, loc)
    220     # object
    221     rcb = _jit_internal.createResolutionCallbackFromClosure(fn)
--> 222     return torch.jit.script(fn, _rcb=rcb)
    223 
    224 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1224         if _rcb is None:
   1225             _rcb = _gen_rcb(obj, _frames_up)
-> 1226         fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
   1227         # Forward docstrings
   1228         fn.__doc__ = obj.__doc__

RuntimeError: 
builtin cannot be used as a value:
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:14:56
def zeros_like(tensor, dtype):
    # type: (Tensor, int) -> Tensor
    return torch.zeros_like(tensor, dtype=dtype, layout=tensor.layout,
                                                        ~~~~~~~~~~~~~ <--- HERE
                            device=tensor.device, pin_memory=tensor.is_pinned())
'zeros_like' is being compiled since it was called from '__torch__.torchvision.models.detection._utils.BalancedPositiveNegativeSampler.__call__'
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:72:12

            # randomly select positive and negative examples
            perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
            perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

            pos_idx_per_image = positive[perm1]
            neg_idx_per_image = negative[perm2]

            # create binary mask from indices
            pos_idx_per_image_mask = zeros_like(
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
                matched_idxs_per_image, dtype=torch.uint8
            )
            neg_idx_per_image_mask = zeros_like(
                matched_idxs_per_image, dtype=torch.uint8
            )

            pos_idx_per_image_mask[pos_idx_per_image] = torch.tensor(1, dtype=torch.uint8)
            neg_idx_per_image_mask[neg_idx_per_image] = torch.tensor(1, dtype=torch.uint8)

question

opened by hsl89 12

Include link to the cell with an error, in failure heading

Papermill currently inserts an error message An Exception was encountered at 'In [NNN]' at the top of a notebook that fails to execute, where NNN is the cell execution count of the failing cell. This patch adjusts the In [NNN] text to be a link to the relevant cell, to make it easy to jump to. This works by inserting another markdown cell immediately before the failing cell, where the markdown cell contains an HTML anchor (specifically, id="papermill-error-cell"). The heading then links to it with a #papermill-error-cell fragment HREF.

This helps reduce the scrolling and searching required for long notebooks where the error cell might be one of dozens, or hidden among long output. For an example of the latter, see https://nbviewer.jupyter.org/gist/huonw/dd13fa49b5876bfea26b69832702d573.

opened by huonw 12
ImportError: attempted relative import with no known parent package
I'm trying to turn into a pip package a notebook which runs using Papermill. There are python file local to the package that I want to import into the notebook but it's giving me an error:

# SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) # sys.path.append(os.path.dirname(SCRIPT_DIR)) from . import utils, preprocessing as prep ImportError: attempted relative import with no known parent package

This way of importing gives this error whether I try to run the notebook manually and through Papermill.

I've also tried importing it directly:

import utils, preprocessing as prep ModuleNotFoundError: No module named 'utils'

The direct import runs fine if I execute the notebook manually, but when I try to run them through Papermill it doesn't work.

In my package parent I have

import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__))) from . import preprocessing from . import utils
bug help wanted
opened by danre07 1

Papermill is not working with an error of click/zmq/nbclient

Hi I'm trying to build a new environment. But I ran into a problem which the papermill is not working. My .ipynb file is really simple. only has one single line of print(3). I can't find anything related to this Traceback in Google. Does anyone know how to solve it?

Sorry I can't provide the full list of my package and their versions. That would be too long. I just leave a version list of the package related at the very end of this page. Let me know if you need more information about it.

Traceback (most recent call last):
  File "/opt/conda/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/cli.py", line 267, in papermill
    execution_timeout=execution_timeout,
  File "/opt/conda/lib/python3.7/site-packages/papermill/execute.py", line 124, in execute_notebook
    **engine_kwargs
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 367, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "/opt/conda/lib/python3.7/site-packages/papermill/clientwrap.py", line 43, in execute
    with self.setup_kernel(**kwargs):
  File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.7/site-packages/nbclient/client.py", line 562, in setup_kernel
    self.start_new_kernel_client()
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/opt/conda/lib/python3.7/site-packages/nbclient/client.py", line 532, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 96, in ensure_async
    result = await obj
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/client.py", line 187, in _async_wait_for_ready
    msg = await self.shell_channel.get_msg(timeout=1)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/channels.py", line 224, in get_msg
    ready = await self.socket.poll(timeout)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 387, in poll
    f = cast(Future, p.poll(timeout))
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 128, in poll
    socket._add_recv_event('poll', future=watcher)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 493, in _add_recv_event
    self._add_io_state(POLLIN)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 663, in _add_io_state
    self._update_handler(self._state)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 679, in _update_handler
    self._get_loop()
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 62, in _get_loop
    self._init_io_state(self._current_loop)
TypeError: _init_io_state() takes 1 positional argument but 2 were given
Could not destroy zmq context for <jupyter_client.asynchronous.client.AsyncKernelClient object at 0x7fa8e282dc10>

here are packages versions which might related to the error:

Python                    3.7.13
click                     8.0.4 
papermill                 2.4.0  
nbclient                  0.5.13
jupyter_client            5.3.4
pyzmq                     24.0.1

bug help wanted

opened by corvettettt 0

Remove parameters from notebook output

🚀 Feature

It would be nice to be able to remove parameters from the outputted notebook

Motivation

I have several parameters which need secrets e.g. API keys, passwords, etc All my parameters come from static places like SecretsManager or Env or Config file

From what I understand I'd have to modify papermill.execute to make another transformation which removes parameters cell, correct?

similar to this https://github.com/nteract/papermill/blob/54f6c038cdae0c70d5fb04691fa465e12aeb62cb/papermill/execute.py#L136
enhancement help wanted

opened by 1oglop1 0
Is it possible to choose which cells to run using papermill.execute_notebook?
I use papermill to run a jupyter notebook. Is it possible that I can choose which cell to run, and which not to run using papermill?

papermill.execute_notebook( 'notebooks.ipynb', None, parameters=dict(params=params), log_output=True )
opened by wxy347 1
Support for Python 3.11

🚀 Feature

I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

Motivation

The papermill is one of the dependencies of Airlfow that need to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.

I'd appreciate if someone in the ntarct team attempted to migrate and have a PR running (and failing possibly until all prerequisite are met so that we can also track the progress and possibly help each other to solve any problems. I know PyArrow is blocking for now, but this is going to be solved soon lilely with https://github.com/apache/arrow/pull/14499 and just opening a failing PR with it might be a good start.

I just opened such PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

https://github.com/apache/airflow/pull/27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)
enhancement help wanted

opened by potiuk 5

📚 Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

Related tags

Overview

Installation

Python Version Support

Usage

Parameterizing a Notebook

Executing a Notebook

Execute via the Python API

Execute via CLI

Supported Name Handlers

Development Guide

Documentation

Comments

🚀 Feature

Motivation

🚀 Feature

Motivation

Owner

nteract

Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

Jupyter notebooks for the code samples of the book "Deep Learning with Python"

Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Code and data for the EMNLP 2021 paper "Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts". Coming soon!

Universal Adversarial Triggers for Attacking and Analyzing NLP (EMNLP 2019)

Code for the paper "Benchmarking and Analyzing Point Cloud Classification under Corruptions"

A framework for analyzing computer vision models with simulated data

Analyzing basic network responses to novel classes

Project page of the paper 'Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network' (ECCVW 2018)

😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

Cross Quality LFW: A database for Analyzing Cross-Resolution Image Face Recognition in Unconstrained Environments

Official repository of the paper "A Variational Approximation for Analyzing the Dynamics of Panel Data". Mixed Effect Neural ODE. UAI 2021.

Source code and notebooks to reproduce experiments and benchmarks on Bias Faces in the Wild (BFW).

Ready-to-use code and tutorial notebooks to boost your way into few-shot image classification.

Repository for scripts and notebooks from the book: Programming PyTorch for Deep Learning

Notebooks em Python para Métodos Eletromagnéticos

Nb workflows - A workflow platform which allows you to run parameterized notebooks programmatically

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.