📚 Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

Related tags

Deep Learning python r scala jupyter pipeline notebook julia nteract publishing notebooks notebook-generator
Overview

CI CI image Documentation Status badge badge PyPI - Python Version Code style: black papermill

papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

Papermill lets you:

  • parameterize notebooks
  • execute notebooks

This opens up new opportunities for how notebooks can be used. For example:

  • Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.
  • Do you want to run a notebook and depending on its results, choose a particular notebook to run next? You can now programmatically execute a workflow without having to copy and paste from notebook to notebook manually.

Papermill takes an opinionated approach to notebook parameterization and execution based on our experiences using notebooks at scale in data pipelines.

Installation

From the command line:

pip install papermill

For all optional io dependencies, you can specify individual bundles like s3, or azure -- or use all. To use Black to format parameters you can add as an extra requires ['black'].

pip install papermill[all]

Python Version Support

This library currently supports Python 3.6+ versions. As minor Python versions are officially sunset by the Python org papermill will similarly drop support in the future.

Usage

Parameterizing a Notebook

To parameterize your notebook designate a cell with the tag parameters.

enable parameters in Jupyter

Papermill looks for the parameters cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with injected-parameters with input parameters in order to overwrite the values in parameters. If no cell is tagged with parameters the injected cell will be inserted at the top of the notebook.

Additionally, if you rerun notebooks through papermill and it will reuse the injected-parameters cell from the prior run. In this case Papermill will replace the old injected-parameters cell with the new run's inputs.

image

Executing a Notebook

The two ways to execute the notebook with parameters are: (1) through the Python API and (2) through the command line interface.

Execute via the Python API

import papermill as pm

pm.execute_notebook(
   'path/to/input.ipynb',
   'path/to/output.ipynb',
   parameters = dict(alpha=0.6, ratio=0.1)
)

Execute via CLI

Here's an example of a local notebook being executed and output to an Amazon S3 account:

$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

NOTE: If you use multiple AWS accounts, and you have properly configured your AWS credentials, then you can specify which account to use by setting the AWS_PROFILE environment variable at the command-line. For example:

$ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

In the above example, two parameters are set: alpha and l1_ratio using -p (--parameters also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters:

$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0

Using -r or --parameters_raw, users can set parameters one by one. However, unlike -p, the parameter will remain a string, even if it may be interpreted as a number or boolean.

$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml

Using -f or --parameters_file, users can provide a YAML file from which parameter values should be read.

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
alpha: 0.6
l1_ratio: 0.1"

Using -y or --parameters_yaml, users can directly provide a YAML string containing parameter values.

$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==

Using -b or --parameters_base64, users can provide a YAML string, base64-encoded, containing parameter values.

When using YAML to pass arguments, through -y, -b or -f, parameter values can be arrays or dictionaries:

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
    - 0.0
    - 1.0
    - 2.0
    - 3.0
linear_function:
    slope: 3.0
    intercept: 1.0"

Supported Name Handlers

Papermill supports the following name handlers for input and output paths during execution:

Development Guide

Read CONTRIBUTING.md for guidelines on how to setup a local development environment and make code changes back to Papermill.

For development guidelines look in the DEVELOPMENT_GUIDE.md file. This should inform you on how to make particular additions to the code base.

Documentation

We host the Papermill documentation on ReadTheDocs.

Comments
  • Unable to write into GCS bucket with papermill[gcs]

    Unable to write into GCS bucket with papermill[gcs]

    When running GCFS application via papermill[gcs]

    papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb

    I'm getting Error: HTTP 429 Rate exceeds.

    Works if output notebook is written locally:

    papermill gs://my-bucket/test.ipynb /tmp/test.ipynb

    Local file size is: 57K

    ls -alh /tmp/test.ipynb 
-rw-r--r--  1 gogasca  wheel    57K Feb 14 10:37 /tmp/test.ipynb

    GCSFS reference https://github.com/dask/gcsfs/issues/130

    How to reproduce?

    pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb

    Logs:

    10
Ending Cell 6------------------------------------------
Exception gcsfs.utils.HtmlError: HtmlError(u'The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.',) in <bound method GCSFile.__del__ of <GCSFile d
pe-sandbox/test.ipynb>> ignored
Traceback (most recent call last):
  File "/usr/local/bin/papermill", line 11, in <module>
    sys.exit(papermill())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/usr/local/lib/python2.7/dist-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-150>", line 2, in close
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1548, in close
    self.flush(force=True)
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-145>", line 2, in flush
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1367, in flush
    self._simple_upload()
  File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-148>", line 2, in _simple_upload
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1465, in _simple_upload
    validate_response(r, path)
  File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 162, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
+ err 'Processing notebook failed'
++ date +%Y-%m-%dT%H:%M:%S%z
+ echo '[2019-02-14T18:32:58+0000]: Processing notebook failed'
[2019-02-14T18:32:58+0000]: Processing notebook failed
+ exit 1

    I already defined:

    export GOOGLE_APPLICATION_CREDENTIALS=/keys/my-project.json
gcloud config set account [email protected]
gcloud auth activate-service-account --key-file=/keys/my-project.json

    in MacOS environment I get similar errors: (Added debugging)

    papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://dpe-sandbox/test.ipynb
Input Notebook:  gs://cloud-samples-data/papermill/samples/test.ipynb
Output Notebook: gs://dpe-sandbox/test.ipynb
('dpe-cloud-mle', 'full_control', None, None, 'none', None)
  0%|                                                                                                   | 0/28 [00:00<?, ?it/s]uploading
11317
uploading
11342
uploading
11372
  4%|███▎                                                                                       | 1/28 [00:01<00:50,  1.88s/it]uploading
11397
uploading
11427
  7%|██████▌                                                                                    | 2/28 [00:02<00:41,  1.59s/it]uploading
11452
uploading
11482
 11%|█████████▊                                                                                 | 3/28 [00:03<00:33,  1.32s/it]uploading
11507
uploading
11537
 14%|█████████████                                                                              | 4/28 [00:04<00:26,  1.10s/it]uploading
11562
429

429
Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 296, in execute_notebook
    nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 352, in execute_managed_notebook
    preprocessor.preprocess(nb_man, kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 27, in preprocess
    nb, resources = self.papermill_process(nb_man, resources)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 81, in papermill_process
    nb_man.cell_complete(nb.cells[index])
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 219, in cell_complete
    self.save()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/bin/papermill", line 10, in <module>
    sys.exit(papermill())
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/cli.py", line 165, in papermill
    cwd=cwd,
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/execute.py", line 90, in execute_notebook
    start_timeout=start_timeout,
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 302, in execute_notebook
    nb_man.notebook_complete()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 249, in notebook_complete
    self.save()
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
    write_ipynb(self.nb, self.output_path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
    return self.get_handler(path).write(buf, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
    return f.write(buf)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
    self.close()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
    self.flush(force=True)
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
    self._simple_upload()
  File "</Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
    return f(self, *args, **kwargs)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
    validate_response(r, path)
  File "/Users/gogasca/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
    bug 
    opened by gogasca 22
  • Read notebook from stdin/write to stdout when path is '-'

    Read notebook from stdin/write to stdout when path is '-'

    This pull requests implements - as stdin/stdout, cf. #372.

    A case when one wants to read notebooks from stdin are Jupytext notebooks (#365). Piping notebooks to stdout can also be useful in case of chained notebook conversions (to html with nbconvert for instance).

    I have included two tests. What remains is probably to

    • [x] review the stdout option, and the tests
    • [ ] mention - as stdin/stdout in Papermill's documentation, or at least in the CLI
    • [x] increase the default value for --log-level when the output path is -

    I have tested the PR with the command below, which prints the notebook (represented as an Hydrogen script) once the injected parameter cell as been added by papermill:

    jupytext notebook_as_script.py --to ipynb -o - | papermill - - --prepare-only -p a 2 --log-level ERROR | jupytext --to py:percent

    where notebook_as_script.py is a notebook represented as a script using Jupytext, e.g.:

    # ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.2'
#       jupytext_version: 1.1.3
#   kernelspec:
#     display_name: Python3
#     language: python
#     name: python3
# ---

# %% {"tags": ["parameters"]}
a = 1

# %%
print('a={}'.format(a))

    As expected, the command above returns the updated representation of the notebook, with an additional cell in between:

    # %% {"tags": ["injected-parameters"]}
# Parameters
a = 2
    opened by mwouts 21
  • ipyleaflet map is not rendered when ruhning in papermill

    ipyleaflet map is not rendered when ruhning in papermill

    I have the following cell:

    from IPython.display import display
from ipyleaflet import Map, Polyline

m = Map(center=(32.2, 34.8), zoom=12)
for r in df.itertuples():
    line = Polyline(
        locations=[[[r.start_latitude, r.start_longitude], [r.end_latitude, r.end_longitude]]],
        weight=15
    )
    m.add_layer(line)

display(m)

    The map is displayed correctly, when running interactively. However, when running the same notebook using papermill it renders just this:

    Map(basemap={'url': 'https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', 'max_zoom': 19, 'attribution': 'Map …
    opened by spektom 20
  • `RuntimeError: Kernel didn't respond in 60 seconds`, when trying to run papermill with python multiprocessing

    `RuntimeError: Kernel didn't respond in 60 seconds`, when trying to run papermill with python multiprocessing

    Hello, I am trying to run multiple parameterized notebooks in parallel. Currently, I am using papermill inside Jupyter Notebook and if I try to use multiprocessing pool to map a list of parameters as pass them to pm.execute_notebook, I get RuntimeError: Kernel didn't respond in 60 seconds. I am running everything with Python 2.7.

    This is the code I use:

    import papermill as pm
import multiprocessing as mp

def run_nb(data):
    d1, d2 = data
    pm.execute_notebook(in_nb, out_nb, parameters = dict(d1=d1, d2=d2) )
    
pool = mp.Pool(4)
pool.map(run_nb, zip(data1, data2))
pool.close()
pool.join()

    It works correctly using the standard python map.

    Btw, is there a known way to produce multiple notebooks in parallel with papermill?

    Thanks!

    upstream 
    opened by franzoni315 18
  • Papermill raises error for id field when running jupyter-notebook 6.2 with the -p flag

    Papermill raises error for id field when running jupyter-notebook 6.2 with the -p flag

    When running papermill v2.2.1 in an environment where jupyter-notebook v6.2 is installed I am seeing the following error when running with parameters (-p flag)

    papermill notebook.ipynb outputs/notebook.ipynb -p my_config --no-progress-bar --log-output

    Output:

    [NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)
[2021-01-18T09:44:24.218Z] Failed validating 'additionalProperties' in code_cell:
[2021-01-18T09:44:24.218Z] On instance['cells'][0]:
[2021-01-18T09:44:24.218Z] {'cell_type': 'code',
[2021-01-18T09:44:24.218Z]  'execution_count': 1,
[2021-01-18T09:44:24.218Z]  'id': 'nasty-bearing',
[2021-01-18T09:44:24.218Z]  'metadata': {'execution': {'iopub.execute_input': '2021-01-18T09:44:22.903942Z',
[2021-01-18T09:44:24.218Z]                             'iopub.status.busy': '2021-01-18T09:44:22.903349Z',
[2021-01-18T09:44:24.218Z]                             'iopub.status.idle': '2021-01-18T09:44:22.905999Z',
[2021-01-18T09:44:24.218Z]                             'shell.execute_reply': '2021-01-18T09:44:22.905474Z'},
[2021-01-18T09:44:24.218Z]               'papermill': {'duration': 0.01294,
[2021-01-18T09:44:24.218Z]                             'end_time': '2021-01-18T09:44:22.906187',
[2021-01-18T09:44:24.218Z]                             'exception': False,
[2021-01-18T09:44:24.218Z]                             'start_time': '2021-01-18T09:44:22.893247',
[2021-01-18T09:44:24.218Z]                             'status': 'completed'},
[2021-01-18T09:44:24.218Z]               'tags': ['injected-parameters']}

    I think this may be due to a change in jupyter-notebook v6.2 that has added an "id" field to the cell properties : https://github.com/jupyter/notebook/pull/5928/ .

    The error does not occur when jupyter-notebook v6.1.6 or earlier is running

    opened by malcolmbovey 16
  • Let user choose how to deal with execution result.

    Let user choose how to deal with execution result.

    Hi, here I add this to let the user choose how to deal with execution result if I use it in an application, I can use variable to deal with the result, no need to read again from the output file. Thanks.

    opened by jarrekk 16
  • Return code of executing notebooks in papermill

    Return code of executing notebooks in papermill

    Hi All,

    Is there a way to capture return code of executing notebooks in papermill API library or CLI methods? When any notebook cell execution encounters an error, it should return an error return code AND when all notebook cells execute successfully it should return a success return code.

    Thanks!

    question 
    opened by vinaykumar80 14
  • expose current output path as $PAPERMILL_OUTPUT_PATH

    expose current output path as $PAPERMILL_OUTPUT_PATH

    As a papermill user I want to expose the output path of the currently run notebook via an environment variable and an API So that the output path is available inside of the executed notebook

    Addresses #231

    The PR adds an interface to Engine to pass environment variables to the executing kernel. This decision was made s.t. remotely executed engines/kernel can make use of this feature, e.g., the kaggle engine from #229. Whether environment variables are the way to go is up for discussion.

    opened by mindsbackyard 14
  • error message suddenly popped up

    error message suddenly popped up

    Hi,

    I have been using papermill and NbConvert to parameterize and convert Jupyter Notebooks. It worked really well until recently I started getting the following messages:

    /home/florathecat/anaconda3/lib/python3.6/site-packages/nbconvert/filters/datatypefilter.py:41: UserWarning: Your element with mimetype(s) dict_keys(['application/papermill.record+json']) is not able to be represented. mimetypes=output.keys())

    The notebooks are still parameterized and converted fine. It is just the error message bugs me. I have tried to update conda, python, nbconvert, and reinstall papermill. None of these seem to work. I'll appreciate if somebody can help.

    Yun

    opened by florathecat 13
  • Some weird error messages when executing a notebook involving pytorch

    Some weird error messages when executing a notebook involving pytorch

    I have a notebook for training a model using pytorch. The notebook runs fine if I run it from browser. But I ran into the following problem when executing it via papermill

    Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/Grammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp0twtlmvs'
Generating grammar tables from /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /home/ubuntu/.cache/black/20.8b1/PatternGrammar3.6.10.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/ubuntu/.cache/black/20.8b1/tmp2_jvdud_'
Executing:   0%|                                                                                   | 0/23 [00:00<?, ?cell/s]Executing notebook with kernel: pytorch_p36
Executing:  22%|████████████████▎                                                          | 5/23 [00:03<00:13,  1.38cell/s]
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/cli.py", line 256, in papermill
    execution_timeout=execution_timeout,
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 118, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/papermill/execute.py", line 230, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-3916aaf64ab2> in <module>
----> 1 from torchvision import datasets, transforms
      2 
      3 datasets.MNIST('data', download=True, transform=transforms.Compose([
      4     transforms.ToTensor(),
      5     transforms.Normalize((0.1307,), (0.3081,))

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/__init__.py in <module>
      1 import warnings
      2 
----> 3 from torchvision import models
      4 from torchvision import datasets
      5 from torchvision import ops

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/__init__.py in <module>
     10 from .shufflenetv2 import *
     11 from . import segmentation
---> 12 from . import detection
     13 from . import video
     14 from . import quantization

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/__init__.py in <module>
----> 1 from .faster_rcnn import *
      2 from .mask_rcnn import *
      3 from .keypoint_rcnn import *

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py in <module>
     11 
     12 from .generalized_rcnn import GeneralizedRCNN
---> 13 from .rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
     14 from .roi_heads import RoIHeads
     15 from .transform import GeneralizedRCNNTransform

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/rpn.py in <module>
      9 from torchvision.ops import boxes as box_ops
     10 
---> 11 from . import _utils as det_utils
     12 from .image_list import ImageList
     13 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py in <module>
     17 
     18 @torch.jit.script
---> 19 class BalancedPositiveNegativeSampler(object):
     20     """
     21     This class samples batches, ensuring that they contain a fixed proportion of positives

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1217         if _rcb is None:
   1218             _rcb = _jit_internal.createResolutionCallback(_frames_up + 1)
-> 1219         _compile_and_register_class(obj, _rcb, qualified_name)
   1220         return obj
   1221     else:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in _compile_and_register_class(obj, rcb, qualified_name)
   1074 def _compile_and_register_class(obj, rcb, qualified_name):
   1075     ast = get_jit_class_def(obj, obj.__name__)
-> 1076     _jit_script_class_compile(qualified_name, ast, rcb)
   1077     _add_script_class(obj, qualified_name)
   1078 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in try_compile_fn(fn, loc)
    220     # object
    221     rcb = _jit_internal.createResolutionCallbackFromClosure(fn)
--> 222     return torch.jit.script(fn, _rcb=rcb)
    223 
    224 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)
   1224         if _rcb is None:
   1225             _rcb = _gen_rcb(obj, _frames_up)
-> 1226         fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
   1227         # Forward docstrings
   1228         fn.__doc__ = obj.__doc__

RuntimeError: 
builtin cannot be used as a value:
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:14:56
def zeros_like(tensor, dtype):
    # type: (Tensor, int) -> Tensor
    return torch.zeros_like(tensor, dtype=dtype, layout=tensor.layout,
                                                        ~~~~~~~~~~~~~ <--- HERE
                            device=tensor.device, pin_memory=tensor.is_pinned())
'zeros_like' is being compiled since it was called from '__torch__.torchvision.models.detection._utils.BalancedPositiveNegativeSampler.__call__'
at /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torchvision/models/detection/_utils.py:72:12

            # randomly select positive and negative examples
            perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
            perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

            pos_idx_per_image = positive[perm1]
            neg_idx_per_image = negative[perm2]

            # create binary mask from indices
            pos_idx_per_image_mask = zeros_like(
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
                matched_idxs_per_image, dtype=torch.uint8
            )
            neg_idx_per_image_mask = zeros_like(
                matched_idxs_per_image, dtype=torch.uint8
            )

            pos_idx_per_image_mask[pos_idx_per_image] = torch.tensor(1, dtype=torch.uint8)
            neg_idx_per_image_mask[neg_idx_per_image] = torch.tensor(1, dtype=torch.uint8)
    question 
    opened by hsl89 12
  • Include link to the cell with an error, in failure heading

    Include link to the cell with an error, in failure heading

    Papermill currently inserts an error message An Exception was encountered at 'In [NNN]' at the top of a notebook that fails to execute, where NNN is the cell execution count of the failing cell. This patch adjusts the In [NNN] text to be a link to the relevant cell, to make it easy to jump to. This works by inserting another markdown cell immediately before the failing cell, where the markdown cell contains an HTML anchor (specifically, id="papermill-error-cell"). The heading then links to it with a #papermill-error-cell fragment HREF.

    This helps reduce the scrolling and searching required for long notebooks where the error cell might be one of dozens, or hidden among long output. For an example of the latter, see https://nbviewer.jupyter.org/gist/huonw/dd13fa49b5876bfea26b69832702d573.

    opened by huonw 12
  • ImportError: attempted relative import with no known parent package

    ImportError: attempted relative import with no known parent package

    I'm trying to turn into a pip package a notebook which runs using Papermill. There are python file local to the package that I want to import into the notebook but it's giving me an error:

    # SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
# sys.path.append(os.path.dirname(SCRIPT_DIR))
from . import utils, preprocessing as prep
ImportError: attempted relative import with no known parent package

    This way of importing gives this error whether I try to run the notebook manually and through Papermill.

    I've also tried importing it directly:

    import utils, preprocessing as prep
ModuleNotFoundError: No module named 'utils'

    The direct import runs fine if I execute the notebook manually, but when I try to run them through Papermill it doesn't work.

    In my package parent I have

    import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__)))

from . import preprocessing
from . import utils
    bug help wanted 
    opened by danre07 1
  • Papermill is not working with an error of click/zmq/nbclient

    Papermill is not working with an error of click/zmq/nbclient

    Hi I'm trying to build a new environment. But I ran into a problem which the papermill is not working. My .ipynb file is really simple. only has one single line of print(3). I can't find anything related to this Traceback in Google. Does anyone know how to solve it?

    Sorry I can't provide the full list of my package and their versions. That would be too long. I just leave a version list of the package related at the very end of this page. Let me know if you need more information about it.

    Traceback (most recent call last):
  File "/opt/conda/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/cli.py", line 267, in papermill
    execution_timeout=execution_timeout,
  File "/opt/conda/lib/python3.7/site-packages/papermill/execute.py", line 124, in execute_notebook
    **engine_kwargs
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 367, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "/opt/conda/lib/python3.7/site-packages/papermill/clientwrap.py", line 43, in execute
    with self.setup_kernel(**kwargs):
  File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.7/site-packages/nbclient/client.py", line 562, in setup_kernel
    self.start_new_kernel_client()
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/opt/conda/lib/python3.7/site-packages/nbclient/client.py", line 532, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/opt/conda/lib/python3.7/site-packages/nbclient/util.py", line 96, in ensure_async
    result = await obj
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/client.py", line 187, in _async_wait_for_ready
    msg = await self.shell_channel.get_msg(timeout=1)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/channels.py", line 224, in get_msg
    ready = await self.socket.poll(timeout)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 387, in poll
    f = cast(Future, p.poll(timeout))
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 128, in poll
    socket._add_recv_event('poll', future=watcher)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 493, in _add_recv_event
    self._add_io_state(POLLIN)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 663, in _add_io_state
    self._update_handler(self._state)
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 679, in _update_handler
    self._get_loop()
  File "/opt/conda/lib/python3.7/site-packages/zmq/_future.py", line 62, in _get_loop
    self._init_io_state(self._current_loop)
TypeError: _init_io_state() takes 1 positional argument but 2 were given
Could not destroy zmq context for <jupyter_client.asynchronous.client.AsyncKernelClient object at 0x7fa8e282dc10>

    here are packages versions which might related to the error:

    Python                    3.7.13
click                     8.0.4 
papermill                 2.4.0  
nbclient                  0.5.13
jupyter_client            5.3.4
pyzmq                     24.0.1
    bug help wanted 
    opened by corvettettt 0
  • Remove parameters from notebook output

    Remove parameters from notebook output

    🚀 Feature

    It would be nice to be able to remove parameters from the outputted notebook

    Motivation

    I have several parameters which need secrets e.g. API keys, passwords, etc All my parameters come from static places like SecretsManager or Env or Config file

    From what I understand I'd have to modify papermill.execute to make another transformation which removes parameters cell, correct?

    similar to this https://github.com/nteract/papermill/blob/54f6c038cdae0c70d5fb04691fa465e12aeb62cb/papermill/execute.py#L136

    enhancement help wanted 
    opened by 1oglop1 0
  • Is it possible to choose which cells to run using papermill.execute_notebook?

    Is it possible to choose which cells to run using papermill.execute_notebook?

    I use papermill to run a jupyter notebook. Is it possible that I can choose which cell to run, and which not to run using papermill?

    papermill.execute_notebook(
                    'notebooks.ipynb',
                    None,
                    parameters=dict(params=params),
                    log_output=True
                )
    opened by wxy347 1
  • Support for Python 3.11

    Support for Python 3.11

    🚀 Feature

    I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

    Motivation

    The papermill is one of the dependencies of Airlfow that need to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

    Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.

    I'd appreciate if someone in the ntarct team attempted to migrate and have a PR running (and failing possibly until all prerequisite are met so that we can also track the progress and possibly help each other to solve any problems. I know PyArrow is blocking for now, but this is going to be solved soon lilely with https://github.com/apache/arrow/pull/14499 and just opening a failing PR with it might be a good start.

    I just opened such PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

    https://github.com/apache/airflow/pull/27264

    I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

    Looking forward to cooperation on that one :)

    enhancement help wanted 
    opened by potiuk 5
Owner
nteract
Interactive computing experiences that allow people to collaborate with ease
nteract
GitHub http://papermill.readthedocs.io/en/latest/
Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

Graph Notebook: easily query and visualize graphs The graph notebook provides an easy way to interact with graph databases using Jupyter notebooks. Us

Amazon Web Services 501 Dec 28, 2022
Jupyter notebooks for the code samples of the book "Deep Learning with Python"

Jupyter notebooks for the code samples of the book "Deep Learning with Python"

François Chollet 16.2k Dec 30, 2022
Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Kim Seonghyeon 2.2k Jan 1, 2023
Code and data for the EMNLP 2021 paper "Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts". Coming soon!

ToxiChat Code and data for the EMNLP 2021 paper "Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts". Install depen

Ashutosh Baheti 11 Jan 1, 2023
Universal Adversarial Triggers for Attacking and Analyzing NLP (EMNLP 2019)

Universal Adversarial Triggers for Attacking and Analyzing NLP This is the official code for the EMNLP 2019 paper, Universal Adversarial Triggers for

Eric Wallace 248 Dec 17, 2022
Code for the paper "Benchmarking and Analyzing Point Cloud Classification under Corruptions"

ModelNet-C Code for the paper "Benchmarking and Analyzing Point Cloud Classification under Corruptions". For the latest updates, see: sites.google.com

Jiawei Ren 45 Dec 28, 2022
A framework for analyzing computer vision models with simulated data

3DB: A framework for analyzing computer vision models with simulated data Paper Quickstart guide Blog post Installation Follow instructions on: https:

3DB 112 Jan 1, 2023
Analyzing basic network responses to novel classes

novelty-detection Analyzing how AlexNet responds to novel classes with varying degrees of similarity to pretrained classes from ImageNet. If you find

Noam Eshed 34 Oct 2, 2022
Project page of the paper 'Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network' (ECCVW 2018)

EPSR (Enhanced Perceptual Super-resolution Network) paper This repo provides the test code, pretrained models, and results on benchmark datasets of ou

Subeesh Vasu 78 Nov 19, 2022
😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

------ Update September 2018 ------ It's been a year since TorchMoji and DeepMoji were released. We're trying to understand how it's being used such t

Hugging Face 865 Dec 24, 2022
Cross Quality LFW: A database for Analyzing Cross-Resolution Image Face Recognition in Unconstrained Environments

Cross-Quality Labeled Faces in the Wild (XQLFW) Here, we release the database, evaluation protocol and code for the following paper: Cross Quality LFW

Martin Knoche 10 Dec 12, 2022
Official repository of the paper "A Variational Approximation for Analyzing the Dynamics of Panel Data". Mixed Effect Neural ODE. UAI 2021.

Official repository of the paper (UAI 2021) "A Variational Approximation for Analyzing the Dynamics of Panel Data", Mixed Effect Neural ODE. Panel dat

Jurijs Nazarovs 7 Nov 26, 2022
Source code and notebooks to reproduce experiments and benchmarks on Bias Faces in the Wild (BFW).

Face Recognition: Too Bias, or Not Too Bias? Robinson, Joseph P., Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. "Face recognition:

Joseph P. Robinson 41 Dec 12, 2022
Ready-to-use code and tutorial notebooks to boost your way into few-shot image classification.

Easy Few-Shot Learning Ready-to-use code and tutorial notebooks to boost your way into few-shot image classification. This repository is made for you

Sicara 399 Jan 8, 2023
Repository for scripts and notebooks from the book: Programming PyTorch for Deep Learning

Repository for scripts and notebooks from the book: Programming PyTorch for Deep Learning

Ian Pointer 368 Dec 17, 2022
Notebooks em Python para Métodos Eletromagnéticos

GeoSci Labs This is a repository of code used to power the notebooks and interactive examples for https://em.geosci.xyz and https://gpg.geosci.xyz. Th

Victor Cezar Tocantins 1 Nov 16, 2021
Nb workflows - A workflow platform which allows you to run parameterized notebooks programmatically

NB Workflows Description If SQL is a lingua franca for querying data, Jupyter sh

Xavier Petit 6 Aug 18, 2022
The tool under this branch fork can be used to crack devices above A12 and up to A15. After cracking, you can also use SSH channel strong opening tool to open SSH channel and activate it with Demo or Shell script. The file can be extracted from my Github homepage, and the SSH channel opening tool can be extracted from Dr238 account.

Welcome to C0xy-A12-A15-Attack-Tool The tool under this branch fork can be used to crack devices above A12 and up to A15. After cracking, you can also

null 13 Dec 23, 2022
A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

Eugenio Herrera 175 Dec 29, 2022