Mercury: easily convert Python notebook to web app and share with others

MLJAR

Last update: Dec 27, 2022

Related tags

Deep Learning python data-science jupyter jupyter-notebook data-visualization notebook-web notebook-application notebook-jupyter jupyter-lab notebook-publish notebooks-jupyter

Overview

Mercury

Share your Python notebooks with others

Easily convert your Python notebooks into interactive web apps by adding parameters in YAML.

Simply add YAML with description of parameters needed in the notebook.
Share notebook with others.
Allow them to execute notebook with selected parameters.
You can decide to show or hide your code.
Easily deploy to the server.

Mercury is a perfect tool to share your Python notebooks with non-programmers.

You can turn your notebook into web app. Sharing is as easy as sending them the URL to your server.
You can add interactive input to your notebook by defining the YAML header. Your users can change the input and execute the notebook.
You can hide your code to not scare your (non-coding) collaborators.
Users can interact with notebook and save they results.
You can share notebook as a web app with multiple users - they dont ovewrite original notebook.

Example

Notebook with YAML config

The YAML config is added as a first raw cell in the notebook.

Web Application from Notebook

The web app generated from the notebook. Code is hidden (optional). User can change parameters, execute notebook with Run button, and save result with Download button.

Check our demo

The demo with several example notebooks is running at http://mercury.mljar.com (running on AWS EC2 t3a.small instance). No need to register.

The demo running at Heroku free dyno http://mercury-demo-1.herokuapp.com (with several notebooks-apps). No need to register. (if dyno is sleeping and notebooks are not loaded, please refresh it and wait a little)

Share mutliple notebooks

You can share as many notebooks as you want. There is gallery with notebooks in the home view of the Mercury. You can select any notebook by clicking Open button.

Convert Notebook to web app with YAML

You need to add YAML at the beginning of the notebook to be able to run it as web application in the Mercury. The YAML configuration should be added in the notebook as Raw cell. It should start and end with a line containing "---". Below examples how it should look like in the Jupyter Notebook and Jupyter Lab:

Allowed parameters in YAML config:

title - string with a title of the notebook. It is used in the app side bar and in the gallery view.
author - string with a author name (optional).
description - string describing the content of the notebook. It is used in the gallery view.
show-code - can be True or False. Default is set to True. It decides if the notebook's code will be displayed or not.
show-prompt - can be True or False. Default is set to True. If set to True the prompt information will be displayed for each cell in the notebook.
params - the parameters that will be used in the notebook. They will be displayed as interactive widgets in the side bar. Each parameter should have unique name that correspont to the variable name used in the code.

Define widget with YAML

Widget name is a variable name

Definition of the widget (in params) starts with the widget name. It will correspond to the variable in the code. The name should be a valid Python variable.

Widget input type

The next thing is to select the input type. It can be: slider, range, select, checkbox, numeric.

Widget label

For each input we need to define a label. It will be a text displayed above (or near) the widget.

Widget default value

You can set a default widget by setting the value. The format of the value depends on the input type:

for slider a value should be a number, example: value: 5,
for range a value should be a list with two numbers, example value: [3,6],
for select with mutli: False a value should be a string, example value: hey,
for select with multi: True a value should be a list of strings, example value: [cześć, hi, hello],
for checkbox a value should be a boolean (True or False), example: value: True,
for numeric a value should be a number, example: value: 10.2.

The rest of parameters depends on widget input type.

Slider

Additional parameters:

min - the minimum value for slider (default is set to 0),
max - the maximum value for slider (default is set to 100).

Example YAML:

params:
    my_variable:
        input: slider
        label: This is slider label
        value: 5
        min: 0
        max: 10

Range

Additional parameters:

min - the minimum value for slider (default is set to 0),
max - the maximum value for slider (default is set to 100).

Example YAML:

params:
    range_variable:
        input: range
        label: This is range label
        value: [3,6]
        min: 0
        max: 10

Select

Additional parameters:

multi - a boolean value that decides if user can select several options (default is set to False).
choices - a list with available choices.

Example YAML:

params:
    select_variable:
        label: This is select label
        input: select
        value: Cześć
        choices: [Cześć, Hi, Hello]
        multi: False

Checkbox

There are no additional parameters.

Example YAML:

params:
    checkbox_variable:
        label: This is checkbox label
        input: checkbox
        value: True

Numeric

Additional parameters:

min - a minimum allowed value (default set to 0),
max - a maximum allowed value (default set to 100),
step - a step value (default set to 1).

params:
    numeric_variable:
        label: This is numeric label
        input: numeric
        value: 5.5
        min: 0
        max: 10
        step: 0.1

Full example YAML

---
title: My notebook
author: Piotr
description: My first notebook in Mercury
params:
    my_variable:
        label: This is slider label
        input: slider
        value: 5
        min: 0
        max: 10
    range_variable:
        label: This is range label
        input: range
        value: [3,6]
        min: 0
        max: 10    
    select_variable:
        label: This is select label
        input: select
        value: Cześć
        choices: [Cześć, Hi, Hello]
        multi: False
    checkbox_variable:
        label: This is checkbox label
        input: checkbox
        value: True
    numeric_variable:
        label: This is numeric label
        input: numeric
        value: 5.5
        min: 0
        max: 10
        step: 0.1
---

Widgets rendered from above YAML config:

Use variables in the code

To use variables in the code simply define the variable with the same name as widget name. You can also assign the same value as defined in YAML. Please define all variables in the one cell (it can be below the cell with YAML config).

When the user will interact with widgets and click the Run button, the code with variables will be updated with user selected values.

Example:

Installation

You can install Mercury directly from PyPi repository with pip command:

pip install mljar-mercury

Installation from GitHub:

pip install -q -U git+https://github.com/mljar/mercury.git@master

Running locally

To run Mercury locally just run:

mercury runserver --runworker

The above command will run server and worker. It will serve Mercury website at http://127.0.0.1:8000. It won't display any notebooks because we didn't add any. Please stop the Mercury server (and worker) for a moment with (Ctrl+C).

Execute the following command to add notebook to Mercury database:

mercury add <path_to_notebook>

Please start the Mercury server to see your apps (created from notebooks).

mercury runserver --runworker

Notebook development with automatic refresh

The Mercury watch command is perfect when you are creating a new notebook and would like to see how it will look like as web app with live changes.

Please run the following command:

mercury watch <path_to_your_notebook>

You can now open the web browser at http://127.0.0.1:8000 and find your notebook. When you change something in the notebook code, markdown or YAML configuration and save the notebook, then it will be automatically refreshed in the web browser. You can track your changes without manual refreshing of the web app.

Running in production

Running in production is easy. We provide several tutorials how it can be done.

Deploy to Heroku using free dyno
Deploy to AWS EC2 using t2.micro
Deploy to Digital Ocean (comming soon)
Deploy to GCP (comming soon)
Deploy to Azure (comming soon)

Running with `docker-compose`

The docker-compose must be run from the Mercury main directory.

Please copy .env.example file and name it .env file. Please point the NOTEBOOKS_PATH to the directory with your notebooks. All notebooks from that path will be added to the Mercury before server start. If the requirements.txt file is available in NOTEBOOKS_PATH all packages from there will be installed.

Please remember to change the DJANGO_SUPERUSER_USERNAME and DJANGO_SUPERUSER_PASSWORD.

To generate new SECRET_KEY (recommended) you can use:

python -c 'from django.core.management.utils import get_random_secret_key; \
            print(get_random_secret_key())'

Please leave SERVE_STATIC=False because in the docker-compose configuration static files are served with nginx.

The docker-compose will automatically read environment variables from .env file. To start the Mercury please run:

docker-compose up --build

To run in detached mode (you can close the terminal) please run:

docker-compose up --build -d

To stop the containers:

docker-compose down

Mercury development

The Mercury consists of three elements:

Frontend written in TypeScript with React+Redux
Server written in Python with Django
Worker written in Python with Celery

Each element needs a separate terminal during development.

Frontend

The user interface code is in the frontend directory. Run all commands from there. Install dependencies:

yarn install

Run frontend:

yarn start

The frontend is served at http://localhost:3000.

Server

The server code is in the mercury directory. Run all commands from there. Please set the virtual environment first:

virtualenv menv
source menv/bin/activate
pip install -r requirements.txt

Apply migrations:

python manage.py migrate

Run the server in development mode (DEBUG=True):

python manage.py runserver

The server is running at http://127.0.0.1:8000.

Worker

The worker code is in the mercury directory (in the apps/notebooks/tasks.py and apps/tasks/tasks.py files). Please activate first the virtual environment (it is using the same virtual environment as server):

source menv/bin/activate

Run the worker:

celery -A server worker --loglevel=info -P gevent --concurrency 1 -E

Mercury Pro

Looking for dedicated support, commercial friendly license and more features? The Mercury Pro is for you. Please see the details at our website.

Comments

Reading notebook without utf-8 encoding

Hi,

Thank you for your amazing work !

I have a problem when I try to convert my notebook with mercury run, I have the following message : Error during notebook initialization. 'charmap' codec can't decode byte 0x9d in position 3522922: character maps to

the notebook runs fine in jupyter and I have no issues ... I can't understand from where it can come from ...

Please help !

Thank you in advance.

Best Regards
bug

opened by doubianimehdi 18
Unable to run docker container (missing ipython_genutils package)

@pplonski With a clone of the repo, I am unable to run a docker container containing the mercury_demo .

The message Problem while loading notebooks. Please try again later or contact Mercury administrator. displays Steps to reproduce:

Clone https://github.com/MarvinKweyu/mercury-docker-demo and run the container
bug

opened by MarvinKweyu 16

Error while writing the code in python functions

Hi, My Jupiter notebook code looks like follows:

---
title: My first notebook
description: my first notebook on mercury
show-code: False
params:
    a:
        input: slider
        label: This is slider label
        value: 5
        min: 0
        max: 100
    b:
        label: This is numeric label
        input: numeric
        value: 5.5
        min: 0
        max: 100
        step: 1
---
a=5
b=6
def add(a,b):
    return a+b
print(add(a+b))

while running it in the web app it gives an error add is not defined. can any one help me with how to overcome this?

opened by hiteshpara 16

can't install mljar mercury with python 3.9.11 , unicode error

Hi !

I'm trying to install mljar mercury on my new pc and I have the following error : Collecting mljar-mercury Using cached mljar-mercury-0.6.8.tar.gz (2.1 MB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [18 lines of output] Traceback (most recent call last): File "C:\Users\doub2420.virtualenvs\wos_parser-3ahBjB9p\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 363, in main() File "C:\Users\doub2420.virtualenvs\wos_parser-3ahBjB9p\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 345, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "C:\Users\doub2420.virtualenvs\wos_parser-3ahBjB9p\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 130, in get_requires_for_build_wheel return hook(config_settings) File "C:\Users\doub2420\AppData\Local\Temp\pip-build-env-1tu8thb3\overlay\Lib\site-packages\setuptools\build_meta.py", line 177, in get_requires_for_build_wheel return self._get_build_requires( File "C:\Users\doub2420\AppData\Local\Temp\pip-build-env-1tu8thb3\overlay\Lib\site-packages\setuptools\build_meta.py", line 159, in _get_build_requires self.run_setup() File "C:\Users\doub2420\AppData\Local\Temp\pip-build-env-1tu8thb3\overlay\Lib\site-packages\setuptools\build_meta.py", line 174, in run_setup exec(compile(code, file, 'exec'), locals()) File "setup.py", line 5, in long_description = fh.read() File "C:\Users\doub2420\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 3796: character maps to [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I have python 3.9.11 Same error using pip ...

Thank you !
bug

opened by doubianimehdi 14
TemplateDoesNotExist at / error

I got TemplateDoesNotExist at / error. Both the code in github main repo and pypi repo have same error. It may be related to 0.3.0 release. Can you check it out please? This is an internal server error related django.

opened by nuhyurduseven 13

Invalid IP and impossible to launch mercury run demo

Hello,

I installed mercury on my pc with the command :

pip install mercury-mljar

The installation finishes but I can't mercury run demo. The text that appears is the following:

  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python310\Scripts\mercury.exe\__main__.py", line 7, in <module>
  File "C:\Python310\lib\site-packages\mercury\mercury.py", line 56, in main
    create_demo_notebook("demo.ipynb")
  File "C:\Python310\lib\site-packages\mercury\demo.py", line 38, in create_demo_notebook
    nbf.write(nb, f)
  File "C:\Python310\lib\site-packages\nbformat\__init__.py", line 200, in write
    fp.write(s)
  File "C:\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f680' in position 129: character maps to <undefined>

I also tried to launch the application by going to http://127.0.0.1:8000 but this site is inaccessible.

I have python 3.10.4 and windows 10.

Thanks in advance if you can help me 😃

Nicolas

bug

opened by Konsilion 10

Nothing appears on the Mercury web page（but appears admin page）.

Hello,everyone.

I am contacting you to ask for your help.

Done： git clone https://github.com/mljar/mercury.git docker-compose up --build There do not seem to be any errors.

And I access to localhost(or 127.0.0.1), But Nothing appears on the web page....

I can access localhost/admin and login.

Can you please tell us what the solution is? By the way, the HTML elements is written as follows (example/to...??)

##.env DEBUG=False SERVE_STATIC=False DJANGO_SUPERUSER_USERNAME=user DJANGO_SUPERUSER_PASSWORD=pass [email protected] SECRET_KEY="w6g9izv(!xoi(*1m!xv@w0^7%au&lu+j_hb62(#0$-n%nasc#8" ALLOWED_HOSTS=0.0.0.0,localhost NOTEBOOKS_PATH=../mercury_demo_1/ TIME_ZONE=Europe/Warsaw

opened by gmoriki 8
Opening / refreshing Mercury page automatically downloads `html` file
Hi @pplonski , thank you for the wonderful tool.

I built a test page and noticed that every time the page loads (in either chrome or edge) it downloads a file called html which contains the following:

<style type="text/css"> .jp-mod-noOutputs { padding: 0px; } .jp-mod-noInput { padding-top: 0px; padding-bottom: 0px; } </style>

Happy to help debug or test if you cannot reproduce on your end!

I am using version: 0.3.0

Cheers, John
bug
opened by jvivian 8

Js resource access

I am have issues in rendering ipydatagrid within your excellent mercury framework. The jupyter notebook renders the datagrid outside of mercury but when run in mercury it appears not to be able to find the .js resources.

I get the following errors

[Open Browser Console for more detailed log - Double click to close this message]
Failed to create view for 'DataGridView' from module 'ipydatagrid' with model 'DataGridModel' from module 'ipydatagrid'
TypeError: c.JupyterPhosphorPanelWidget is not a constructor
    at g._createElement (https://cdn.jsdelivr.net/npm/ipydatagrid@%5E1.1.12/dist/index.js:2:997709)

looking at the console I see the following

Failed to load resource: the server responded with a status of 404 (Not Found)
:8988/media/temp/ipydatagrid.js:1
Falling back to https://cdn.jsdelivr.net/npm/ for [ipydatagrid@^1.1.12](mailto:ipydatagrid@%5e1.1.12)

not sure how to resolve this - any suggestions?

Is it possible to make the js files static so they do not need to be downloaded?

Thanks

David

opened by beebeed 7

Uploaded file not showing up in dir? (running Mercury locally)

Hello - I've been trying to set up Mercury for my notebook and can't seem to get the upload-file functionality to work. The documentation mentions that any file that's uploaded through the widget will appear in the same directory as the notebook, but nothing is appearing.

Here is what the (locally run) Mercury UI looks like after I drag a test file ('mushroom.csv') into the upload widget:

And here is what appears after I hit 'Run':

I'd assumed that "3_mushroom.csv" should appear in my notebook's directory, but I'm not seeing it there. Maybe I should be looking elsewhere?

I am also seeing this in the console when I open up dev mode on the page, but don't know how to interpret (or if it's even related to the problem I'm having):

(I'm not very familiar with web app dev, so forgive me if I'm missing something completely obvious.)

opened by MogwaiMomo 7
mercury.exe instead of mercury

During: mercury watch

c:\programdata\anaconda3\python.exe: can't open file 'C:\Users<username>\AppData\Roaming\Python\Python38\Scripts\mercury': [Errno 2] No such file or directory

instead mercury it should look for mercury.exe

I have to copy paste mercury.exe and rename it to mercury to make it work

temporary fix:
bug

opened by shivendra2015iiit 7
Dynamic widget choices?

Cześć :) I was wondering if it would be possible to refer callable/function returning a list instead of providing a hard-coded list of choices to Drop down widget YAML definition? Idea being to do some queries or database call in that function to create list of choices dynamically. I appreciate if code is run top to bottom then potentially the choices would be then generated after first notebook run, but this is something that can be dealt with.

Not sure if this would be already somehow supported, if not could you please describe a little how Mercury does that first RAW cell parsing with some reference to the code, maybe it is something that could be added?

opened by simplynail 4
Heroku: Add multiple repos to single app

Mercury CLI has the command add to add a notebook file into the server as one of the apps it can run. Is it possible to do this kind of deployment on Heroku? Say, if I wanted to have a gallery of apps created by students where each student has their own repo?

opened by connorferster 3

Releases(v1.1.6)

v1.1.6(Sep 2, 2022)
Bug fixes

#140, #141 fix numeric input

Source code(tar.gz)
Source code(zip)
v1.1.5(Jul 28, 2022)
Bug fixes

(#151) fix windows path in output files

Source code(tar.gz)
Source code(zip)
v1.1.4(Jul 28, 2022)
Enhacements

(#149) allow colon in text input

Source code(tar.gz)
Source code(zip)
v1.1.3(Jul 12, 2022)
Enhancements

(#137) don't switch to default view between notebooks runs

Bug fixes

(#126) remove blinking of history execution

(#131) remove old auth token from localStorage

(#135) disable execution history from watch mode

Source code(tar.gz)
Source code(zip)
v1.1.2(Jul 11, 2022)
Enhancements

(#75) use PostreSQL database in docker-compose deployments

Bug fixes

(#122) improve validation for numeric input

(#125) add padding-bottom for change password button

Source code(tar.gz)
Source code(zip)
v1.1.1(Jul 8, 2022)
Bug fixes

(#123) fix static files versions

(#124) add redirects for login and account addresses

Source code(tar.gz)
Source code(zip)
v1.1.0(Jul 1, 2022)
Enhancements

(#87) add markdown text to the sidebar

(#115) show execution history

Bug fixes

(#116) execute with empty list in multi select

(#109) fix URL in HuggingFace deployments

(#95) fix typo in ClearTaskView

Docs

(#108, #55) improve docs :)

Execution history demo

Source code(tar.gz)
Source code(zip)
v1.0.1(Jun 22, 2022)
Bug fixes

#113 fix typo in email settings

Source code(tar.gz)
Source code(zip)
v1.0.0(Jun 1, 2022)
Enhancements

#97 - schedule notebooks

#99 - send email notifications about scheduled notebooks status

#98 - export notebook to PDF

#86 - move output files to sidebar

#105 - add information about presentation shortcuts in the sidebar

Source code(tar.gz)
Source code(zip)
v0.8.6(May 25, 2022)
Enhancements

#98 export notebook as PDF

Source code(tar.gz)
Source code(zip)
v0.8.0(May 18, 2022)
Enhancements

#97 Schedule notebooks

Source code(tar.gz)
Source code(zip)
v0.7.0(May 6, 2022)
Enhancements

(#8) Notebook as REST API. You can easily execute the notebook as REST API and return JSON response. Please see the docs.

Source code(tar.gz)
Source code(zip)
v0.6.10(Apr 19, 2022)
Enhancements

(#85) add option to deploy Mercury based app made from Python notebook to the HuggingFace Spaces

Source code(tar.gz)
Source code(zip)
v0.6.8(Apr 11, 2022)
Fixes

Update nbconvert version to >=6.5.0

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 21, 2022)
Enhancements

#56 #57 added authentication, user can select with who to share the notebook

#67 allow embedding

#62 add house icon

#63 add a message about private notebooks in case of the pro version

#64 #65 add certbot for SSL certificate issue and renewal to docker-compose

Bug fixes

#58 fix problem with notebook reading (thank you @doubianimehdi)

#66 add missing package ipython_genutils (thank you @MarvinKweyu)

Source code(tar.gz)
Source code(zip)
v0.5.1(Feb 18, 2022)
Realease 0.5.1

Enhancements

(#47) Add run command - it greatly simplify the process of deployment on Heroku

(#48) Add demo option in run command

(#39) Add anaconda recipe

Source code(tar.gz)
Source code(zip)
v0.4.1(Feb 17, 2022)
Release 0.4.1

Bug fixes

(#45) Fixed: Mercury doesn't show the executed notebook if error occurred

Source code(tar.gz)
Source code(zip)
v0.4.0(Feb 10, 2022)
Release 0.4.0

Enhancements

(#37) Add option to create output files for download

(#18) Add delete and list commands

(#36) Add with the same notebook path will update existing notebook

(#3) Add Clear tasks button to remove previous runs

Bug fixes

(#35) Add exe extension to mercury binary on Windows

Source code(tar.gz)
Source code(zip)
v0.2.0(Jan 21, 2022)

Added file and text input widgets.
Source code(tar.gz)
Source code(zip)