Data App Performance Tests

Overview

Data App Performance Tests

My hypothesis is that

The different architectures of Dash, Panel and Streamlit makes a difference if you want to create snappy applications.

Framework Server Communication Protocol Built in state Update cycle
Dash Flask http No Specific code reruns and UI updates
Panel Tornado web sockets Yes, Automatic Specific code reruns and UI updates
Streamlit Tornado web sockets Yes, Manual Rerun script top to bottom with caching

I want to test that hypothesis and elaborate on it.

Test Setup

In order to fairly and reproducibly compare the frameworks I will pin requirements as much as possible and run in Docker.

Docker

You can build with

docker build -f "Dockerfile" -t dataappperformance:latest "."

and run interactively with

docker run --rm -it  -p 80:80 dataappperformance:latest

You can list all tests with

$ docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke --list'
Available tasks:

  page-load.dash
  page-load.panel
  page-load.streamlit
  slider-plot.dash
  slider-plot.panel
  slider-plot.streamlit

So to run for example the Panel slider-plot app you would run

docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.panel'

To run the page load test you would have to add port 8089 to the mix like

docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.panel'

Test Results

Click the images to dive into the test and its results

Drag Slider with Plot Updating Performance Test

Only works for Panel

Slider with Plot Test.

Page Load and Refresh

Dash clear winner. Panel server shows blocking behaviour. Streamlit is very fast in theory but probably slowest in practice.

Page Load and Refresh Test.

You might also like...
Django-google-optimize is a Django application designed to make running server side Google Optimize A/B tests easy.

Django-google-optimize Django-google-optimize is a Django application designed to make running Google Optimize A/B tests easy. Here is a tutorial on t

Run ISP speed tests and save results
Run ISP speed tests and save results

SpeedMon Automatically run periodic internet speed tests and save results to a variety of storage backends. Supported Backends InfluxDB v1 InfluxDB v2

LuluTest is a Python framework for creating automated browser tests.
LuluTest is a Python framework for creating automated browser tests.

LuluTest LuluTest is an open source browser automation framework using Python and Selenium. It is relatively lightweight in that it mostly provides wr

Given some test cases, this program automatically queries the oracle and tests your Cshanty compiler!

The Diviner A complement to The Oracle for compilers class. Given some test cases, this program automatically queries the oracle and tests your compil

Statistical tests for the sequential locality of graphs

Statistical tests for the sequential locality of graphs You can assess the statistical significance of the sequential locality of an adjacency matrix

Fail tests that take too long to run

GitHub | PyPI | Issues pytest-fail-slow is a pytest plugin for making tests fail that take too long to run. It adds a --fail-slow DURATION command-lin

Automated tests for OKAY websites in Python (Selenium) - user friendly version

Okay Selenium Testy Aplikace určená k testování produkčních webů společnosti OKAY s.r.o. Závislosti K běhu aplikace je potřeba mít v počítači nainstal

a wrapper around pytest for executing tests to look for test flakiness and runtime regression

bubblewrap a wrapper around pytest for assessing flakiness and runtime regressions a cs implementations practice project How to Run: First, install de

Travel through time in your tests.

time-machine Travel through time in your tests. A quick example: import datetime as dt

Comments
  • Page Load Performance Test

    Page Load Performance Test

    Page Load Test

    In this test I would like to test the initial page load. One way to test this is the time between request is sent and response is received. For Dash this will be an ok measure. But for Streamlit and Panel this is only part of the answer. Because they don't transfer everything in the request-response cycle. Streamlit only transfers a minimum amount of data to open a web socket connection. The rest is transfered through that. Panel is somewhere in between Streamlit and Dash. It transfers parts of the response initially. But opens a web socket to transfer the rest. So I figured I would do a first test of how long time it takes before the user actually sees a rendered page.

    Request - Response Test

    Dash - Close up

    docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.dash'
    

    image

    Panel - Fails big time

    docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.dash'
    

    image

    As you can see in the screenshot Panels performance is orders of magnitude lower. One minor cause is probably that it sends the most data across. But as you can also see the max response time is +1min. As you can see in the video below the server is not responding at all for more than a minute. As Streamlit and Panel are both running on the Tornado server I would expect them both to be pretty quick. I've filed a bug report with Panel here https://github.com/holoviz/panel/issues/3105. I believe something in the implementation of the Bokeh/ Panel server must be wrong.

    Streamlit - Most Performant

    docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.streamlit'
    

    image

    Discussion

    If you are making apps for 1-30 users that are not refreshing often as I am the Panel issue is not that significant. But it will be expensive to scale to a lot of concurrent users currently.

    Videos

    The videos have been speeded up 4 times to be able to upload them.

    https://user-images.githubusercontent.com/42288570/149674485-0ce3b302-81f0-4df5-89eb-6386d3154688.mp4

    https://user-images.githubusercontent.com/42288570/149674490-7653e4c4-4ede-45f2-b322-a4a66c464726.mp4

    https://user-images.githubusercontent.com/42288570/149674489-bb82bab4-95e4-4865-b7e3-58193c8bf251.mp4

    Page Refresh Test

    Here we just click the refresh button as fast as we "feel" its possible to do.

    • Dash: CLEAR WINNER
    • Panel: Seems relative fast and faster than Streamlit. But just as in previous test suddenly the server blocks making this the WORST PERFORMER.
    • Streamlit seems to load consistently slow compared to the others. But its better to be consistent than to block for longer periods of time.

    https://user-images.githubusercontent.com/42288570/149675481-cd0122a9-2d12-4e2a-b066-235063cca18c.mp4

    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.dash'
    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.panel'
    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.streamlit'
    

    Promo Gif

    page-refresh-compressed

    opened by MarcSkovMadsen 1
  • Drag slider with plot updating performance test

    Drag slider with plot updating performance test

    Drag Slider with Plot Updating Performance Test

    I've created a basic application with a slider and a Plotly plot. I want to explore my data/ model interactively by dragging the slider fast back and forth.

    As you can see below only Panel can deliver the performance needed.

    You can find the source code here.

    Test Results

    | Framework | Supports drag events | Snappy experience | Comment | |-|-|-|-| | Dash | yes | no | Dash supports dragging the slider but quickly gives up | Panel | yes | yes | No problem | | Streamlit | no | no | Seems not supported.

    https://user-images.githubusercontent.com/42288570/149648993-5a850b36-cbcc-4b1f-9908-3778557759e0.mp4

    Discussion

    • Dash: As you can see Dash quickly gives up. I believe that the combination of Flask and http will never be able to provide really snappy updates. I've been told that an additional issue is that dash sends a lot of data back and forth on each update.
    • Panel: I've always been impressed by the performance of Bokeh. Panel builds on Bokeh and is capable of achieving similar performance. My hypothesis is that Panel would be even more snappy with Bokeh/ Holoviews plots.
    • Streamlit: I could not find a way to react to slider dragging. I believe the run script top to bottom architecture of Streamlit will never be able to provide really snappy updates. See this discussion Make st.slider and st.color_picker only rerun script on mouse up and Improve performance of Streamlit to update plots.

    For example Dask Dashboard is developed in Bokeh and is very, very performant. I don't believe its possible to build a performant application like that in Streamlit or Dash. It would be in Panel though.

    I would expect interactive applications with streaming to perform much better in Panel than Dash. Streamlit does not really support interactive applications with streaming.

    Another noticeable difference is just how much code it takes to make a basic Dash application.

    Source

    https://user-images.githubusercontent.com/42288570/149649005-24f9f4d2-b84f-45d2-888a-c2b4b618c66b.mp4 https://user-images.githubusercontent.com/42288570/149649006-44307849-273b-4b35-b3f1-bf657f8b5a36.mp4 https://user-images.githubusercontent.com/42288570/149649007-eca75cf9-fdf6-4304-85ef-6eea26ccd3c0.mp4

    Promo Gif

    slider-plot-comparision-promo

    Promo Video

    https://user-images.githubusercontent.com/42288570/149649087-f1f94166-f776-4b21-bbe9-fa9425d7dae0.mp4

    Commands

    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.panel'
    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.streamlit'
    docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.dash'
    
    opened by MarcSkovMadsen 0
Owner
Marc Skov Madsen
Data, Models and Analytics Ninja. PhD, CFA® and Lead Data Scientist Developer at Ørsted. Developer of awesome-panel.org and awesome-streamlit.org
Marc Skov Madsen
Let your Python tests travel through time

FreezeGun: Let your Python tests travel through time FreezeGun is a library that allows your Python tests to travel through time by mocking the dateti

Steve Pulec 3.5k Dec 29, 2022
pytest splinter and selenium integration for anyone interested in browser interaction in tests

Splinter plugin for the pytest runner Install pytest-splinter pip install pytest-splinter Features The plugin provides a set of fixtures to use splin

pytest-dev 238 Nov 14, 2022
User-oriented Web UI browser tests in Python

Selene - User-oriented Web UI browser tests in Python (Selenide port) Main features: User-oriented API for Selenium Webdriver (code like speak common

Iakiv Kramarenko 575 Jan 2, 2023
The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries. An example o

pytest-dev 9.6k Jan 2, 2023
a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)

pytest-sugar pytest-sugar is a plugin for pytest that shows failures and errors instantly and shows a progress bar. Requirements You will need the fol

Teemu 963 Dec 28, 2022
:game_die: Pytest plugin to randomly order tests and control random.seed

pytest-randomly Pytest plugin to randomly order tests and control random.seed. Features All of these features are on by default but can be disabled wi

pytest-dev 471 Dec 30, 2022
Selects tests affected by changed files. Continous test runner when used with pytest-watch.

This is a pytest plug-in which automatically selects and re-executes only tests affected by recent changes. How is this possible in dynamic language l

Tibor Arpas 614 Dec 30, 2022
Docker-based integration tests

Docker-based integration tests Description Simple pytest fixtures that help you write integration tests with Docker and docker-compose. Specify all ne

Avast 326 Dec 27, 2022
To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations

To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations, a lot of data has to be collected to ensure the variance of the tests. This respository was established to collect a lot of different test data and related test cases of different member states in a standardized manner. Each member state can generate a folder in this section.

null 160 Jul 25, 2022
Show surprise when tests are passing

pytest-pikachu pytest-pikachu prints ascii art of Surprised Pikachu when all tests pass. Installation $ pip install pytest-pikachu Usage Pass the --p

Charlie Hornsby 13 Apr 15, 2022