Live training loss plot in Jupyter Notebook for Keras, PyTorch and others

Piotr Migdał

Last update: Jan 8, 2023

Related tags

Overview

livelossplot

Don't train deep learning models blindfolded! Be impatient and look at each epoch of your training!

(RECENT CHANGES, EXAMPLES IN COLAB, API LOOKUP, CODE)

A live training loss plot in Jupyter Notebook for Keras, PyTorch and other frameworks. An open-source Python package by Piotr Migdał, Bartłomiej Olechno and others. Open for collaboration! (Some tasks are as simple as writing code docstrings, so - no excuses! :))

from livelossplot import PlotLossesKeras

model.fit(X_train, Y_train,
          epochs=10,
          validation_data=(X_test, Y_test),
          callbacks=[PlotLossesKeras()],
          verbose=0)

(The most FA)Q: Why not TensorBoard?
A: Jupyter Notebook compatibility (for exploration and teaching). The simplicity of use.

Installation

To install this version from PyPI, type:

pip install livelossplot

To get the newest one from this repo (note that we are in the alpha stage, so there may be frequent updates), type:

pip install git+git://github.com/stared/livelossplot.git

Examples

Look at notebook files with full working examples:

keras.ipynb - a Keras callback
minimal.ipynb - a bare API, to use anywhere
bokeh.ipynb - a bare API, plots with Bokeh (open it in Colab to see the plots)
pytorch.ipynb - a bare API, as applied to PyTorch
2d_prediction_maps.ipynb - example of custom plots - 2d prediction maps (0.4.1+)
poutyne.ipynb - a Poutyne callback (Poutyne is a Keras-like framework for PyTorch)
torchbearer.ipynb - an example using the built in functionality from torchbearer (torchbearer is a model fitting library for PyTorch)
neptune.py and neptune.ipynb - a Neptune.AI
matplotlib.ipynb - a Matplotlib output example
various_options.ipynb - an extended API for metrics grouping and custom outputs

You run examples in Colab.

Overview

Text logs are easy, but it's easy to miss the most crucial information: is it learning, doing nothing or overfitting? Visual feedback allows us to keep track of the training process. Now there is one for Jupyter.

If you want to get serious - use TensorBoard, . But what if you just want to train a small model in Jupyter Notebook? Here is a way to do so, using livelossplot as a plug&play component

from livelossplot import ...

PlotLosses for a generic API.

plotlosses = PlotLosses()
plotlosses.update({'acc': 0.7, 'val_acc': 0.4, 'loss': 0.9, 'val_loss': 1.1})
plot.send()  # draw, update logs, etc

There are callbacks for common libraries and frameworks: PlotLossesKeras, PlotLossesKerasTF, PlotLossesPoutyne, PlotLossesIgnite.

Feel invited to write, and contribute, your adapter. If you want to use a bare logger, there is MainLogger.

from livelossplot.outputs import ...

Plots: MatplotlibPlot, BokehPlot.

Loggers: ExtremaPrinter (to standard output), TensorboardLogger, TensorboardTFLogger, NeptuneLogger.

To use them, initialize PlotLosses with some outputs:

plotlosses = PlotLosses(outputs=[MatplotlibPlot(), TensorboardLogger()])

There are custom matplotlib plots in livelossplot.outputs.matplotlib_subplots you can pass in MatplotlibPlot arguments.

If you like to plot with Bokeh instead of matplotlib, use

plotlosses = PlotLosses(outputs=[BokehPlot()])

Trivia

It started as this gist. Since it went popular, I decided to rewrite it as a package.

Oh, and I am in general interested in data vis, see Simple diagrams of convoluted neural networks (and overview of deep learning architecture diagrams):

A good diagram is worth a thousand equations — let’s create more of these!

...or my other data vis projects.

Todo

If you want more functionality - open an Issue or even better - prepare a Pull Request.

Comments

Modularization: a serious rewrite
The goal is version 0.5.0:

extendable input plugins (kind of right now)

extendable output plugins

type hints (Python 3.5+)

tests

Some things won't be backward compatible.

A project with @bartolo1024. So far this branch does not work - work in progress.

To do:

[x] write MainLogger

[x] write PlotLoss

[x] adapt matplotlib plugin (a big thing...)

[x] adapt extrema printing

[x] adapt input callbacks

[x] adapt TensorBoard plugin

[x] adapt Neptune plugin

[x] write tests

[x] write GitHub actions

[x] update Jupyter examples (and add Colab links)
opened by stared 14
LiveLossPlot: ModuleNotFoundError: No module named 'neptune'

from livelossplot.keras import PlotLossesCallback callbacks_list = [PlotLossesCallback()]

ModuleNotFoundError Traceback (most recent call last) in 4 checkpoint = ModelCheckpoint("C:\home\jupyter\experiments\models", 5 monitor='acc', verbose=1, save_best_only=True, mode='max') ----> 6 callbacks_list = [checkpoint, PlotLossesCallback()]

~\AppData\Local\conda\conda\envs\tesseract\lib\site-packages\livelossplot\keras.py in init(self, **kwargs) 7 def init(self, **kwargs): 8 keras.callbacks.Callback.init(self) ----> 9 _PlotLossesCallback.init(self, **kwargs)

~\AppData\Local\conda\conda\envs\tesseract\lib\site-packages\livelossplot\generic_keras.py in init(self, **kwargs) 26 class _PlotLossesCallback(): 27 def init(self, **kwargs): ---> 28 self.liveplot = PlotLosses(**kwargs) 29 30 def on_train_begin(self, logs={}):

~\AppData\Local\conda\conda\envs\tesseract\lib\site-packages\livelossplot\generic_plot.py in init(self, figsize, cell_size, dynamic_x_axis, max_cols, max_epoch, metric2title, series_fmt, validation_fmt, plot_extrema, fig_path, target) 37 self.plot_extrema = plot_extrema 38 self.target = target ---> 39 from .neptune_integration import neptune_send_plot 40 self.fig_path = fig_path 41

~\AppData\Local\conda\conda\envs\tesseract\lib\site-packages\livelossplot\neptune_integration.py in ----> 1 import neptune 2 3 ctx = neptune.Context() 4 5

ModuleNotFoundError: No module named 'neptune'

opened by bobloki 13
PlotLossesKeras not working due to TypeError

For Keras version 2.1.3 the PlotLossesKeras crashes due to error in line 33 of keras_plot.py

Reason for this is: self.metric2printable['loss'] = self.metric2printable.get(self.model.loss, self.model.loss) + " (cost function)" where this breaks due to self.model.loss being a keras function. This results in

TypeError: unsupported operand type(s) for +: 'function' and 'str'

opened by fculinovic 13

PlotLossesKeras has an issue with 'lr'

I'm training a CNN model and I wanted to both a) be able to reduce the optimizer learning rate when it hits a plateau with ReduceLROnPlateau() and b) visually monitor the losses with livelossplot's PlotKerasLosses(). This combination in callbacks gives me a KeyError: 'lr'. When dropping either one of the functions the training works as it should. Hence my guess is that there must be a compatibility issue with the two.

Keras = 2.2.4 livelossplot = 0.3.4

Here's some code:

from keras.callbacks import ReduceLROnPlateau
from livelossplot import PlotLossesKeras

...

learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', 
                                            patience=3, 
                                            verbose=1, 
                                            factor=0.5, 
                                            min_lr=0.00001)

history = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),
                              epochs = epochs, 
                              validation_data = (X_val,Y_val),
                              verbose = 1, 
                              steps_per_epoch=X_train.shape[0] // batch_size,
                              callbacks=[learning_rate_reduction, PlotLossesKeras()])

And here's the error I get:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-26-bd8223167e59> in <module>
      5                               verbose = 1,
      6                               steps_per_epoch=X_train.shape[0] // batch_size,
----> 7                               callbacks=[learning_rate_reduction, PlotLossesKeras()])

~/anaconda3/envs/upwork/lib/python3.7/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name + '` call to the ' +
     90                               'Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

~/anaconda3/envs/upwork/lib/python3.7/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1416             use_multiprocessing=use_multiprocessing,
   1417             shuffle=shuffle,
-> 1418             initial_epoch=initial_epoch)
   1419 
   1420     @interfaces.legacy_generator_methods_support

~/anaconda3/envs/upwork/lib/python3.7/site-packages/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
    249                     break
    250 
--> 251             callbacks.on_epoch_end(epoch, epoch_logs)
    252             epoch += 1
    253             if callback_model.stop_training:

~/anaconda3/envs/upwork/lib/python3.7/site-packages/keras/callbacks.py in on_epoch_end(self, epoch, logs)
     77         logs = logs or {}
     78         for callback in self.callbacks:
---> 79             callback.on_epoch_end(epoch, logs)
     80 
     81     def on_batch_begin(self, batch, logs=None):

~/anaconda3/envs/upwork/lib/python3.7/site-packages/livelossplot/generic_keras.py in on_epoch_end(self, epoch, logs)
     63 
     64     def on_epoch_end(self, epoch, logs={}):
---> 65         self.liveplot.update(logs.copy())
     66         self.liveplot.draw()

~/anaconda3/envs/upwork/lib/python3.7/site-packages/livelossplot/generic_plot.py in update(self, log)
     80         self.logs.append(log)
     81         if self.plot_extrema:
---> 82             self._update_extrema(log)
     83 
     84     def draw(self):

~/anaconda3/envs/upwork/lib/python3.7/site-packages/livelossplot/generic_plot.py in _update_extrema(self, log)
     69     def _update_extrema(self, log):
     70         for metric, value in log.items():
---> 71             extrema = self.metrics_extrema[metric]
     72             if _is_unset(extrema['min']) or value < extrema['min']:
     73                 extrema['min'] = float(value)

KeyError: 'lr'

opened by mjkvaak 9

Which other data do you need in the plots?

I am thinking what is the best amount of data displayed? Maybe it is good to add other features - like the numerical value of current metrics, or maximal values.

Though, I don't want to make it too cluttered. So, I am interested in your feedback what is missing (or what you would consider more a distraction).
enhancement question

opened by stared 8
KeyError : 'metrics' Please Help

This error has started appearing after I upgraded my tensorflow version to 2.2

~/.local/lib/python3.5/site-packages/livelossplot/generic_keras.py in on_train_begin(self, logs) 29 30 def on_train_begin(self, logs={}): ---> 31 self.liveplot.set_metrics([metric for metric in self.params['metrics'] if not metric.startswith('val_')]) 32 33 # slightly convolved due to model.complie(loss=...) stuff

KeyError: 'metrics'

opened by hemangjoshi37a 7
Add support for unicode in README.md

In setup.py, the contents of README.md are read as a string, but this fails because it contains a unicode character. By using the codecs library, the unicode character can be included in the string.

I wasn't sure if I needed to do anything special to import other packages in setup.py. Let me know if there's something I need to change.

opened by jonahweissman 7

Setup outputs as str

🚀 Feature

Just to improve user's experience with this part of code:

Currently

from livelossplot import PlotLosses
from livelossplot.outputs import MatplotlibPlot, TensorboardLogger

outputs=[MatplotlibPlot(), TensorboardLogger()]
plotlosses = PlotLosses(outputs=outputs)

from livelossplot import PlotLosses

outputs=["matplotlib", "tensorboard", ...]
plotlosses = PlotLosses(outputs=outputs)

@stared what do you think ?

opened by vfdev-5 6

series_fmt in custom_series notebook

The example from custom_series notebook doesn't work:

TypeError: init() got an unexpected keyword argument 'series_fmt'

Was series_fmt parameter in an older version?

opened by lambdaofgod 6
Adding Bokeh backend
Bokeh, as an alternative backend to matplotlib, would make it simple to:

toggle between in-notebook plots and external plots (so that it would work seamlessly with both Jupyter Notebook Python scripts)

give mouseover view of exact numerical values (no more eye squeezing!)

zoom if needed

I don't want to have it instead of matplotlib (as this one makes it easier to save plots, so that they are visible in Jupyter Notebooks on GitHub), but as an additional backend.

My experience with Bokeh is limited, so this issue is open for YOUR contribution! :)
enhancement help wanted
opened by stared 5
Fix build for Python 3.7 and Python 3.10

This PR fixes the current build. There are several modifications. First, it seems that the pdoc3 package or one of its dependencies does not support Python 3.7. I thus removed it from CI. Second, I fixed formatting for Yapf. Third, I changed some arguments used in the bokeh logger because they are/will be removed. And fourth, it seems that the Neptune logger is a bit out of date and that there was some modifications to the library. I just commented the faulty line in the test.

opened by freud14 4
Prediction plot fix

A fix to #137.

While 2d_prediction_map.ipynb should work with this version, I saw there is a lot of inconsistency with the custom chart display. In general, custom plots fell victim to refactoring.

opened by stared 0

'Plot2d' object has no attribute 'set_output_mode'

🐛 Bug description

The example notebook on 2d prediction maps throws the following error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_22241/3692360535.py in <module>
      5                                     margin=0.2, h=0.02, device=device)
      6 plot2d.predict = plot2d._predict_pytorch
----> 7 liveloss = PlotLosses(outputs=[plot2d])
      8 
      9 criterion = nn.CrossEntropyLoss()

/usr/local/lib/python3.8/dist-packages/livelossplot/plot_losses.py in __init__(self, outputs, mode, **kwargs)
     30         self.outputs = [getattr(livelossplot.outputs, out)() if isinstance(out, str) else out for out in outputs]
     31         for out in self.outputs:
---> 32             out.set_output_mode(mode)
     33 
     34     def update(self, *args, **kwargs):

AttributeError: 'Plot2d' object has no attribute 'set_output_mode'

Environment

livelossplot version: 0.5.5
torch: 1.11.0+cpu
OS: Linux
Environment: Both Jupyter Lab and Colab
Python version: 3.8.0

opened by deepak-s-h 1

WARNING : tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time
tensorflow:Callback

Hi, Thank you very much for this amazing tool it helps a lot. Just a have a simple warning that will make this tool great if it is fixed. This warning occurs when I use PlotLossesKeras callback.

WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.1298s vs `on_train_batch_end` time: 0.2397s). Check your callbacks.

Environment

livelossplot==0.5.3

tensorflow==2.4.0

Keras==2.4.3

Windows 10

Jupyter Lab

Python version 3.6.8

Thank you
opened by essanhaji 2
Bokeh vis not working in Colab
🐛 Bug description

Environment

livelossplot version (e.g., 0.5.0): 0.5.1

Version of relevant libraries (e.g. TensorFlow if you use it) -

OS (e.g., Linux): Collaboratory

Environment in which the error occurred (e.g. Jupyter Notebook, Jupyter Lab): Collaboratory

Python version (e.g. 3.7.1): 3.6.9

Any other relevant information: I was trying to run the minimal Bokeh example and it shows this for each iteration: and nothing more.
opened by foxale 2
Prevent clearing of output other than the livelossplot graph
I noticed that any output besides the loss graph of livelossplot gets cleared at each training iteration. (anytime I call liveloss.draw())

Do you have a way of keeping that output ? It seems that this mechanism is at the core of your package:

# livelossplot/core.py:15 def draw_plot(logs, metrics, figsize=None, max_epoch=None, max_cols=2, validation_fmt="val_{}", metric2title={}): clear_output(wait=True) # I see what you did here plt.figure(figsize=figsize)

So I suspect that finding a workaround will be pretty difficult. With that said, I still would like to know if anyone has suggestions.
opened by pinouchon 4