This library is a location of the LegacyLogger for PyTorch Lightning.

Overview

neptune-contrib

PyPI version Build Status

Downloads Downloads Downloads

Documentation

See neptune-contrib documentation site

Installation

Get prerequisites

  • python versions 3.5.6/3.6 are supported

Install lib

pip install neptune-contrib

Install additional packages depending on which submodule you want to use. For example:

pip install neptune-contrib[monitoring]

Installation options are: [bots], [hpo], [monitoring], [versioning], [viz] and [all].

Getting help

If you get stuck, don't worry we are here to help. The best order of communication is:

Contributing

If you see something that you don't like you are more than welcome to contribute! There are many options:

  • Participate in discussions on neptune community forum or neptune community Spectrum
  • Submit a feature request or a bug here, on Github
  • Submit a pull request that deals with an open feature-request or bug
  • Spread a word about neptune-contrib in your community
Comments
  • Server error with sacred NeptuneObserver

    Server error with sacred NeptuneObserver

    Everything was working nicely for 2 hours. Then, I started getting: WARNING - neptune.internal.channels.channels_values_sender - Failed to send channel value: Server error. Please try again later. and later a number of following stacktraces:

    Exception in thread Thread-26:                                                                                            
    Traceback (most recent call last):                                                                                            
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917, in _bootstrap_inner                   
        self.run()                                                                                                                         
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 121, in run
        self._process_batch()                                                                                             
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 133, in _process_batch
        self._send_values(self._values_batch)                                 
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 158, in _send_values
        self._experiment._send_channels_values(channels_with_values)                                                                       
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1080, in _send_channels_values
        self._backend.send_channels_values(self, channels_with_values)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188, in wrapper               
        return func(*args, **kwargs)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 513, in send_channels_values
        channelsValues=input_channels_values                                                         
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 254, in __call__                       
        request_config=request_config,                                                 
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 193, in request                           
        self.authenticated_request(sanitized_params),
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 234, in authenticated_request
        return self.apply_authentication(requests.Request(**request_params))                                                                                  
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 239, in apply_authentication
        return self.authenticator.apply(request)                                                                                                                 
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 87, in apply
        self.auth.refresh_token_if_needed()                                   
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188, in wrapper
        return func(*args, **kwargs)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 51, in refresh_token_if_needed
        self._refresh_token()
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 54, in _refresh_token
        self.session.refresh_token(self.session.auto_refresh_url)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 374, in refresh_token
        self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
        self.token = parse_token_response(body, scope=scope)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
        validate_token_parameters(params)
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in validate_token_parameters
        raise MissingTokenError(description="Missing access token parameter.")
    oauthlib.oauth2.rfc6749.errors.MissingTokenError: (missing_token) Missing access token parameter.
    
    opened by wjaskowski 16
  • Dev hiplot

    Dev hiplot

    Added function that allows users to create parallel coordinates plot.

    Few use cases:

    1. Visualize interactive plot in Notebook
    2. Save visual as standalone html

    Regarding plotting:

    1. compare multiple experiments
    2. inspect experiments lineage
    opened by kamil-kaczmarek 12
  •  assert df.shape[0] != 0, 'No experiments to show. Try other filters.'

    assert df.shape[0] != 0, 'No experiments to show. Try other filters.'

    Im trying to use hiplot with neptune. I get the following exception:

    Traceback (most recent call last):
      File "c:/Users/ali/Desktop/pythonProjects/Deep-learning-algo-master/src/TestDriver.py", line 204, in <module>
        make_parallel_coordinates_plot(html_file_path='my_visual.html',
      File "C:\Users\ali\miniconda3\envs\iptoki\lib\site-packages\neptunecontrib\viz\parallel_coordinates_plot.py", line 141, in make_parallel_coordinates_plot
        assert df.shape[0] != 0, 'No experiments to show. Try other filters.'
    AssertionError: No experiments to show. Try other filters.
    

    in my code: I call the make_parallel_coordinates_plot after the optimization step. So in Neptune, I can see for example, in my workspace, the number of the experiment let say ali-22. I tried to give the experiment id explicitly by setting the experiment_id = 'ali-22' in make_parallel_coordinates_plot but the same exception is thrown. Any suggestion to solve this is so appreciated.

    opened by ali3assi 10
  • Xgboost integration

    Xgboost integration

    Integration with XGBoost.

    What it does?

    • logs metrics during training
    • at the end of training it logs model, importances, visualized trees -> you can switch off al of them if you wish
    • docstrings ready

    More info

    • it works with train, cv and sklearn API form XGBoost
    • in the case of CV we log model, importances and trees for each folds' booster -> it can be really cool to the users.

    @jakubczakon please wait until all tests are green :)

    opened by kamil-kaczmarek 7
  • plz upgrade

    plz upgrade

    i'm trying on my pycharm with conda envs, this lightGBM case on neptune.ai, it throw a series of errors, great software, plz upgrade

    F:\Anaconda\envs\kaggle\python.exe "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\pydevconsole.py" --mode=client --port=3967
    import sys; print('Python %s on %s' % (sys.version, sys.platform))
    sys.path.extend(['F:\\iPynb\\LendingClub', 'F:/iPynb/LendingClub'])
    Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.
    PyDev console: using IPython 7.18.1
    Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] on win32
    In[2]: runfile('F:/iPynb/LendingClub/main.py', wdir='F:/iPynb/LendingClub')
    Traceback (most recent call last):
      File "F:\Anaconda\envs\kaggle\lib\site-packages\IPython\core\interactiveshell.py", line 3417, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-2-512b071ef48b>", line 1, in <module>
        runfile('F:/iPynb/LendingClub/main.py', wdir='F:/iPynb/LendingClub')
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
        pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
        exec(compile(contents+"\n", file, 'exec'), glob, loc)
      File "F:/iPynb/LendingClub/main.py", line 14, in <module>
        from scikitplot.metrics import plot_roc, plot_confusion_matrix, plot_precision_recall
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scikitplot\__init__.py", line 2, in <module>
        from . import metrics, cluster, decomposition, estimators
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scikitplot\metrics.py", line 14, in <module>
        from sklearn.metrics import confusion_matrix
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\__init__.py", line 80, in <module>
        from .base import clone
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\base.py", line 21, in <module>
        from .utils import _IS_32BIT
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\__init__.py", line 23, in <module>
        from .class_weight import compute_class_weight, compute_sample_weight
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\class_weight.py", line 7, in <module>
        from .validation import _deprecate_positional_args
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\validation.py", line 25, in <module>
        from .fixes import _object_dtype_isnan, parse_version
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\fixes.py", line 18, in <module>
        import scipy.stats
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\stats\__init__.py", line 388, in <module>
        from .stats import *
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\stats\stats.py", line 174, in <module>
        from scipy.spatial.distance import cdist
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\__init__.py", line 102, in <module>
        from ._geometric_slerp import geometric_slerp
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\_geometric_slerp.py", line 8, in <module>
        from scipy.spatial.distance import euclidean
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\distance.py", line 121, in <module>
        from ..special import rel_entr
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\special\__init__.py", line 645, in <module>
        from ._ellip_harm import (
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\special\_ellip_harm.py", line 4, in <module>
        from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm
      File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
        module = self._system_import(name, *args, **kwargs)
      File "_ellip_harm_2.pyx", line 9, in init scipy.special._ellip_harm_2
    ModuleNotFoundError: No module named 'scipy._lib._ccallback'```
    opened by elsieyin 6
  • Support multiple results in NeptuneObserver completed_event()

    Support multiple results in NeptuneObserver completed_event()

    When using sacred the result variable can be either a float (which NeptuneObserver code currently supports) or multiple results (a tuple, which causes a crash do to neptune.log_metrics() not supporting tuples).

    This is a quick fix I made ... but it would be nice that if one of the elements of the tuple is object (e.g. If parts of my results is a buffer / dataset) I could also store it

    opened by david1309 6
  • different way of disambiguating binary_classification_metrics

    different way of disambiguating binary_classification_metrics

    Hello,

    I started to use log_binary_classification_metrics, which is quite handy. I've incorporated it into an evaluation pipeline, which is run every N steps while training. That is fine as for the numeric variables this method logs to neptune. However the super-handy metric_charts are all added to a single big 'bucket' in neptune, and I cannot understand which charts are from which epoch/step. see e.g. here I have logged this twice and I have two confusion matrices, etc...: image

    I could use the 'prefix' parameter to log_binary_classification_metrics, and store epoch/step numer there - but this makes my metrics messy, as every numeric parameter at every step start to have different name. what I suggest is either:

    1. one can pass additional parameter like 'title', which would be included in the title of every generated image
    2. make 'prefix' work only for charts?

    you may have other ideas on how to solve this problem easily, I hope I made it understandable

    enhancement 
    opened by kretes 5
  • IndexError for specific values on calling `log_binary_classification_metrics`

    IndexError for specific values on calling `log_binary_classification_metrics`

    Given code:

    y_true = np.array([1, 1, 0, 0])
    y_pred = np.array([[0.00519159, 0.9948084 ],
            [0.00550756, 0.99449253],
            [0.00536389, 0.9946362 ],
            [0.00568774, 0.99431235]])
    
    log_binary_classification_metrics(y_true, y_pred, threshold=0.5)
    

    I get:

    IndexError                                Traceback (most recent call last)
    <ipython-input-24-9c6f1c89bf03> in <module>
    ----> 1 log_binary_classification_metrics(y_true, y_pred, threshold=0.5)
    
    ...site-packages/neptunecontrib/monitoring/metrics.py in log_binary_classification_metrics(y_true, y_pred, threshold, experiment, prefix)
         82     log_classification_report(y_true, y_pred[:, 1] > threshold, experiment=_exp, prefix=prefix)
         83     log_class_metrics(y_true, y_pred[:, 1] > threshold, experiment=_exp, prefix=prefix)
    ---> 84     log_class_metrics_by_threshold(y_true, y_pred[:, 1], experiment=_exp, prefix=prefix)
         85     log_roc_auc(y_true, y_pred, experiment=_exp, prefix=prefix)
         86     log_precision_recall_auc(y_true, y_pred, experiment=_exp, prefix=prefix)
    
    ...site-packages/neptunecontrib/monitoring/metrics.py in log_class_metrics_by_threshold(y_true, y_pred_pos, experiment, channel_name, prefix)
        284     _exp = experiment if experiment else neptune
        285 
    --> 286     figs = _plot_class_metrics_by_threshold(y_true, y_pred_pos)
        287 
        288     for fig in figs:
    
    ...site-packages/neptunecontrib/monitoring/metrics.py in _plot_class_metrics_by_threshold(y_true, y_pred_positive)
        711             continue
        712         else:
    --> 713             best_thres, best_score = _get_best_thres(scores_by_thres, name)
        714             fig, ax = plt.subplots()
        715             ax.plot(scores_by_thres['threshold'], scores_by_thres[name])
    
    ...site-packages/neptunecontrib/monitoring/metrics.py in _get_best_thres(scores_by_thres, name)
        806         best_res = scores_by_thres[scores_by_thres[name] == scores_by_thres[name].max()][['threshold', name]]
        807     position = len(best_res) // 2
    --> 808     result = best_res.iloc[position].to_dict()
        809     return result['threshold'], result[name]
    
    ...site-packages/pandas/core/indexing.py in __getitem__(self, key)
       1766 
       1767             maybe_callable = com.apply_if_callable(key, self.obj)
    -> 1768             return self._getitem_axis(maybe_callable, axis=axis)
       1769 
       1770     def _is_scalar_access(self, key: Tuple):
    
    ...site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
       2136 
       2137             # validate the location
    -> 2138             self._validate_integer(key, axis)
       2139 
       2140             return self._get_loc(key, axis=axis)
    
    ...site-packages/pandas/core/indexing.py in _validate_integer(self, key, axis)
       2061         len_axis = len(self.obj._get_axis(axis))
       2062         if key >= len_axis or key < -len_axis:
    -> 2063             raise IndexError("single positional indexer is out-of-bounds")
       2064 
       2065     def _getitem_tuple(self, tup: Tuple):
    
    IndexError: single positional indexer is out-of-bounds
    

    I am not sure what's wrong with hits particular set of predictions (at least from metrics perspective ;-) )

    bug 
    opened by kretes 4
  • core dumped when logger level = 10

    core dumped when logger level = 10

    Context: SacredObserver but probably related to Neptune itself.

    Whenever I use logger level of 10, I get core dumped error. logger level of >=20 works fine. Stacktrace:

    (...)
    I1010 13:52:19.454688 140211946456832 connectionpool.py:393] https://ui.neptune.ml:443 "POST /api/backend/v1/channels/system?experimentId=26f075d2-2412-4971-81c9-c5322745a03c HTTP/1.1" 200 82
    I1010 13:52:19.455582 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
    I1010 13:52:19.456417 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
    Fatal Python error: Cannot recover from stack overflow.
    
    Thread 0x00007f852299f700 (most recent call first):
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 300 in wait
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/queue.py", line 179 in get
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 111 in run
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap
    
    Current thread 0x00007f85a3430700 (most recent call first):
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 42 in _get_sep
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 147 in basename
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 327 in __init__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1488 in makeRecord
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1518 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 393 in _make_request
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600 in urlopen
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 392 in result
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 290 in _get_incoming_response
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 124 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 197 in response
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
      ...
    
    Thread 0x00007f85a2c2f700 (most recent call first):
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 913 in read
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 1037 in recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 84 in _recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 102 in recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 452 in _recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 396 in recv_strict
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 309 in recv_header
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 361 in recv_frame
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 377 in recv_frame
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 343 in recv_data_frame
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 330 in recv_data
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/websocket_client_adapter.py", line 34 in recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/reconnecting_websocket.py", line 43 in recv
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/threads/aborting_thread.py", line 34 in run
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap
    
    Thread 0x00007f85a7844740 (most recent call first):
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 854 in acquire
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 903 in handle
      File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 585 in create_hardware_metric
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/hardware/metrics/service/metric_service_factory.py", line 40 in create
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 151 in _run_hardware_metrics_reporting_thread
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 87 in start
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 987 in _start
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/projects.py", line 431 in create_experiment
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/__init__.py", line 230 in create_experiment
      File "/home/wojciech/projects/magellan/sacred_utils.py", line 134 in started_event
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 326 in _emit_started
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 230 in __call__
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 212 in run
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 263 in run_commandline
      File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 140 in automain
      File "main.py", line 716 in <module>
    [1]    10623 abort (core dumped)  main.py
    
    opened by wjaskowski 4
  • git commit hash and NeptuneObserver

    git commit hash and NeptuneObserver

    I have noticed that NeptuneObserver doesn't respect the git hash and the source file names that sacred retrieved but has its own methods to retrieve them. In general, this is wrong because it breaks the RunObserver contract from the user point of view. Thus surprised me a lot and I spent o lot of time to understand that.

    Although, I could live with that but in certain situations, NeptuneObserver fails to get the proper git commit hash while sacred doesn't. I couldn't track down the exact reason for that but it happens when

    1. I run my experiment script (git repo) from a different script
    2. my working directory (outside git repo) is different from my experiment script directory (git repo)

    A similar situation was with sources. One solution was to set manually the base_dir in NeptuneObserver (but notice: sacred has its own base_dir mechanism - why not to use it?). But when I set my base_dir to e.g. /home/name/x/y, then the whole tree is then in the sources (starting with home), which is not ideal.

    A second (not very elegant) workaround is to use os.chdir() - then the sources are gathered correctly but this doesn't affect git hash which is still missing, unfortunately.

    opened by wjaskowski 4
  • parameters: 'None', 'False', 'True'

    parameters: 'None', 'False', 'True'

    The parameter values of None, False and True get converted to strings of 'None', 'False' and 'True' when downloaded neptune.get_experiment(). Easy to work-around but irritating.

    I am not sure whether this is NeptuneObserver issue or Neptune issue since I am working solely with sacred.

    opened by wjaskowski 3
Releases(0.28.1)
Owner
neptune.ai
Metadata store for MLOps
neptune.ai
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

Yingtian Liu 6 Mar 17, 2022
Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Pytorch Lightning 1k Jan 2, 2023
Pytorch Lightning Distributed Accelerators using Ray

Distributed PyTorch Lightning Training on Ray This library adds new PyTorch Lightning accelerators for distributed training using the Ray distributed

null 166 Dec 27, 2022
Pytorch Lightning Distributed Accelerators using Ray

Distributed PyTorch Lightning Training on Ray This library adds new PyTorch Lightning plugins for distributed training using the Ray distributed compu

null 167 Jan 2, 2023
PyTorch Lightning implementation of Automatic Speech Recognition

lasr Lightening Automatic Speech Recognition An MIT License ASR research library, built on PyTorch-Lightning, for developing end-to-end ASR models. In

Soohwan Kim 40 Sep 19, 2022
A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Tom 50 Dec 16, 2022
A PyTorch Lightning solution to training OpenAI's CLIP from scratch.

train-CLIP ?? A PyTorch Lightning solution to training CLIP from scratch. Goal ⚽ Our aim is to create an easy to use Lightning implementation of OpenA

Cade Gordon 396 Dec 30, 2022
A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning This is a small repo illustrating how to use WebDataset on ImageNet. usi

null 50 Dec 16, 2022
Semantic Segmentation with Pytorch-Lightning

This is a simple demo for performing semantic segmentation on the Kitti dataset using Pytorch-Lightning and optimizing the neural network by monitoring and comparing runs with Weights & Biases.

Boris Dayma 58 Nov 18, 2022
Cookiecutter PyTorch Lightning

Cookiecutter PyTorch Lightning Instructions # install cookiecutter pip install cookiecutter

Mazen 8 Nov 6, 2022
Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Gyeongjae Choi 17 Sep 23, 2021
A PyTorch Lightning Callback for pushing models to the Hugging Face Hub 🤗⚡️

hf-hub-lightning A callback for pushing lightning models to the Hugging Face Hub. Note: I made this package for myself, mostly...if folks seem to be i

Nathan Raw 27 Dec 14, 2022
The spiritual successor to knockknock for PyTorch Lightning, get notified when your training ends

Who's there? The spiritual successor to knockknock for PyTorch Lightning, to get a notification when your training is complete or when it crashes duri

twsl 70 Oct 6, 2022
Neural Scene Flow Fields using pytorch-lightning, with potential improvements

nsff_pl Neural Scene Flow Fields using pytorch-lightning. This repo reimplements the NSFF idea, but modifies several operations based on observation o

AI葵 178 Dec 21, 2022
Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Pytorch Lightning 1k Jan 6, 2023
Lava-DL, but with PyTorch-Lightning flavour

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Sami BARCHID 4 Oct 31, 2022