Hey @sergioburdisso,

Thank you for this awesome project! Currently the evaluation class only supports single label classification, even though SS3 inherently supports multilabel classification. These are the steps (I see) needed to support multilabel classification evaluation:

• Take the output of classify_multilabel
• Convert result to binarized vector (same length as confidence vector)
• Implement multilabel classification metrics usage (e.g. Hamming Loss)
• Adopt Gridsearch

@sergioburdisso It would be a great feature to have custom preprocessing in the Live Test. This will enable us to visually understand the words, sentences, and paragraphs that helped the model to classify a particular document after custom preprocessing.

Hey @sergioburdisso,

as far as I understand the SS3 framework, there is an inconsistency between the initialization of SS3 and its documentation. The initialization describes the parameter sn_m as method used to compute the sanction (sn) function [...]

However in the actual initialization the only function that changes based on the sn_m parameter is the significance function (see here).

I would be great if you could have a look at it and tell me whether I'm wrong? 😄

Best, Florian

I'm getting an error list index out of range. I'm not sure what happened here. I'm using the latest built as of posting (just installed it prior to using it here), my python is 3.6 if I remember correctly.

EDIT: I don't know why but restarting the kernel fixed it.

openjournals/joss-reviews#3934

This package has good documentation. Going through the examples I came up with a feature request, which would greatly benefit introducing newcomers and prototyping code.

I like the first example in README to be straightforward and copy-paste ready, which is not the case here (looking at missing code ...).

How about implementing some import_dataset(url) / download(url) functionality in utils or Dataset that would, for example, download the dataset .zip file and unpack it (sample code) so that one can load the data into exemplary code:

from pyss3 import SS3

Dataset.import_dataset("https://github.com/sergioburdisso/pyss3/blob/master/examples/datasets/movie_review.zip")
x_train, y_train = Dataset.load_from_files("movie_review/train")
x_test, y_test = Dataset.load_from_files("movie_review/test")

clf = SS3()
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)

Implementation details and naming may vary, but it would be nice to easily run code from README.

Hey ,

[Note] : I have pandas dataframe contain 2 columns as ,

1. Text
2. Label

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x_data,
                                                    y_data ,
                                                    test_size = 0.2, 
                                                    shuffle=False)

train () and fit() methods are not working

here is a reference code

How to fix it?

Thanks

I am eager to use the SS3 classifier for text classification task in my master's thesis. Unfortunately when I run it I get a division by zero error message, see image. My text seems fairly clean (although not yet cleaned exactly the right way) to me, so I am not sure what is causing this.

Is there anything you suspect might be going wrong which I could try? Or anywhere where the data criteria are listed (I've looked but maybe I've overlooked)?

I included the data structure (pandas series), some of what my data looks like and the error.

Many thanks!

Hey @sergioburdisso,

for multilabel classification the file structure described in the topic categorization tutorial is not efficient since the text related to multiple label has to be stored in multiple files. My current approach is to write the text to one file linewise and the respective labels to another file, also linewise.

# Writing Data
dataset = {"Text 1": ["label1", "label2"], 
           "Text 2": ["label2", "label3"], 
           "Text 3": ["label1"]}

for text, labels in dataset.items():

  with open('text.txt', 'a+') as text_file:
    text_file.write(text + '\n')

  with open('labels.txt', 'a+') as label_file:
    label_file.write(';'.join(labels) + '\n')

The result is the following:

# cat text.txt
Text 1
Text 2
Text 3

# cat labels.txt
label1;label2
label2;label3
label1

It would be great if util.Dataset.load_from_files could be adjusted to also support this! But I'm also open for other suggestions on how to tackle that problem :)

Thanks for your hard work!

Hey @sergioburdisso,

I've noticed taht you fixed recently the Multilabel fit issue #6 But the Live_Test.run(clf, X_test, y_test) still does not accept y_test as List[List[str]] It would be really great to have it If you don't have time maybe I could submit a PR? Olivier

Hello, while I was trying to optimise for hyperparameters I have found that there is little I could do to avoid having a large printout to stdout.

I tried to mute this by doing the following:

import pyss3
pyss3.Print.set_quiet(True)

but besides, it is also tqdm that produces progress bars, hence I propose propagating Print.__quiet__to tqdm's disable parameter, for example:

tqdm(..., disable=Print.__quiet__)

Thanks :) Keep up the good work!

https://github.com/openjournals/joss-reviews/issues/3934

Hi, I enjoy working with http://tworld.io/ss3, which is highlighted in README. It is also good to see a statement of need with the two references there. Could we add the same to the welcome page of the documentation at https://pyss3.readthedocs.io/en/latest?

My comments on the software paper wrt to the JOSS submission:

• In the abstract, you declare two useful tools and then say: "For instance, one of these tools provides (...)". Since there are only two tools, I'd also suggest describing the other one as well. Additionally, instead of "for instance", I'd go with one sentence per each tool. This way, I can only guess what the other functionality is.
• The last sentence in the abstract is quite long, and I would consider breaking it into shorter pieces.
• My understanding is that your input is the implementation of the SS3 algorithm. Therefore, I'd be happy to see a bit more details about the algorithm to make the paper self-contained. Also, the title gives the impression of you introducing a new model/algorithm. I believe the input of this work is an implementation? This distinction is not very clear to me. Also, does it mean the SS3 algorithm was proposed without any implementation? That sounds a bit confusing.
• Is the explanation tool model-specific, or it works with any classification method? What is your exact input here (the explanation algorithm or GUI?). If it's a model-agnostic explanation, perhaps it could be implemented in a different package?
• github -> GitHub?
• since explanations of the models are not the primary contribution of this work (?), you could consider adding a reference to some work in this area.
• Line 40: "On the other hand" doesn't contrast with anything, and definitely no "On the one hand". Maybe you could consider rephrasing this
• Footnote 2 has an issue with spacing – no space between "ArXiv" and a bracket

I ticked all the boxes for this part anyways. An exciting paper overall. I like the examples particularly. However, it's not clear what the exact contributions are: model implementation and the explanation GUI (or algorithm)?

In reference to https://github.com/openjournals/joss-reviews/issues/3934

https://github.com/openjournals/joss-reviews/issues/3934 Hi, I hope these comments help in improving the paper.

Comments

1. The paper's title could see a change. It says "PySS3: A new interpretable and simple machine learning model for text classification", but the model is named "SS3" and seems not new. The title of the repository seems more accurate, "A Python package implementing a new simple and interpretable model for text classification", but even then one could drop "new" and use the PyPI package's title, e.g. "PySS3: A Python package implementing the SS3 interpretable text classifier [with interactive/visualization tools for explainable AI]". Just an example to be considered.
2. I would recommend the authors to highlight in the article the software's aspect of "interactive" (explanation, analysis) and (model, machine learning) "monitoring" as this seems both novel and emerging in discussions lately.
3. In the end, it would be useful to release a stable version 1.0 of the package (on GitHub, PyPI) and mark that in the paper, e.g. in the Summary section.

Summary

• L10. "implements novel machine learning model" - It might not be seen as novel when the model was already published in 2019 and extended in 2020.
• L11. mentioning "two useful tools" without describing what the second does seems off

Statement of need This part discusses mainly the need for open-source implementation of the machine learning models. However, as I see it, the significant contributions of the software/paper, distinguishing it from the previous work, are the Live_Test/Evaluation tools allowing for visual explanation and hyperparameter optimization. This could be further underlined.

State of the field The paper lacks a brief discussion on packages in the field of interpretable and explainable machine learning. In that, I suggest the authors reference/compare to the following software related to interactive explainability:

1. Wexler et al. "The What-If Tool: Interactive Probing of Machine Learning Models" (IEEE TVCG, 2019) https://doi.org/10.1109/TVCG.2019.2934619
2. Tenney et al. "The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models" (EMNLP, 2020) http://doi.org/10.18653/v1/2020.emnlp-demos.15
3. Benjamin Hoover, Hendrik Strobelt, Sebastian Gehrmann. "exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models" (ACL, 2020) https://www.doi.org/10.18653/v1/2020.acl-demos.22
4. [Ours] "dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Python" (JMLR, 2021) https://www.jmlr.org/papers/v22/20-1473.html

Other possibly missing/useful references:

1. Pedregosa et al. "Scikit-learn: Machine Learning in Python" (JMLR, 2011) https://www.jmlr.org/papers/v12/pedregosa11a.html
2. Christoph Molnar "Interpretable Machine Learning - A Guide for Making Black Box Models Explainable" (book, 2018) https://christophm.github.io/interpretable-ml-book
3. Cynthia Rudin "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead" (Nature Machine Intelligence, 2019) https://doi.org/10.1038/s42256-019-0048-x
4. [Ours] "modelStudio: Interactive Studio with Explanations for ML Predictive Models" (JOSS, 2019) https://doi.org/10.21105/joss.01798

Implementation

• L48 github -> GitHub
• L54 "such as the one introduced later by the same authors" -> "by us" would be easier to read
• L57 missing the citation of scikit-learn

Illustrative examples

1. In the beginning, it lacks a brief description of the predictive task used for the example (dataset name, positive/negative text classification, etc.).
2. Also, it could now be updated with the Dataset.load_from_url() function.

Conclusions Again, I have doubts that the machine learning model is "novel", as it has been previously published etc.. It might be misunderstood as "introducing a novel machine learning model".

Hi! A way pass a scorer function (e.g using sklearn's make_scorer) on Evaluation would make this pyss3 even greater.

Any plans on this?

This is a very interesting project. Thank you!