Here's an implementation of the function discussed this morning .. Nearly finished but not quite there yet. A couple of items..

• I've done the hacky sq_to_dist voodoo so the distance matrix is kept condensed to preserve memory, and the condensed form makes for nicer output.
• In hindsight intra/inter are terrible choices of words, I'll rename these in the next commit so the code is a little easier to read :)
• It currently takes the gene labels as a 1D 'membership' vector.. is this suitable? If we're inputting a MongoDB collection item, this can be cast into the right form easily enough..

• Files written using all functions now have compression=1 enabled by default, which should save quite a bit of HDD space.
• A new montage function takes a stream of images and feeds back a (reduced) stream of montaged and channel stacked images. This should eventually supplant batch_stitch_stack.

Added a couple of doctests to the simpler functions in preprocessing.py Added DBSCAN and KMeans clustering to cluster.py

Unsure how to add a test to something non-deterministic like DBSCAN -- maybe apply clustering to the iris dataset with a fixed random-state?

Re: PR https://github.com/microscopium/microscopium/pull/87

I'm getting a javascript error related to updating the datatable with selected data from the scatterplot. It doesn't seem like a fatal error, but should still be fixed:

error handling message Message 'PATCH-DOC' (revision 1): DeserializationError("Seq(Any) expected a list or None, got {'0': -0.43731416321298383, '1': -0.4933948783766794}",)

Plan of attack - set the table.source to a CDSView of the scatterplot source selected indices.

Some other possibly related info:

• https://github.com/bokeh/bokeh/issues/7417#event-1898832123
• https://github.com/bokeh/bokeh/pull/8310

This PR allows the selection of a globally optimum threshold for object detection in input images, instead of computing a different threshold for each image. This is enabled by the global illumination correction from a previous PR.

So this was pretty easy, see the PyMongo docs. PyMongo 3.x returns a cursor from an aggregation query now. The cursor isn't accessed with the result key anymore.

So if we ever want to recursively loop through nested dictionaries created with yaml.safe_load() and turn any numeric values into floats, I recently wrote code to do this in another project.

https://github.com/DeMarcoLab/fibsem/pull/63/files

import yaml


def load_config(yaml_filename):
    """Load user input from yaml settings file.
    Parameters
    ----------
    yaml_filename : str
        Filename path of user configuration file.
    Returns
    -------
    dict
        Dictionary containing user input settings.
    """
    with open(yaml_filename, "r") as f:
        settings = yaml.safe_load(f)
    settings = _format_dictionary(settings)
    return settings


def _format_dictionary(dictionary):
    """Recursively traverse dictionary and covert all numeric values to flaot.
    Parameters
    ----------
    dictionary : dict
        Any arbitrarily structured python dictionary.
    Returns
    -------
    dictionary
        The input dictionary, with all numeric values converted to float type.
    """
    for key, item in dictionary.items():
        if isinstance(item, dict):
            _format_dictionary(item)
        elif isinstance(item, list):
            dictionary[key] = [_format_dictionary(i) for i in item]
        else:
            if item is not None:
                try:
                    dictionary[key] = float(dictionary[key])
                except ValueError:
                    pass
    return dictionary

This pull request supersedes PR https://github.com/microscopium/microscopium/pull/86. When this is merged, that can be closed.

I believe this also closes https://github.com/microscopium/microscopium/issues/80, unless you also want to include the 'nearest neighbors' view like Sarah had in the bokeh prototype here (see issue https://github.com/microscopium/microscopium/issues/73, branch https://github.com/birdsarah/microscopium/tree/bokeh_proto).

New features include:

• [x] Enforced a consistent aspect ratio when using the zoom tools on image plots.
• [x] Add instructions to README on running the web app locally (ie: not using bokeh serve ...)
• [x] Bokeh tooltips added when you hover over
• [x] Export selected data to csv with javascript callback to download_data.js
• [x] Prints web app display to paper or pdf with javascript callback
• [x] If the csv input contains a 'group' column, scatterplot data will have colors assigned to each group, plus a clickable legend to hide or display each group of data.

And a discussion of possible future problems, in order of severity:

1. Some really weird stuff is happening with the bokeh 'save' toolbar button. I've removed it for now and instead included a button that will print the whole webpage to paper or pdf. Details: It works fine to the PCA plot, but is inconsistent for the selected images plot. If you open the Chrome developer Javascript console, then you can (sometimes) magically make it work. It seems that this magical workaround doesn't work so well if you're not using bokeh serve though. I'm also pretty sure it's not entirely related to the next javascript error I'll talk about, because this odd save behaviour still happens when those lines of code are commented out.

2. I get a javascript error related to updating the datatable with the selected data. It doesn't seem like a fatal error, but should still be fixed: error handling message Message 'PATCH-DOC' (revision 1): DeserializationError("Seq(Any) expected a list or None, got {'0': -0.43731416321298383, '1': -0.4933948783766794}",)

3. Javascript doesn't support objects like python tuples, so I couldn't work out how to export dataframe columns containing these to csv (like the 'neighbors' column). So I'm excluding those for now, but I bet someone will want this!

4. It seems that bokeh's datatable doesn't support index values other than the usual 0, 1, 2, ... etc. So even though the pandas dataframe has the index values set to something fancy, we don't see those index values on the webpage or in the csv download of selected data.

5. I haven't chosen to make the table resize itself when the browser is resized, I think that would probably end up looking silly. Let me know if you think I should reconsider. I do still think it's worthwhile having a table of selected datapoints on display, since it's a good visual clue if you haven't selected anything.

After your comments on my big PR on the UI I'm like "NOW LOOK AT MORE CODE BWAHAHAH".

Summary of PR:

• The cellomics_semantic_filename has been updated to support the filename format used by the BBBC17/HT29 dataset. The HT29 dataset has files in the format:

AS_09125_050116110001_A01f00d1.DIB

ie, there is an underscore (or is that a lodash? ;)) in the screen name! Bah!! I've updated the cellomics_semantic_filename function so it can correctly parse screen names with and without underscores. It also ignores the '_stitched' tag that gets added to stitched files.

• The snail stitch function has been re-written to accept the spiral configuration as an argument. Through it'd be nice to write this function such that I could just input snail_stitch((3, 2), 'left') and it'd do the rest, so I'd like to add this as an issue (however low priority).
• Tests! A more comprehensive test has been written for the cellomics_semantic_filename function, along with a test for make_key2file and two tests for the stitching function. I adapted your method of generating test images from test_pre.py. Only thing I'm not sure about are the image_files_25 and image_files_6 functions. Is there any way to generate the number of images where I'd just need to call image_files(25) and image_files(6) ?

A small app I'm working on is using SQLite3 for the database, and I think it offers a few advantages over MongoDB, especially for a single-user app like ours that will be run locally.

Pros:

• Database lives in self contained file, no daemon is required to access the data. This is the big advantage.
• Smaller footprint than Mongo.
• Well supported in Python/Node.js

Cons:

• Key-value pairs of documents in Mongo feel more 'Pythonic'
• Data is not kept natively as JSON document (parsers exist however)
• SQL syntax
• Does not support arrays as a value type.

Seeing as I am writing code to process the Cellomics files, I thought it might be worth trialling SQLite for storing data. Any thoughts?

Calculate Haralick texture features of whole images (using skimage.feature.greycomatrix and skimage.feature.greycoprops).

Closes https://github.com/microscopium/microscopium/issues/129

TODO:

• [ ] Check/update default values for input arguments distances, angles, and levels. Is distances=[5], angles=[0], levels=256 sensible?
• [ ] Add Haralick texture features to the default_feature_map() function. Yay or nay?
• [x] Write unit tests. (Currently there are no unit tests for features.py except the two docstring examples, fyi)

References:

• Example from the scikit-image docs: GLCM Texture Features
• The GLCM Tutorial Home Page, http://www.fp.ucalgary.ca/mhallbey/tutorial.htm

Running the tests locally using the environment provided by environment.yml, I get the following test failure:

_______________ [doctest] microscopium.io.temporary_hdf5_dataset _______________
EXAMPLE LOCATION UNKNOWN, not showing all tests of that example
??? >>> with temporary_hdf5_dataset(shape, 'float32') as dset:
??? ...     dset[:] = ar
??? ...     np.testing.assert_allclose(dset, ar)
UNEXPECTED EXCEPTION: OSError('Unable to open file (file signature not found)')
Traceback (most recent call last):
  File "/Users/jni/conda/envs/mic/lib/python3.9/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest microscopium.io.temporary_hdf5_dataset[2]>", line 1, in <module>
  File "/Users/jni/conda/envs/mic/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/Users/jni/projects/microscopium/microscopium/io.py", line 132, in temporary_hdf5_dataset
    f = h5py.File(hdf5_filename)
  File "/Users/jni/conda/envs/mic/lib/python3.9/site-packages/h5py/_hl/files.py", line 424, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "/Users/jni/conda/envs/mic/lib/python3.9/site-packages/h5py/_hl/files.py", line 190, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)
/Users/jni/projects/microscopium/microscopium/io.py:None: UnexpectedException

Travis has just announced a change in their pricing structure, and no longer offer free builds for open source projects. https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

We should switch to another CI platform (maybe Github Actions).

We often use clustering for validation of the results (eg: 5-fold cross validation & k-means clustering to assess the quality of results). But k-means requires you to know how many clusters are expected which might bias the outcome, and isn't easily transferable to datasets where this is unknown.

It might be worth investigating the newly available star-clustering method: https://github.com/josephius/star-clustering

From the README:

The Star Clustering algorithm is a clustering technique that is loosely inspired and analogous to the process of star system formation. Its purpose is as an alternative clustering algorithm that does not require knowing the number of clusters in advance or any hyperparameter tuning.

Refreshing the web page causes the microscopium app to crash.

How to reproduce this

I ran the example command from our README page:

python -m microscopium.serve tests/testdata/images/data.csv -c tests/testdata/images/settings.yaml

Then opened http://localhost:5000/ which displays the app, then refreshed the web page.

Error message

Here's the relevant part of the traceback:

  File "C:\Users\genevieb\Documents\GitHub\microscopium\microscopium\serve.py", line 357, in makedoc
    prepare_xy(source, settings)  # get the default embedding columns
  File "C:\Users\genevieb\Documents\GitHub\microscopium\microscopium\serve.py", line 49, in prepare_xy
    default_embedding = settings['embeddings']['default']
KeyError: 'default'

Probable cause

It seems the switch_embeddings_button_group function is deleting this dictionary key in line 305 of serve.py. I don't know if there was a reason for us to have that line in there.

Hot on the heels of #145, Bokeh has hit 1.3 with several new features that could be useful for us:

• a file open dialog could be used for launching an empty microscopium server and then selecting the input file from there, which is a great usability improvement.
• a text input widget could be used to filter points based on search.

https://blog.bokeh.org/posts/release-1-3-0

If the color-columns in tests/testdata/images/settings.yaml refer to a csv column that can be parsed as numbers instead of strings then the microscopium server will fail to launch.

It is important to have this be allowed, because often you want to group things by say, animal number.

How to reproduce

If you edit tests/testdata/images/settings.yaml and change the line that says:

color-columns:
  categorical:
    - gene

and change it to say this instead:

color-columns:
  categorical:
    - x.umap

And then you can launch the web app with:

python -m microscopium.serve tests/testdata/images/data.csv -c tests/testdata/images/settings.yaml

Error message

You don't see an error until you try to load/refresh localhost in your browser. Then you see a: 500: Internal Server Error

and the traceback in the terminal is:

$ python -m microscopium.serve tests/testdata/images/data.csv -c tests/testdata/images/settings.yaml
ready!
WARNING:bokeh.server.util:Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
ERROR:tornado.application:Uncaught exception GET / (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/server/views/doc_handler.py", line 55, in get
    session = yield self.get_session()
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/server/views/session_handler.py", line 77, in get_session
    session = yield self.application_context.create_session_if_needed(session_id, self.request)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/server/contexts.py", line 215, in create_session_if_needed
    self._application.initialize_document(doc)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/application/application.py", line 178, in initialize_document
    h.modify_document(doc)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/application/handlers/function.py", line 133, in modify_document
    self._func(doc)
  File "/Users/genevieve/Documents/GitHub/microscopium-universe/microscopium-feature/microscopium/serve.py", line 349, in makedoc
    embed = embedding(source, settings)
  File "/Users/genevieve/Documents/GitHub/microscopium-universe/microscopium-feature/microscopium/serve.py", line 204, in embedding
    group_filter = GroupFilter(column_name=color_column, group=group)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/models/filters.py", line 111, in __init__
    super(GroupFilter, self).__init__(**kw)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/models/filters.py", line 62, in __init__
    super(Filter, self).__init__(**kw)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/model.py", line 307, in __init__
    super(Model, self).__init__(**kwargs)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/has_props.py", line 253, in __init__
    setattr(self, name, value)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/has_props.py", line 280, in __setattr__
    super(HasProps, self).__setattr__(name, value)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/property/descriptors.py", line 545, in __set__
    self._internal_set(obj, value, setter=setter)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/property/descriptors.py", line 766, in _internal_set
    value = self.property.prepare_value(obj, self.name, value)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/property/bases.py", line 327, in prepare_value
    raise e
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/property/bases.py", line 320, in prepare_value
    self.validate(value)
  File "/Users/genevieve/anaconda3/envs/microscopium/lib/python3.6/site-packages/bokeh/core/property/bases.py", line 452, in validate
    raise ValueError(msg)
ValueError: expected a value of type str, got -0.4722998308884122 of type float64
ERROR:tornado.access:500 GET / (::1) 41.45ms