I had this error running pytest in python 3.9.5

  File "/Users/clemat/.pyenv/versions/3.9.5/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/Users/clemat/.pyenv/versions/3.9.5/envs/modelkit/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/Users/clemat/dev/modelkit/tests/assets/conftest.py", line 18, in <module>
    from modelkit.assets.manager import AssetsManager
  File "/Users/clemat/dev/modelkit/modelkit/__init__.py", line 3, in <module>
    from modelkit.core.library import ModelLibrary, load_model  # NOQA
  File "/Users/clemat/dev/modelkit/modelkit/core/__init__.py", line 1, in <module>
    from modelkit.core.library import ModelLibrary, load_model
  File "/Users/clemat/dev/modelkit/modelkit/core/library.py", line 35, in <module>
    from modelkit.core.model import Asset, AsyncModel, Model
  File "/Users/clemat/dev/modelkit/modelkit/core/model.py", line 31, in <module>
    from modelkit.utils.cache import Cache, CacheItem
  File "/Users/clemat/dev/modelkit/modelkit/utils/cache.py", line 15, in <module>
    class CacheItem(NamedTuple, Generic[ItemType]):
  File "/Users/clemat/.pyenv/versions/3.9.5/lib/python3.9/typing.py", line 1881, in _namedtuple_mro_entries
    raise TypeError("Multiple inheritance with NamedTuple is not supported")
TypeError: Multiple inheritance with NamedTuple is not supported
======================================================================================= short test summary info ========================================================================================
ERROR  - TypeError: Multiple inheritance with NamedTuple is not supported

Hi !

This PR aims at adding a way to pickle the ModelLibrary which, for the moment, is impossible due to storage drivers (especially boto3 and gcs).

It introduces a MODELKIT_LAZY_DRIVER environment variable which, if set, will prevent the Storage Providers from storing the drivers (boto3, gcs, azure), instead, the configuration settings will be stored allowing the corresponding Storage Provider to build it on the fly.

This will make way easier the use of python libraries which use pickle such as Apache Spark and multi-processing, leveraging modelkit.

Thanks for reviewing, as always!

Tired of having to add empty CONFIGURATIONS attributes to your models?

Look no further, in this PR, I introduce what I think is a neat little feature for Models. Whenever there is no CONFIGURATIONS dict defined, the model will still be available with a name corresponding to its class name, but snaked case.

So,

from modelkit import Model, ModelLibrary

class ModelWithoutConfig(Model):
    def _predict(self, item):
        return item

library = ModelLibrary(models=ModelWithoutConfig)
model = library.get("model_without_config")

This is much better, in particular because unless the type is specified fully, mypy will complain, so the current alternative is much more verbose:

class ModelWithoutConfig(Model):
    CONFIGURATIONS : Dict[str, Any] = {
        "model_without_config": {}
    }

    def _predict(self, item):
        return item

What do you think?

In this draft, I propose an overview of most Modelkit's features, by solving a Sentiment Analysis problem.

I am pretty sure they are still typos, but they will be fixed anytime soon!

Looking forward to having your opinion on this.

In this PR I add support for assets stored in Azure blob storage. It's relatively simple, since I only need to correctly implement the driver in assets/drivers/azure.py.

The rest follows: I add an assets manager fixture in tests/assets/conftest.py, and then duplicate a bunch of tests using the same approach as for other drivers.

After change done in #123,

Abstract base model that were previously bypassed during configure step because of missing CONFIGURATIONS dict are now considered as valid Model to load.

Hence breaking usage where those Abstract base model are part of the same module than the Concrete derived model and we decide to load full python module.

With this change modelkit can support this use-case with small adaptation of client code:

class YourABCModel(AbstractMixin, Asset):
    ...

class YourDerivedModel(ConcreteMixin, YourABCModel):
   ...

Doing so will ensure ABC model are again bypassed by configure step but not the Derived model.

I noticed a model _load_time and _load_memory_increment doesn't include its dependencies info. It makes sense but I feel that, as we push for composable models, it is more informative to cumulate the model data if all of its dependencies. @victorbenichoux what do you think? Is there a specific need to get the model load time and memory without its dependencies?

As of today, assets can only be versioned with numbers (eg: 1.4) https://cornerstone-ondemand.github.io/modelkit/assets/managing_assets/

Modelkit should support date versionning (eg: 2021-10-16) to support organisations where models are trained on a monthy/weekly/daily basis

This PR aims at introducing test_cases in the CONFIGURATIONS map defined at class level, so that to be able to write tests restricted to a given model.

Moreover, TEST_CASES is now the way to go if you want to define upper-level class tests, which will be ran for every model configuration.

E.g.:

class TestableModel(Model[ModelItemType, ModelItemType]):
    CONFIGURATIONS: Dict[str, Dict] = {
        "some_model_a": { 
            "test_cases": {
                "cases": [
                    {"item": {"x": 1}, "result": {"x": 1}},
                    {"item": {"x": 2}, "result": {"x": 2}},
                ],
            }
        },
        "some_model_b": {},
    }
    def _predict(self, item):
        return item

Thanks for reviewing

Installing with pip install modelkit spacy results in click 8.1.0 and spacy 3.2.3 being installed. But click 8.1.0 contains a breaking change and is incompatible with spaCy 3.2.3 (https://github.com/explosion/spaCy/issues/10564).

Until updated spaCy is realeased, you need to install with pyenv exec pip install modelkit spacy "click==8.0.4"

Current state

During asset creation development process, or if you want to embed assets in a container at build time, Modelkit can be configured to use StorageProvider with LocalStorageDriver and a AssetsDir pointing to the same location of the LocalStorage. This allow to only use local file and do not duplicate them on local file system.

However, modelkit still requires to have write permission on the filesystem because the management of lock files is not disabled in this condition.

Fix

Idea is to disable the lock files management in this configuration. It is fine to do that since we are guaranteed that we will not download anything in this configuration

Sometimes it makes sense to group helper utilities together in in the library search path.

However, we don't always want these files to be loaded if for example the model itself is not used.

• The problem is that modelkit traverses and loads all python modules in the search path. It would be useful to be able to skip over specific files.

This could maybe be done with a common naming convention, for example prefixing files that should be ignored with some standard prefix, eg. library._my_module would be ignored and skipped over (this follows private convention, with the underscore indicating a module that should not be exposed publicly), while library.my_module would be traversed and loaded.

If 2 models have the same configuration name, modelkit will silently use one of them.

It should at least warn the user but maybe we just want to raise an Exception ?

Currently breaking a prediction_batch example, breaks the call and raise the error

we may want another option like (for example) returning all the batches returns except the breaking ones (set to none or set to the Exception) and a mask or something like that

Currently asset new or asset update push data from local to MODELKIT_STORAGE_PROVIDER

https://cornerstone-ondemand.github.io/modelkit/assets/managing_assets/#create-a-new-asset

which mean asset must be on local storage to be pushed

It could be interesting to be able to directly push asset from remote to remote without writing on local disk

Note: the feature seems to be partially for gcs but it just download locally to repush :

https://github.com/Cornerstone-OnDemand/modelkit/blob/6e71fe78155887fd349df4907b3633ade72d565c/modelkit/assets/cli.py#L201

(idk if it still works)

So maybe at least implement an automatic redownload + push for s3

Or better find a way to do it directly (idk if it is possible at least with 2 remotes with same credentials)