Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices

Overview

MOSEC

PyPI version PyPi Downloads License Check status

Model Serving made Efficient in the Cloud.

Introduction

Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.

  • Highly performant: web layer and task coordination built with Rust πŸ¦€ , which offers blazing speed in addition to efficient CPU utilization powered by async I/O
  • Ease of use: user interface purely in Python 🐍 , by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
  • Dynamic batching: aggregate requests from different users for batched inference and distribute results back
  • Pipelined stages: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
  • Cloud friendly: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
  • Do one thing well: focus on the online serving part, users can pay attention to the model performance and business logic

Installation

Mosec requires Python 3.6 or above. Install the latest PyPI package with:

pip install -U mosec

Usage

Write the server

Import the libraries and set up a basic logger to better observe what happens.

import logging

from mosec import Server, Worker
from mosec.errors import ValidationError

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
    "%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s"
)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)

Then, we build an API to calculate the exponential with base e for a given number. To achieve that, we simply inherit the Worker class and override the forward method. Note that the input req is by default a JSON-decoded object, e.g., a dictionary here (wishfully it receives data like {"x": 1}). We also enclose the input parsing part with a try...except... block to reject invalid input (e.g., no key named "x" or field "x" cannot be converted to float).

import math


class CalculateExp(Worker):
    def forward(self, req: dict) -> dict:
        try:
            x = float(req["x"])
        except KeyError:
            raise ValidationError("cannot find key 'x'")
        except ValueError:
            raise ValidationError("cannot convert 'x' value to float")
        y = math.exp(x)  # f(x) = e ^ x
        logger.debug(f"e ^ {x} = {y}")
        return {"y": y}

Finally, we append the worker to the server to construct a single-stage workflow, and we specify the number of processes we want it to run in parallel. Then we run the server.

if __name__ == "__main__":
    server = Server()
    server.append_worker(
        CalculateExp, num=2
    )  # we spawn two processes for parallel computing
    server.run()

Run the server

After merging the snippets above into a file named server.py, we can first have a look at the command line arguments:

python server.py --help

Then let's start the server...

python server.py

and in another terminal, test it:

curl -X POST http://127.0.0.1:8000/inference -d '{"x": 2}'

or check the metrics:

curl http://127.0.0.1:8000/metrics

That's it! You have just hosted your exponential-computing model as a server! πŸ˜‰

Example

More ready-to-use examples can be found in the Example section. It includes:

  • Multi-stage workflow
  • Batch processing worker
  • PyTorch deep learning models:
    • sentiment analysis
    • image recognition

Contributing

We welcome any kind of contribution. Please give us feedback by raising issues or directly contribute your code and pull request!

Comments
  • decode binary for easier log information

    decode binary for easier log information

    • Decode ids to int to aid debugging

    Logs Before

    received 10 tasks with ids: [b'\x00\x00\x00\x00', b'\x00\x00\x00\x01', b'\x00\x00\x00\x02', b'\x00\x00\x00\x03', b'\x00\x00\x00\x04', b'\x00\x00\x00\x05', b'\x00\x00\x00\x06', b'\x00\x00\x00\x07', b'\x00\x00\x00\x08', b'\x00\x00\x00\t']
    

    Logs After

    received 10 tasks with ids: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    
    opened by ghost 6
  • [FEATURE]Benchmark for AI system

    [FEATURE]Benchmark for AI system

    Our community lacks some benchmark for server-side end to end AI system. For examples, the face recognition system, smart tracfic system with tens of models, and ocr-system. A complete comparison for the throughout and Latency across different hardware platforms and also libraries is necessary. Usability might also be considered. This is both beneficial to hardware manufactures, and also Democratisation of AI.

    enhancement 
    opened by ShiyangZhang 5
  • [BUG] `GLIBC_2.29` not found in Ubuntu 18.04

    [BUG] `GLIBC_2.29` not found in Ubuntu 18.04

    Describe the bug mosec will raise the following error in Ubuntu 18.04:

    /data/anaconda3/envs/mosec-p38/lib/python3.8/site-packages/mosec/bin/mosec: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /data/anaconda3/envs/mosec-p38/lib/python3.8/site-packages/mosec/bin/mosec)
    

    To Reproduce Just run the Sentiment Analysis example.

    Desktop (please complete the following information):

    • OS: Ubuntu 18.04
    • Library Version: 0.2.1
    • Rust Version: unknown
    • Python Version: 3.8.12

    Additional context I tried to use mosec in Ubuntu 20.04 and it worked. So if Ubuntu 20.04 or above is required, I think it's better to put it on the document.

    bug 
    opened by secsilm 5
  • feat: lint with pyright

    feat: lint with pyright

    • fix #176
    • both mypy and pyright are kept here, I wonder if we should find a way to make type-lint success when at least one of mypy and pyright works
    opened by kemingy 4
  • [BUG]NameError: name 'plasma' is not defined

    [BUG]NameError: name 'plasma' is not defined

    Describe the bug NameError: name 'plasma' is not defined

    To Reproduce Steps to reproduce the behavior:

    import base64
    import json
    import os
    
    import cv2
    import numpy as np
    import yaml
    from mosec import Worker, Server
    from mosec.errors import ValidationError
    
    from inference import Inference
    
    
    def _get_model():
        if os.path.exists('config.yaml'):
            config = open('config.yaml', mode='r', encoding='utf-8')
            config = yaml.load(config, Loader=yaml.FullLoader)
    
            model = Inference(
                det_model_path=config['det_model_path'],
                rec_model_path=config['rec_model_path'],
                device=config['device'],
                dict_path=config['dict_path'],
                rec_std=0.5, rec_mean=0.5, threshold=0.7,
                angle_classes=config['angle_classes'],
                angle_classify_model_path=config['angle_model_path'],
                object_classes=None,
                object_classify_model_path=None
            )
            return model, config
        else:
            raise FileNotFoundError('must have a config.yaml file!')
    
    
    class OCRInference(Worker):
        def __init__(self):
            super(OCRInference, self).__init__()
            self.model, self.config = _get_model()
    
        def forward(self, req: dict):
            try:
                image = req["image"]
                save_name = req['saveName']
                im = np.frombuffer(base64.b64decode(image), np.uint8)
                im = cv2.imdecode(im, 1)
                result = self.model.infer(img=im,
                                          img_save_name=save_name,
                                          cut_image_save_path=self.config['cut_image_save_path'],
                                          need_angle=self.config['need_angle'],
                                          need_object=self.config['need_object'])
                return json.dumps({'status': 1, 'result': result})
    
            except KeyError as err:
                raise ValidationError(f"cannot find key {err}")
            except Exception as err:
                raise ValidationError(f"cannot decode as image data: {err}")
    
    
    if __name__ == "__main__":
        server = Server()
    
        server.append_worker(OCRInference, num=2, max_batch_size=16)
        server.run()
    
    

    Desktop (please complete the following information):

    • OS: [e.g. Ubuntu 20.04]
    • Library Version: [e.g. 0.1.0]
    • Rust Version: [e.g. 1.55.0]
    • Python Version: [e.g. 3.8.5]

    Additional context Add any other context about the problem here.

    bug 
    opened by morestart 4
  • chore(deps-dev): bump httpx from 0.20.0 to 0.23.0 in /requirements

    chore(deps-dev): bump httpx from 0.20.0 to 0.23.0 in /requirements

    Bumps httpx from 0.20.0 to 0.23.0.

    Release notes

    Sourced from httpx's releases.

    Version 0.23.0

    0.23.0 (23rd May, 2022)

    Changed

    • Drop support for Python 3.6. (#2097)
    • Use utf-8 as the default character set, instead of falling back to charset-normalizer for auto-detection. To enable automatic character set detection, see the documentation. (#2165)

    Fixed

    • Fix URL.copy_with for some oddly formed URL cases. (#2185)
    • Digest authentication should use case-insensitive comparison for determining which algorithm is being used. (#2204)
    • Fix console markup escaping in command line client. (#1866)
    • When files are used in multipart upload, ensure we always seek to the start of the file. (#2065)
    • Ensure that iter_bytes never yields zero-length chunks. (#2068)
    • Preserve Authorization header for redirects that are to the same origin, but are an http-to-https upgrade. (#2074)
    • When responses have binary output, don't print the output to the console in the command line client. Use output like <16086 bytes of binary data> instead. (#2076)
    • Fix display of --proxies argument in the command line client help. (#2125)
    • Close responses when task cancellations occur during stream reading. (#2156)
    • Fix type error on accessing .request on HTTPError exceptions. (#2158)

    Version 0.22.0

    0.22.0 (26th January, 2022)

    Added

    Fixed

    • Don't perform unreliable close/warning on __del__ with unclosed clients. (#2026)
    • Fix Headers.update(...) to correctly handle repeated headers (#2038)

    Version 0.21.3

    0.21.3 (6th January, 2022)

    Fixed

    • Fix streaming uploads using SyncByteStream or AsyncByteStream. Regression in 0.21.2. (#2016)

    Version 0.21.2

    0.21.2 (5th January, 2022)

    Fixed

    • HTTP/2 support for tunnelled proxy cases. (#2009)
    • Improved the speed of large file uploads. (#1948)

    Version 0.21.1

    ... (truncated)

    Changelog

    Sourced from httpx's changelog.

    0.23.0 (23rd May, 2022)

    Changed

    • Drop support for Python 3.6. (#2097)
    • Use utf-8 as the default character set, instead of falling back to charset-normalizer for auto-detection. To enable automatic character set detection, see the documentation. (#2165)

    Fixed

    • Fix URL.copy_with for some oddly formed URL cases. (#2185)
    • Digest authentication should use case-insensitive comparison for determining which algorithm is being used. (#2204)
    • Fix console markup escaping in command line client. (#1866)
    • When files are used in multipart upload, ensure we always seek to the start of the file. (#2065)
    • Ensure that iter_bytes never yields zero-length chunks. (#2068)
    • Preserve Authorization header for redirects that are to the same origin, but are an http-to-https upgrade. (#2074)
    • When responses have binary output, don't print the output to the console in the command line client. Use output like <16086 bytes of binary data> instead. (#2076)
    • Fix display of --proxies argument in the command line client help. (#2125)
    • Close responses when task cancellations occur during stream reading. (#2156)
    • Fix type error on accessing .request on HTTPError exceptions. (#2158)

    0.22.0 (26th January, 2022)

    Added

    Fixed

    • Don't perform unreliable close/warning on __del__ with unclosed clients. (#2026)
    • Fix Headers.update(...) to correctly handle repeated headers (#2038)

    0.21.3 (6th January, 2022)

    Fixed

    • Fix streaming uploads using SyncByteStream or AsyncByteStream. Regression in 0.21.2. (#2016)

    0.21.2 (5th January, 2022)

    Fixed

    • HTTP/2 support for tunnelled proxy cases. (#2009)
    • Improved the speed of large file uploads. (#1948)

    0.21.1 (16th November, 2021)

    Fixed

    • The response.url property is now correctly annotated as URL, instead of Optional[URL]. (#1940)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies python chore 
    opened by dependabot[bot] 3
  • chore(cargo): bump tokio from 1.21.2 to 1.22.0

    chore(cargo): bump tokio from 1.21.2 to 1.22.0

    Bumps tokio from 1.21.2 to 1.22.0.

    Release notes

    Sourced from tokio's releases.

    Tokio v1.22.0

    Added

    • runtime: add Handle::runtime_flavor (#5138)
    • sync: add Mutex::blocking_lock_owned (#5130)
    • sync: add Semaphore::MAX_PERMITS (#5144)
    • sync: add merge() to semaphore permits (#4948)
    • sync: add mpsc::WeakUnboundedSender (#5189)

    Added (unstable)

    • process: add Command::process_group (#5114)
    • runtime: export metrics about the blocking thread pool (#5161)
    • task: add task::id() and task::try_id() (#5171)

    Fixed

    • macros: don't take ownership of futures in macros (#5087)
    • runtime: fix Stacked Borrows violation in LocalOwnedTasks (#5099)
    • runtime: mitigate ABA with 32-bit queue indices when possible (#5042)
    • task: wake local tasks to the local queue when woken by the same thread (#5095)
    • time: panic in release mode when mark_pending called illegally (#5093)
    • runtime: fix typo in expect message (#5169)
    • runtime: fix unsync_load on atomic types (#5175)
    • task: elaborate safety comments in task deallocation (#5172)
    • runtime: fix LocalSet drop in thread local (#5179)
    • net: remove libc type leakage in a public API (#5191)
    • runtime: update the alignment of CachePadded (#5106)

    Changed

    • io: make tokio::io::copy continue filling the buffer when writer stalls (#5066)
    • runtime: remove coop::budget from LocalSet::run_until (#5155)
    • sync: make Notify panic safe (#5154)

    Documented

    • io: fix doc for write_i8 to use signed integers (#5040)
    • net: fix doc typos for TCP and UDP set_tos methods (#5073)
    • net: fix function name in UdpSocket::recv documentation (#5150)
    • sync: typo in TryLockError for RwLock::try_write (#5160)
    • task: document that spawned tasks execute immediately (#5117)
    • time: document return type of timeout (#5118)
    • time: document that timeout checks only before poll (#5126)
    • sync: specify return type of oneshot::Receiver in docs (#5198)

    Internal changes

    • runtime: use const Mutex::new for globals (#5061)
    • runtime: remove Option around mio::Events in io driver (#5078)
    • runtime: remove a conditional compilation clause (#5104)
    • runtime: remove a reference to internal time handle (#5107)
    • runtime: misc time driver cleanup (#5120)
    • runtime: move signal driver to runtime module (#5121)
    • runtime: signal driver now uses I/O driver directly (#5125)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies rust chore 
    opened by dependabot[bot] 3
  • [FIX] python package: including rust source files

    [FIX] python package: including rust source files

    I just found that this doesn't really fix this issue.

    Python package is unpredictable. :(

    Originally posted by @kemingy in https://github.com/mosecorg/mosec/issues/127#issuecomment-1151181557

    opened by kemingy 3
  • fix audit: regex 1.5.4

    fix audit: regex 1.5.4

    Crate:         regex
    Version:       1.5.4
    Title:         Regexes with large repetitions on empty sub-expressions take a very long time to parse
    Date:          2022-03-08
    ID:            RUSTSEC-2022-0013
    URL:           https://rustsec.org/advisories/RUSTSEC-2022-0013
    Solution:      Upgrade to >=1.5.5
    Dependency tree:
    regex 1.5.4
    └── tracing-subscriber 0.3.8
        └── mosec 0.3.4
    
    opened by kemingy 3
  • [BUG] security vulnerabilities

    [BUG] security vulnerabilities

    ❯ cargo audit                                                                                                    
        Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
          Loaded 395 security advisories (from /home/keming/.cargo/advisory-db)
        Updating crates.io index
        Scanning Cargo.lock for vulnerabilities (100 crate dependencies)
    Crate:         chrono
    Version:       0.4.19
    Title:         Potential segfault in `localtime_r` invocations
    Date:          2020-11-10
    ID:            RUSTSEC-2020-0159
    URL:           https://rustsec.org/advisories/RUSTSEC-2020-0159
    Solution:      No safe upgrade is available!
    Dependency tree:
    chrono 0.4.19
    └── tracing-subscriber 0.2.19
        └── mosec 0.3.1
    
    Crate:         thread_local
    Version:       1.1.3
    Title:         Data race in `Iter` and `IterMut`
    Date:          2022-01-23
    ID:            RUSTSEC-2022-0006
    URL:           https://rustsec.org/advisories/RUSTSEC-2022-0006
    Solution:      Upgrade to >=1.1.4
    Dependency tree:
    thread_local 1.1.3
    └── tracing-subscriber 0.2.19
        └── mosec 0.3.1
    
    Crate:         tokio
    Version:       1.9.0
    Title:         Data race when sending and receiving after closing a `oneshot` channel
    Date:          2021-11-16
    ID:            RUSTSEC-2021-0124
    URL:           https://rustsec.org/advisories/RUSTSEC-2021-0124
    Solution:      Upgrade to >=1.8.4, <1.9.0 OR >=1.13.1
    Dependency tree:
    tokio 1.9.0
    β”œβ”€β”€ mosec 0.3.1
    └── hyper 0.14.11
        └── mosec 0.3.1
    
    error: 3 vulnerabilities found!
    

    Will create a PR to upgrade the version.

    bug 
    opened by kemingy 3
  • chore(pip): bump httpx from 0.23.0 to 0.23.1

    chore(pip): bump httpx from 0.23.0 to 0.23.1

    Bumps httpx from 0.23.0 to 0.23.1.

    Release notes

    Sourced from httpx's releases.

    Version 0.23.1

    0.23.1

    Added

    Fixed

    • Don't drop empty query parameters. (#2354)

    Removed

    • Drop .read/.aread from SyncByteStream/AsyncByteStream (#2407)
    • Drop RawURL. (#2241)
    Changelog

    Sourced from httpx's changelog.

    0.23.1

    Note: The 0.23.1 release should have used a proper version bump, rather than a minor point release. There are API surface area changes that may affect some users. See the "Removed" section of these release notes for details.

    Added

    Fixed

    • Don't drop empty query parameters. (#2354)

    Removed

    • Upload files must always be opened in binary mode. (#2400)
    • Drop .read/.aread from SyncByteStream/AsyncByteStream. (#2407)
    • Drop RawURL. (#2241)
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies python chore 
    opened by dependabot[bot] 2
  • [FEATURE] Custom error code and error message

    [FEATURE] Custom error code and error message

    Is your feature request related to a problem? Please describe. Currently, only EncodingError and DecodingError are available. It would be useful to have custom HTTP error codes and error messages for easier debugging.

    Describe the solution you'd like E.g. raise CustomError(message="GPU out of memory error", error_code=500)

    enhancement 
    opened by delonleo 4
  • [FEATURE] add numbin as a IPC serialization method

    [FEATURE] add numbin as a IPC serialization method

    According to the benchmark, numbin should be better than pickle.

    BTW, pickle is not secure to loads. (Should be okay for our use case, but not recommended)

    enhancement 
    opened by kemingy 0
  • [FEATURE] worker middleware

    [FEATURE] worker middleware

    The middleware will be placed in:

    • "read from socket" > input middleware > "forward function"
    • "return from forward function" > output middleware > "send to socket"

    by default, de/serialization methods are placed here

    Some of the use cases:

    • arrow plasma shared memory (requires init and process manager)
    • observability: analysis of the distribution of input and output
    • data validation (similar to pydantic, more like a converter)
    enhancement 
    opened by kemingy 0
  • chore: better README

    chore: better README

    The current README doesn't give users a very intuitive introduction to the feature of MOSEC.

    Let's make it more user-friendly.

    • [ ] image to demonstrate the features
    • [ ] better tutorial example
    enhancement 
    opened by kemingy 2
  • make sure all the examples can run with the latest code

    make sure all the examples can run with the latest code

    How about locally we do lint for all codes including those under examples, but we do not do that in CI? Local dev environment should have those third parties installed. Doing full lint for examples benefits code styles. It's better to have a CI to make sure all the examples can run with the latest code.

    Originally posted by @kemingy in https://github.com/mosecorg/mosec/issues/180#issuecomment-1173289612

    opened by kemingy 0
Releases(0.4.3)
  • 0.4.3(Nov 2, 2022)

    What's Changed

    • Fix discord invite link by @lkevinzc in https://github.com/mosecorg/mosec/pull/226
    • chore: fix rs version and discord link by @kemingy in https://github.com/mosecorg/mosec/pull/227
    • chore(cargo): refine required features by @kemingy in https://github.com/mosecorg/mosec/pull/230
    • Update discord link by @lkevinzc in https://github.com/mosecorg/mosec/pull/231
    • chore: rm socket after shutdown by @kemingy in https://github.com/mosecorg/mosec/pull/233
    • feat: improve the performance of IPC protocol send by @kemingy in https://github.com/mosecorg/mosec/pull/235
    • refact: use traceback for warmup error by @lkevinzc in https://github.com/mosecorg/mosec/pull/236
    • chore: add release template by @kemingy in https://github.com/mosecorg/mosec/pull/237

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.4.2...0.4.3

    Source code(tar.gz)
    Source code(zip)
  • 0.4.2(Oct 6, 2022)

    What's Changed

    • chore: add py version support badge by @kemingy in https://github.com/mosecorg/mosec/pull/214
    • chore(cargo): bump prometheus from 0.13.1 to 0.13.2 by @dependabot in https://github.com/mosecorg/mosec/pull/215
    • chore(cargo): bump clap from 3.2.20 to 3.2.22 by @dependabot in https://github.com/mosecorg/mosec/pull/216
    • chore(cargo): bump tokio from 1.21.0 to 1.21.1 by @dependabot in https://github.com/mosecorg/mosec/pull/217
    • chore(cargo): bump once_cell from 1.14.0 to 1.15.0 by @dependabot in https://github.com/mosecorg/mosec/pull/219
    • chore: use actions artifact to update docs by @kemingy in https://github.com/mosecorg/mosec/pull/220
    • fix: mypy type check for type alias by @kemingy in https://github.com/mosecorg/mosec/pull/223
    • chore(cargo): bump tokio from 1.21.1 to 1.21.2 by @dependabot in https://github.com/mosecorg/mosec/pull/222
    • refact: use argh as the CLI argument parser by @kemingy in https://github.com/mosecorg/mosec/pull/224
    • feat: generate random socket path suffix for each service by @kemingy in https://github.com/mosecorg/mosec/commit/75b72e23e1325053ab49dcd3fd4878c7163d3fb7
    • feat: check addr:port's availability by @kemingy in https://github.com/mosecorg/mosec/pull/225

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.4.1...0.4.2

    Source code(tar.gz)
    Source code(zip)
  • 0.4.1(Sep 5, 2022)

    Highlight

    • :tada: we support msgpack, check the example

    What's Changed

    • chore(cargo): bump async-channel from 1.6.1 to 1.7.1 by @dependabot in https://github.com/mosecorg/mosec/pull/201
    • chore(cargo): bump clap from 3.2.16 to 3.2.17 by @dependabot in https://github.com/mosecorg/mosec/pull/202
    • chore: update CI os version by @kemingy in https://github.com/mosecorg/mosec/pull/198
    • chore: skip existing packages for pypi upload by @kemingy in https://github.com/mosecorg/mosec/pull/199
    • feat: lint with pyright by @kemingy in https://github.com/mosecorg/mosec/pull/195
    • chore(cargo): bump once_cell from 1.13.0 to 1.13.1 by @dependabot in https://github.com/mosecorg/mosec/pull/203
    • fix: log warn abnormal tasks by @kemingy in https://github.com/mosecorg/mosec/pull/204
    • feat: msgpack mixin by @kemingy in https://github.com/mosecorg/mosec/pull/205
    • chore(cargo): bump tokio from 1.20.1 to 1.21.0 by @dependabot in https://github.com/mosecorg/mosec/pull/206
    • chore(cargo): bump clap from 3.2.17 to 3.2.20 by @dependabot in https://github.com/mosecorg/mosec/pull/207
    • chore(cargo): bump once_cell from 1.13.1 to 1.14.0 by @dependabot in https://github.com/mosecorg/mosec/pull/208
    • feat: add byte size to debug log by @kemingy in https://github.com/mosecorg/mosec/pull/211
    • chore: add msgpack example by @kemingy in https://github.com/mosecorg/mosec/pull/209
    • doc: fix resnet example by @kemingy in https://github.com/mosecorg/mosec/pull/212

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.4.0...0.4.1

    Source code(tar.gz)
    Source code(zip)
  • 0.4.0(Aug 10, 2022)

    Support more distributions

    • mosec-0.4.0-cp37-cp37m-macosx_10_9_x86_64.whl
    • mosec-0.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-cp37-cp37m-musllinux_1_1_x86_64.whl
    • mosec-0.4.0-cp38-cp38-macosx_10_9_x86_64.whl
    • mosec-0.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-cp38-cp38-musllinux_1_1_x86_64.whl
    • mosec-0.4.0-cp39-cp39-macosx_10_9_x86_64.whl
    • mosec-0.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-cp39-cp39-musllinux_1_1_x86_64.whl
    • mosec-0.4.0-cp310-cp310-macosx_10_9_x86_64.whl
    • mosec-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-cp310-cp310-musllinux_1_1_x86_64.whl
    • mosec-0.4.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl
    • mosec-0.4.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
    • mosec-0.4.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    • mosec-0.4.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
    • mosec-0.4.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

    What's Changed

    • fix: mkdocs interface dependencies by @kemingy in https://github.com/mosecorg/mosec/pull/164
    • feat: scan github actions every week by @kemingy in https://github.com/mosecorg/mosec/pull/163
    • fix: pypi source distribution by @kemingy in https://github.com/mosecorg/mosec/pull/162
    • chore: fix dependabot commit prefix by @kemingy in https://github.com/mosecorg/mosec/pull/165
    • chore(actions): bump actions/upload-artifact from 2 to 3 by @dependabot in https://github.com/mosecorg/mosec/pull/166
    • chore(actions): bump actions/setup-python from 2 to 4 by @dependabot in https://github.com/mosecorg/mosec/pull/167
    • chore(actions): bump actions/download-artifact from 2 to 3 by @dependabot in https://github.com/mosecorg/mosec/pull/168
    • chore: update cargo lock by @kemingy in https://github.com/mosecorg/mosec/pull/173
    • feat: switch to setuptools-scm for versioning by @kemingy in https://github.com/mosecorg/mosec/pull/175
    • chore(cargo): bump clap from 3.2.5 to 3.2.6 by @dependabot in https://github.com/mosecorg/mosec/pull/177
    • chore(cargo): bump tracing-subscriber from 0.3.11 to 0.3.14 by @dependabot in https://github.com/mosecorg/mosec/pull/179
    • chore(cargo): bump clap from 3.2.6 to 3.2.8 by @dependabot in https://github.com/mosecorg/mosec/pull/178
    • Fix linting in examples folder by @thinkcache in https://github.com/mosecorg/mosec/pull/180
    • chore(cargo): bump hyper from 0.14.19 to 0.14.20 by @dependabot in https://github.com/mosecorg/mosec/pull/183
    • chore(cargo): bump once_cell from 1.12.0 to 1.13.0 by @dependabot in https://github.com/mosecorg/mosec/pull/182
    • chore(cargo): bump tokio from 1.19.2 to 1.20.0 by @dependabot in https://github.com/mosecorg/mosec/pull/184
    • chore(cargo): bump clap from 3.2.8 to 3.2.12 by @dependabot in https://github.com/mosecorg/mosec/pull/185
    • chore(cargo): bump clap from 3.2.12 to 3.2.14 by @dependabot in https://github.com/mosecorg/mosec/pull/188
    • chore(cargo): bump bytes from 1.1.0 to 1.2.0 by @dependabot in https://github.com/mosecorg/mosec/pull/189
    • chore(cargo): bump tracing-subscriber from 0.3.14 to 0.3.15 by @dependabot in https://github.com/mosecorg/mosec/pull/190
    • chore: add more pypi categories by @kemingy in https://github.com/mosecorg/mosec/pull/187
    • chore(cargo): bump tokio from 1.20.0 to 1.20.1 by @dependabot in https://github.com/mosecorg/mosec/pull/193
    • chore(cargo): bump tracing from 0.1.35 to 0.1.36 by @dependabot in https://github.com/mosecorg/mosec/pull/191
    • chore(cargo): bump clap from 3.2.14 to 3.2.16 by @dependabot in https://github.com/mosecorg/mosec/pull/192
    • chore(cargo): bump bytes from 1.2.0 to 1.2.1 by @dependabot in https://github.com/mosecorg/mosec/pull/194
    • change to cibuildwheel by @kemingy in https://github.com/mosecorg/mosec/pull/174
    • release 0.4.0 by @kemingy in https://github.com/mosecorg/mosec/pull/196
    • fix CI release by @kemingy in https://github.com/mosecorg/mosec/pull/197

    New Contributors

    • @thinkcache made their first contribution in https://github.com/mosecorg/mosec/pull/180

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.6...0.4.0

    Source code(tar.gz)
    Source code(zip)
  • 0.3.6(Jun 18, 2022)

    What's Changed

    • change to lazy log by @kemingy in https://github.com/mosecorg/mosec/pull/129
    • ignore IDE config file, install rustfmt nightly in Makefile by @kemingy in https://github.com/mosecorg/mosec/pull/128
    • update int_ge_1 validation error explanation by @FerdinandZhong in https://github.com/mosecorg/mosec/pull/130
    • feat: exception traceback by @kemingy in https://github.com/mosecorg/mosec/pull/131
    • feat: use pyproject for configs by @kemingy in https://github.com/mosecorg/mosec/pull/132
    • Add license to each file by @kemingy in https://github.com/mosecorg/mosec/pull/133
    • change to pylint by @kemingy in https://github.com/mosecorg/mosec/pull/134
    • fix: convert size from str to int by @kemingy in https://github.com/mosecorg/mosec/pull/135
    • add custom env example by @lkevinzc in https://github.com/mosecorg/mosec/pull/142
    • fix env doc by @kemingy in https://github.com/mosecorg/mosec/pull/143
    • fix ci pull request path condition by @kemingy in https://github.com/mosecorg/mosec/pull/145
    • fix: only build source distribution once by @kemingy in https://github.com/mosecorg/mosec/pull/147
    • add dependabot.yml by @kemingy in https://github.com/mosecorg/mosec/pull/149
    • Update README.md by @lkevinzc in https://github.com/mosecorg/mosec/pull/150
    • fix: pylint no-self-use plugin by @kemingy in https://github.com/mosecorg/mosec/pull/151
    • fix: always run test since it's free for public repo now by @kemingy in https://github.com/mosecorg/mosec/pull/152
    • chore(pip): update requests requirement from ~=2.26 to ~=2.28 by @dependabot in https://github.com/mosecorg/mosec/pull/153
    • chore(cargo): bump once_cell from 1.10.0 to 1.12.0 by @dependabot in https://github.com/mosecorg/mosec/pull/154
    • chore(cargo): bump tracing from 0.1.32 to 0.1.34 by @dependabot in https://github.com/mosecorg/mosec/pull/155
    • chore(cargo): bump tracing-subscriber from 0.3.10 to 0.3.11 by @dependabot in https://github.com/mosecorg/mosec/pull/156
    • chore(cargo): bump clap from 3.1.8 to 3.2.5 by @dependabot in https://github.com/mosecorg/mosec/pull/159
    • chore(cargo): bump prometheus from 0.13.0 to 0.13.1 by @dependabot in https://github.com/mosecorg/mosec/pull/158
    • feat: run CI check for lock file changes by @kemingy in https://github.com/mosecorg/mosec/pull/160

    New Contributors

    • @FerdinandZhong made their first contribution in https://github.com/mosecorg/mosec/pull/130
    • @dependabot made their first contribution in https://github.com/mosecorg/mosec/pull/153

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.5...0.3.6

    Source code(tar.gz)
    Source code(zip)
  • 0.3.5(May 7, 2022)

    What's Changed

    • fix audit: regex 1.5.4 by @kemingy in https://github.com/mosecorg/mosec/pull/124
    • change to checkout@v3 by @kemingy in https://github.com/mosecorg/mosec/pull/125
    • add citation by @kemingy in https://github.com/mosecorg/mosec/pull/126
    • add rust src file to python package by @kemingy in https://github.com/mosecorg/mosec/pull/127

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.4...0.3.5

    Source code(tar.gz)
    Source code(zip)
  • 0.3.4(Mar 5, 2022)

    What's Changed

    • fix publish ci by @kemingy in https://github.com/mosecorg/mosec/pull/120
    • use ipc wrapper when the flag=ok by @kemingy in https://github.com/mosecorg/mosec/pull/122

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.3...0.3.4

    Source code(tar.gz)
    Source code(zip)
  • 0.3.3(Feb 19, 2022)

    What's Changed

    • fix clap parse by @kemingy in https://github.com/mosecorg/mosec/pull/119
    • fix CI cache by @kemingy in https://github.com/mosecorg/mosec/pull/118

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.2...0.3.3

    Source code(tar.gz)
    Source code(zip)
  • 0.3.2(Feb 5, 2022)

    What's Changed

    • panic when there is a clippy warning by @kemingy in https://github.com/mosecorg/mosec/pull/114
    • add ci cache by @kemingy in https://github.com/mosecorg/mosec/pull/115
    • fix security vulnerabilities by @kemingy in https://github.com/mosecorg/mosec/pull/117

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.1...0.3.2

    Source code(tar.gz)
    Source code(zip)
  • 0.3.1(Dec 30, 2021)

    What's Changed

    • fix plasma import bug by @kemingy in https://github.com/mosecorg/mosec/pull/104
    • fix test with 3rd opt-in features by @kemingy in https://github.com/mosecorg/mosec/pull/105
    • fix the test CI in macOS by @kemingy in https://github.com/mosecorg/mosec/pull/106
      • change the release image to Ubuntu18.04 and macOS10.15
    • add debug tutorial by @kemingy in https://github.com/mosecorg/mosec/pull/109

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.0...0.3.1

    Source code(tar.gz)
    Source code(zip)
  • 0.3.1a1(Dec 30, 2021)

    What's Changed

    • fix plasma import bug by @kemingy in https://github.com/mosecorg/mosec/pull/104
    • fix test with 3rd opt-in features by @kemingy in https://github.com/mosecorg/mosec/pull/105

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.3.0...0.3.1a1

    Source code(tar.gz)
    Source code(zip)
  • 0.3.0(Dec 28, 2021)

    What's Changed

    • add per worker env settings by @lkevinzc in https://github.com/mosecorg/mosec/pull/82
    • Fix CI by @lkevinzc in https://github.com/mosecorg/mosec/pull/83
    • upgrade to rust 2021 edition by @kemingy in https://github.com/mosecorg/mosec/pull/78
    • fix doc-gen by @lkevinzc in https://github.com/mosecorg/mosec/pull/90
    • run the lint and test when it's ready for review by @kemingy in https://github.com/mosecorg/mosec/pull/92
    • decode binary for easier log information by @delonleo in https://github.com/mosecorg/mosec/pull/96
    • fix lint by @lkevinzc in https://github.com/mosecorg/mosec/pull/99
    • Make sure input and model are on the same device by @secsilm in https://github.com/mosecorg/mosec/pull/98
    • stop support py36 by @lkevinzc in https://github.com/mosecorg/mosec/pull/100
    • support plasma shm IPC by @lkevinzc in https://github.com/mosecorg/mosec/pull/94
    • fix gen-doc dependency by @lkevinzc in https://github.com/mosecorg/mosec/pull/101
    • README highlight shell by @kemingy in https://github.com/mosecorg/mosec/pull/102

    New Contributors

    • @delonleo made their first contribution in https://github.com/mosecorg/mosec/pull/96
    • @secsilm made their first contribution in https://github.com/mosecorg/mosec/pull/98

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.2.1...0.3.0

    Source code(tar.gz)
    Source code(zip)
  • 0.2.1(Oct 23, 2021)

    What's Changed

    • Update readme by @kemingy in https://github.com/mosecorg/mosec/pull/68
    • Add multiprocess test service by @lkevinzc in https://github.com/mosecorg/mosec/pull/69
    • Fix readme by @lkevinzc in https://github.com/mosecorg/mosec/pull/70
    • Use tempfile func by @kemingy in https://github.com/mosecorg/mosec/pull/71
    • Add python metric example by @kemingy in https://github.com/mosecorg/mosec/pull/73
    • Custom err msg by @lkevinzc in https://github.com/mosecorg/mosec/pull/76
    • Draft qualitative comparison by @lkevinzc in https://github.com/mosecorg/mosec/pull/72
    • Refined worker and init logic by @lkevinzc in https://github.com/mosecorg/mosec/pull/79

    Full Changelog: https://github.com/mosecorg/mosec/compare/0.2.0...0.2.1

    Source code(tar.gz)
    Source code(zip)
  • 0.2.0(Sep 30, 2021)

    • fix typo and grammar issues (#67)
    • update readme (#66)
    • add nlp example; prune pydantic dependency (#64)
    • remove routerify (#63)
    • fix package ci (#61)
    Source code(tar.gz)
    Source code(zip)
  • 0.1.0(Sep 27, 2021)

  • 0.1.0a2(Sep 13, 2021)

  • 0.1.0a1(Sep 12, 2021)

MLflow App Using React, Hooks, RabbitMQ, FastAPI Server, Celery, Microservices

Katana ML Skipper This is a simple and flexible ML workflow engine. It helps to orchestrate events across a set of microservices and create executable

Tom Xu 8 Nov 17, 2022
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14.5k Jan 7, 2023
A high performance and generic framework for distributed DNN training

BytePS BytePS is a high performance and general distributed training framework. It supports TensorFlow, Keras, PyTorch, and MXNet, and can run on eith

Bytedance Inc. 3.3k Dec 28, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jan 8, 2023
TensorFlow Decision Forests (TF-DF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models.

TensorFlow Decision Forests (TF-DF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models. The library is a collection of Keras models and supports classification, regression and ranking. TF-DF is a TensorFlow wrapper around the Yggdrasil Decision Forests C++ libraries. Models trained with TF-DF are compatible with Yggdrasil Decision Forests' models, and vice versa.

null 538 Jan 1, 2023
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

CatBoost 6.9k Jan 5, 2023
PyTorch extensions for high performance and large scale training.

Description FairScale is a PyTorch extension library for high performance and large scale training on one or multiple machines/nodes. This library ext

Facebook Research 2k Dec 28, 2022
High performance implementation of Extreme Learning Machines (fast randomized neural networks).

High Performance toolbox for Extreme Learning Machines. Extreme learning machines (ELM) are a particular kind of Artificial Neural Networks, which sol

Anton Akusok 174 Dec 7, 2022
High performance Python GLMs with all the features!

High performance Python GLMs with all the features!

QuantCo 200 Dec 14, 2022
XGBoost-Ray is a distributed backend for XGBoost, built on top of distributed computing framework Ray.

XGBoost-Ray is a distributed backend for XGBoost, built on top of distributed computing framework Ray.

null 92 Dec 14, 2022
A framework for building (and incrementally growing) graph-based data structures used in hierarchical or DAG-structured clustering and nearest neighbor search

A framework for building (and incrementally growing) graph-based data structures used in hierarchical or DAG-structured clustering and nearest neighbor search

Nicholas Monath 31 Nov 3, 2022
An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.

Ray provides a simple, universal API for building distributed applications. Ray is packaged with the following libraries for accelerating machine lear

null 23.3k Dec 31, 2022
ο»ΏGreykite: A flexible, intuitive and fast forecasting library

The Greykite library provides flexible, intuitive and fast forecasts through its flagship algorithm, Silverkite.

LinkedIn 1.7k Jan 4, 2023
LibTraffic is a unified, flexible and comprehensive traffic prediction library based on PyTorch

LibTraffic is a unified, flexible and comprehensive traffic prediction library, which provides researchers with a credibly experimental tool and a convenient development framework. Our library is implemented based on PyTorch, and includes all the necessary steps or components related to traffic prediction into a systematic pipeline.

null 432 Jan 5, 2023
Meerkat provides fast and flexible data structures for working with complex machine learning datasets.

Meerkat makes it easier for ML practitioners to interact with high-dimensional, multi-modal data. It provides simple abstractions for data inspection, model evaluation and model training supported by efficient and robust IO under the hood.

Robustness Gym 115 Dec 12, 2022
flexible time-series processing & feature extraction

tsflex is a toolkit for flexible time-series processing & feature extraction, making few assumptions about input data. Useful links Documentation Exam

PreDiCT.IDLab 206 Dec 28, 2022
A flexible CTF contest platform for coming PKU GeekGame events

Project Guiding Star: the Backend A flexible CTF contest platform for coming PKU GeekGame events Still in early development Highlights Not configurabl

PKU GeekGame 14 Dec 15, 2022
machine learning model deployment project of Iris classification model in a minimal UI using flask web framework and deployed it in Azure cloud using Azure app service

This is a machine learning model deployment project of Iris classification model in a minimal UI using flask web framework and deployed it in Azure cloud using Azure app service. We initially made this project as a requirement for an internship at Indian Servers. We are now making it open to contribution.

Krishna Priyatham Potluri 73 Dec 1, 2022
Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale.

Model Search Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers sp

AriesTriputranto 1 Dec 13, 2021