Python AsyncIO data API to manage billions of resources

Overview

Introduction

https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat https://travis-ci.org/plone/guillotina.svg?branch=master Test Coverage Python Versions License Chat Docker Cloud Build Status

Please read the detailed docs

This is the working project of the next generation Guillotina server based on asyncio.

Dependencies

  • Python >= 3.7
  • PostgreSQL >= 9.6

Quickstart

We use pip

pip install guillotina

Run PostgreSQL

If you don't have a PostgreSQL server to play with, you can run one with Docker.

Download and start the Docker container by running

make run-postgres

Run the server

To run the server

g

Then...

curl http://localhost:8080

Or, better yet, use Postman to start playing with API.

You can also navigate in your Guillotina server with its built-in web admin interface by visiting http://localhost:8080/+admin/.

Deploy on Heroku

Read more Guillotina-Heroku.

Getting started with development

Using pip (requires Python > 3.7)

git clone [email protected]:plone/guillotina.git
cd guillotina
python3.7 -m venv .
./bin/pip install -r requirements.txt
./bin/pip install -r contrib-requirements.txt
./bin/pip install -e '.[test]'
./bin/pre-commit install

Run tests

We're using pytest

./bin/pytest guillotina

and for test coverage

./bin/pytest --cov=guillotina guillotina/

With file watcher...

./bin/ptw guillotina --runner=./bin/py.test

To run tests with cockroach db

USE_COCKROACH=true ./bin/pytest guillotina

Default

Default root access can be done with AUTHORIZATION header : Basic root:root

Docker

You can also run Guillotina with Docker!

First, run PostgreSQL

docker run --rm \
    -e POSTGRES_DB=guillotina \
    -e POSTGRES_USER=guillotina \
    -p 127.0.0.1:5432:5432 \
    --name postgres \
    postgres:9.6

Then, run Guillotina

docker run --rm -it \
    --link=postgres -p 127.0.0.1:8080:8080 \
    plone/guillotina:latest \
    g -c '{"databases": [{"db": {"storage": "postgresql", "dsn": "postgres://guillotina:@postgres/guillotina"}}], "root_user": {"password": "root"}}'

This assumes you have a config.yaml in your current working directory

Chat

Join us to talk about Guillotina at https://gitter.im/plone/guillotina

Comments
  • Add key-value to schema.Dict fails silently

    Add key-value to schema.Dict fails silently

    Hi! Here is my weekly issue/pr :P

    I'm opening this issue because I found something weird with fields of type dict with default values (maybe list are affected as well)

    Example

    I have a custom content-type like this:

    class ICustomContentType(IItem):
        images = schema.Dict(
            key_type=schema.TextLine(),
            value_type=schema.TextLine(),
            required=False,
            default={}
        )
    

    And I implemented an endpoint that adds a key-value pair to field images.

    @configure.service(
        method="PATCH",
        name="@update-dict",
        permission="guillotina.AddContent"
    )
    async def update_default_dict(context, request):
        context.images['new-key'] = 'value'
        request._txn.register(context)
        assert len(context.images) == 0  # wat?
    

    After this endpoint retrieve the object and the field images is {}.

    BUT if I replace the previous code with context.images = {'new-key': 'value'} then everything works fine.

    Is this a bug or I'm not using default value of field properly?

    Reproduce

    I implemented a test that reproduces the problem: https://github.com/masipcat/guillotina/commit/9be9d11a073f89bb526a7ee63aedb2f8aaddbe15

    Thank you!

    opened by masipcat 36
  • Proposal: remove __txn__ from object

    Proposal: remove __txn__ from object

    thoughts:

    • __txn__ feels magical
    • adding objects to container always means using parent.__txn__
    • in order to make this work most of the time, you need to have the magic in transaction manager which would reuse txn objects. Otherwise, parents wouldn't be able to add children across transaction boundaries without reassigning __txn__ all the time. So it added weird cross dependencies
    • conceptually, it's much easier to simply to txn.add(ob) -- explicit is better than implicit..

    implications:

    • maybe all object mutate api methods should accept a txn argument to be able to customize which txn to register it against(in case you are manually working with multiple transactions).
    • I'd also like to find a nice way to auto retry conflict errors when manually working with txn objects. In practice, writing a generic api for this is difficult though.
    opened by vangheem 14
  • @items pagination is broken

    @items pagination is broken

    The @items service filters out the result based on permissions. The pagination is made prior to filtering, leading to exotic paging information in the final result.

    opened by cdevienne 13
  • Improvements in contrib.dbusers

    Improvements in contrib.dbusers

    Things I'm working on:

    1. Ensure that there are subscribers of events Added, Modified and Removed for User and Group (this is a continuation of @jordic PR)
    2. Implement the IGroups utility for dbusers. The problem I see so far is that get_principal() needs to be a coroutine so we can load the group from groups folder and fill the GuillotinaGroup with the roles and permissions. (see https://github.com/plone/guillotina/pull/852/files#r368751920)
    opened by masipcat 12
  • Proposal: behavior aliases

    Proposal: behavior aliases

    Instead of needing to reference complete behavior interfaces names, be able to register a behavior with an alias.

    For example, a PATCH like this:

    "guillotina.behaviors.dublincore.IDublinCore": {
                        "title": "foo",
                        "description": "bar",
                        "tags": ["one", "two"]
                    }
    

    Could be changed to:

    "dublincore": {
                        "title": "foo",
                        "description": "bar",
                        "tags": ["one", "two"]
                    }
    

    Registering behavior would change to add "alias" option for the configurator.

    GIP 
    opened by vangheem 12
  • Use orjson 3.x

    Use orjson 3.x

    Hi,

    As orjson 3.x has many fixes, see https://github.com/ijl/orjson/releases, I can suggest to use this version instead of 2.6.0

    Besides, it will fix https://github.com/the-benchmarker/web-frameworks/pull/3600

    Regards,

    opened by waghanza 11
  • Automatic docker builds

    Automatic docker builds

    Do we have any kind of automatic docker image builds? (On every commit to master, or on every new release)

    We need it for guillotina_react. To be able to launch a guillotina (ideally) from a docker, and do some cypress testing on it.

    opened by jordic 11
  • Add 'db_schema' to postgresql storage config

    Add 'db_schema' to postgresql storage config

    Closes https://github.com/plone/guillotina/issues/424

    This is the easiest approach I found but there are some alternative to this implementation:

    1. Instead of storing schema.table_name in attribute _objects_table_name we can change all the sql statatements to accept the 'schema' as a parameter.
    2. Execute SET SCHEMA {schema} when opening a connection
    3. Ideas?
    opened by masipcat 10
  •  get_object_by_oid load all the object's parents

    get_object_by_oid load all the object's parents

    In the current implementation, get_object_by_oid load all the object's parents. It is a little expensive.

    Although we can cache objects from redis, we can fix it better.

    when load from get_object_by_oid, we can just use a lazy parent:

    obj.__parent__ = LazyResource(obj)
    

    The LazyResource is loaded from db or cache when required.

    opened by panjunyong 9
  • Fix validation of object ids on transactions

    Fix validation of object ids on transactions

    When a create_content_in_container is done on a object there is not validation that the id has been already choosen and the conflict apears on serializing at the DB.

    opened by bloodbare 9
  • being able to call do_action with super powers using the utility

    being able to call do_action with super powers using the utility

    In some situations in my internal code, I want to be able to switch the state of the workflow on behalf of other users without checking any security policies.

    opened by nilbacardit26 8
  • adding @update-catalog endpoint

    adding @update-catalog endpoint

    This goes in conjunction with this: https://github.com/plone/guillotina_elasticsearch/pull/105

    With ES nowadays, if we want to index new fields using index_field the only way to update the mappings and settings of the index using the API of guillotina is to do a POST to @ catalog, which deletes the whole index, and creates it again. Then we need to reindex all the data calling POST @catalog-reindex. With this change, (regarding ES) when adding new index fields we can just call @update-catalog and delegate the logic (if necessary) to the catalog utility.

    opened by nilbacardit26 1
  • Bump pillow from 9.0.1 to 9.3.0

    Bump pillow from 9.0.1 to 9.3.0

    Bumps pillow from 9.0.1 to 9.3.0.

    Release notes

    Sourced from pillow's releases.

    9.3.0

    https://pillow.readthedocs.io/en/stable/releasenotes/9.3.0.html

    Changes

    ... (truncated)

    Changelog

    Sourced from pillow's changelog.

    9.3.0 (2022-10-29)

    • Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [wiredfool]

    • Initialize libtiff buffer when saving #6699 [radarhere]

    • Inline fname2char to fix memory leak #6329 [nulano]

    • Fix memory leaks related to text features #6330 [nulano]

    • Use double quotes for version check on old CPython on Windows #6695 [hugovk]

    • Remove backup implementation of Round for Windows platforms #6693 [cgohlke]

    • Fixed set_variation_by_name offset #6445 [radarhere]

    • Fix malloc in _imagingft.c:font_setvaraxes #6690 [cgohlke]

    • Release Python GIL when converting images using matrix operations #6418 [hmaarrfk]

    • Added ExifTags enums #6630 [radarhere]

    • Do not modify previous frame when calculating delta in PNG #6683 [radarhere]

    • Added support for reading BMP images with RLE4 compression #6674 [npjg, radarhere]

    • Decode JPEG compressed BLP1 data in original mode #6678 [radarhere]

    • Added GPS TIFF tag info #6661 [radarhere]

    • Added conversion between RGB/RGBA/RGBX and LAB #6647 [radarhere]

    • Do not attempt normalization if mode is already normal #6644 [radarhere]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump codecov from 2.0.15 to 2.0.16

    Bump codecov from 2.0.15 to 2.0.16

    Bumps codecov from 2.0.15 to 2.0.16.

    Changelog

    Sourced from codecov's changelog.

    2.0.16

    • fixed reported command injection vulnerability.
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 2
  • Issuing subrequests

    Issuing subrequests

    Currently, in a get (read) request, it is difficult to issue a patch (write) request. Is there a (easy) way/mechanism to make sub-requests, like plone.subrequest packages.

    opened by jianaijun 0
  • Deserialize pickles in a asyncio executor + converted db functions reader() and IWriter.serialize() to async

    Deserialize pickles in a asyncio executor + converted db functions reader() and IWriter.serialize() to async

    I measured that big objects >50kB take a lot of time to deserialize, about 50, 100, 200ms depending on the size.. This change avoids blocking all requests in the process during deserialization.

    I'm not sure if we should consider this a breaking change or if it's ok as a patch version

    opened by masipcat 3
  • Review current conflicts resolution logic

    Review current conflicts resolution logic

    I was reviewing this part and I don't understand why the strategy resolve_readcommitted (the default one) needs the tpc_vote logic from resolve strategy.

    If I'm not wrong, these are the cases we are trying to mitigate:

    1. Cache invalidation race condition: an object is modified but the next request that wants to modify the object reads the stale object. During the write to the storage, we want to detect that the old txn id is not the latest in postgres. This is already implemented in the UPDATE sql, using the WHERE tid = $7::bigint. If the WHERE clause doesn't match it would not update the object and we'll raise the ConflictError.

    2. Two concurrent requests modify the same object "at the same time". At the same time could mean two things: a. Both requests read the same object, the first starts the pg transaction and writes the changes and finished. When the second transaction starts will try to update the object and I'd would happen the error in point 1. The Old TID doesn't match and the ConflictError is raised. b. Both requets read the same object, the first request starts the transaction and writes the changes. At the same time, before committing the pg transaction the second request starts a transaction and tries to write the same object. In that case two things could happen:

      1. same thing as previous points: TID mismatch -> Conflict.
      2. deadlock and both requests fail

    My conclusion is that in all these situations postgres/SQL queries already handle the conflicts, and there is no need for the extra logic. The only case I could think this is useful is for Cockroach DB.

    ./cc @bloodbare @vangheem

    Originally posted by @masipcat in https://github.com/plone/guillotina/pull/1154#discussion_r748486685

    opened by masipcat 0
Owner
Plone Foundation
Core of the enterprise CMS Plone and related projects. For add-ons visit: https://github.com/collective
Plone Foundation
WebSocket and WAMP in Python for Twisted and asyncio

Autobahn|Python WebSocket & WAMP for Python on Twisted and asyncio. Quick Links: Source Code - Documentation - WebSocket Examples - WAMP Examples Comm

Crossbar.io 2.4k Jan 6, 2023
A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs.

cx_Oracle_async A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs. Easy to use , buy may not

null 36 Dec 21, 2022
A micro web-framework using asyncio coroutines and chained middleware.

Growler master ' dev Growler is a web framework built atop asyncio, the asynchronous library described in PEP 3156 and added to the standard library i

null 687 Nov 27, 2022
Lemon is an async and lightweight API framework for python

Lemon is an async and lightweight API framework for python . Inspired by Koa and Sanic .

Joway 29 Nov 20, 2022
Endpoints is a lightweight REST api framework written in python and used in multiple production systems that handle millions of requests daily.

Endpoints Quickest API builder in the West! Endpoints is a lightweight REST api framework written in python and used in multiple production systems th

Jay Marcyes 30 Mar 5, 2022
APIFlask is a lightweight Python web API framework based on Flask and marshmallow-code projects

APIFlask APIFlask is a lightweight Python web API framework based on Flask and marshmallow-code projects. It's easy to use, highly customizable, ORM/O

Grey Li 705 Jan 4, 2023
A minimal, extensible, fast and productive API framework for Python 3.

molten A minimal, extensible, fast and productive API framework for Python 3. Changelog: https://moltenframework.com/changelog.html Community: https:/

Bogdan Popa 980 Nov 28, 2022
A public API written in Python using the Flask web framework to determine the direction of a road sign using AI

python-public-API This repository is a public API for solving the problem of the final of the AIIJC competition. The task is to create an AI for the c

Lev 1 Nov 8, 2021
An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API.

django-rest-client An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API. Suitable for APIs made wit

Certego 4 Aug 25, 2022
Python Wrapper for interacting with the Flutterwave API

Python Flutterwave Description Python Wrapper for interacting with the Flutterwa

William Otieno 32 Dec 14, 2022
REST API framework designed for human beings

Eve Eve is an open source Python REST API framework designed for human beings. It allows to effortlessly build and deploy highly customizable, fully f

eve 6.6k Jan 7, 2023
Restful API framework wrapped around MongoEngine

Flask-MongoRest A Restful API framework wrapped around MongoEngine. Setup from flask import Flask from flask_mongoengine import MongoEngine from flask

Close 525 Jan 1, 2023
🔥 Fire up your API with this flamethrower

?? Fire up your API. Documentation: https://flama.perdy.io Flama Flama aims to bring a layer on top of Starlette to provide an easy to learn and fast

José Antonio Perdiguero 216 Dec 26, 2022
Fully featured framework for fast, easy and documented API development with Flask

Flask RestPlus IMPORTANT NOTICE: This project has been forked to Flask-RESTX and will be maintained by by the python-restx organization. Flask-RESTPlu

Axel H. 2.7k Jan 4, 2023
Flask-Potion is a RESTful API framework for Flask and SQLAlchemy, Peewee or MongoEngine

Flask-Potion Description Flask-Potion is a powerful Flask extension for building RESTful JSON APIs. Potion features include validation, model resource

DTU Biosustain 491 Dec 8, 2022
Restful API framework wrapped around MongoEngine

Flask-MongoRest A Restful API framework wrapped around MongoEngine. Setup from flask import Flask from flask_mongoengine import MongoEngine from flask

Close 505 Feb 11, 2021
REST API framework designed for human beings

Eve Eve is an open source Python REST API framework designed for human beings. It allows to effortlessly build and deploy highly customizable, fully f

eve 6.3k Feb 17, 2021
Fully featured framework for fast, easy and documented API development with Flask

Flask RestPlus IMPORTANT NOTICE: This project has been forked to Flask-RESTX and will be maintained by by the python-restx organization. Flask-RESTPlu

Axel H. 2.5k Feb 17, 2021
Flask-Potion is a RESTful API framework for Flask and SQLAlchemy, Peewee or MongoEngine

Flask-Potion Description Flask-Potion is a powerful Flask extension for building RESTful JSON APIs. Potion features include validation, model resource

DTU Biosustain 484 Feb 3, 2021