Hey @nicolaiarocci, I'm trying to find a way to leverage Eve to create versioned documents. This feature might end up being a standalone extension, but I thought I'd start an "issue" here so that there was a recording of the conversion.

I'd like to extend the common pattern of /resource/ and /resource/_id_/ endpoints with /resource/_id_/versions/ and /resource/_id_/versions/_version_/. I'd like to implement versioning by storing shadow copies of a model as opposed to storing the changes between versions. In my intended application, I'll actually be referencing / retrieving a specific version of a resource resource often than I would retrieve the latest version of a resource. Only GET methods would be allowed on the two additional version endpoints, and a PATCH/UPDATE to /resource/_id_/ would also insert a new shadow copy.

I originally liked the idea of storing shadow copies within the object itself, but storing the copies as separate documents in a separate collection probably lends itself better to the MongoDB document size limit Your support for sub-modules lends itself well to storing the shadow copies in a second collection. I think I would then make use of the hooks to automatically write to the shadow collection on update. Could you give me an example of how to write to a different resource from inside of a hook other then the resource in request? I don't see an example of this in the event hooks documentation.

Once the shadow copies are actually stored in the database, the next step is to support a versioned document reference. I guess I'd have to create a custom data type for this and then extend the validation rules.

I'd like to use this versioned resource pattern for many (if not all) of the models in my database. I'd likely create a reusable function to generate the required configuration bits for me. However, it would be pretty darn sweet if all i had to do was flip a "version_controlled" flag in the resource configuration... :)

What do you think? Any suggestions on how to implement something like this?

Thanks!

In my app, the reason I'm wanting to keep versioned documents in the first place is for auditing purposes. Without going into too much detail, events occur in our system, and we process those events according to a constantly-evolving set of rules that are defined in a MongoDB collection that is managed via an Eve API (if I have my wish). "Evolving" means that individual rules are frequently updated but also occasionally DELETED. When a rule is applied to an event, we record it (e.g. "Rule X, version N, was applied.") Later, if anything erroneous is found to have happened, we can go look at exactly how the rule was defined at the time. This currently breaks with DELETE, though, because deletion results in removal of the archived versions of the document, as well.

The workaround I currently have in mind is to basically have a deleted boolean field, and filter them out by default for that resource. This is clunky because it means that deletion would be done with PATCH/PUT and it just leaves the primary collection cluttered. IMO, there needs to be a way to support DELETE on the resource items while still preserving the auditability.

The behavior I would expect is that if item X is deleted, /resource/X would result in a 404, but that /resource/X?version=N would still work. Clearly, though, there's plenty of room for interpretation as to what the right semantics would be, so I haven't done any work on this. I'm just throwing it up here for consideration for the time being.

Take a look at this documentation: http://python-eve.org/authentication.html#user-restricted

Right now, it suggests:

AUTH_FIELD = 'user_id'

and

class BCryptAuth(BasicAuth):
    def check_auth():
        ...
        self.user_id = account['_Id']
        return ...

There are a few problems here. First, this update is required:

if account and '_id' in account:
     self.user_id = account['_id']

Second, if you want a user to be able to query his/her own user object...

• /users/ should return a list with a single user
• /users/<_id> should return a single user

...then the AUTH_FIELD property needs to be a property that the user object has. That's why I would suggest implementing the following change:

• ensure that the user schema has a user_id field, and updating the self.user_id = account['_id'] line to self.user_id = account['user_id']

(EDIT: I had another suggestion, but then I realized that it doesn't actually work)

On the account management page (http://python-eve.org/tutorials/account_management.html), under "Accounts with Token Authentication", it seems like a malicious user can just create an admin or superuser through the "/accounts/" endpoint, which would be troublesome.

Shouldn't the "/accounts/" endpoint be secured so that accounts that are created only have the "user" role? Then, an existing admin or superuser can just manually go into the database and update a newly created user so that he/she has the admin or superuser role.

I might be missing something, so if that is the case please let me know.

Updates:

1. Location: line 287 of base.py getattr(self.app.auth, auth_field) ==> self.app.auth.user_id

2. Location: line 114 of post.py getattr(app.auth, auth_field) ==> app.auth.user_id

The reason:

In auth.py, self.user_id is hard_coded:

def __init__(self):
    self.user_id = None

With curr_req_auth_value = getattr(self.app.auth, auth_field), resource-specified auth_field's will not work.

(NOTE: put.py is already correct)

The downside of the current naming system for Eve's event hooks is that a resource can never be named "items", or else problems will arise.

A more robust naming scheme would be something like this:

"on_getting_resource" "on_getting_resource_%s" % resource "on_getting_item" "on_getting_item_%s" % item_title

Ensure we handle database concurrency in case there are concurrent accesses to the database that modify the same objects. The current implementation will ensure "_etag" is consistent and retry the PUT/PATCH operation if not.

The diff is messy as there's been a lot of indented lines, but in a nutshell the PUT/PATCH operations are surrounded by a while True loop that's continued if there's a mid-air collision, otherwise the loop carries on in execution and either throws or returns.

There was a suggested use of find_and_modify(), but I couldn't find a solution using this function beside the use of etag that would allow us not to modify the database if there was indeed a collision. I also noted when I tried this solution that there was somehow date mismatches between what find_and_modify() returns and the original document which would cause comparisons to fail (some would be offset-aware while other wouldn't).

Let me know what you think.

Currently, the POST and PUT endpoints require the following format:

single object:

POST /user
{
"user1": { "email": "[email protected]", "password": "iRock"}
}

multiple objects:

{
"user1": { "email": "[email protected]", "password": "iRock"},
"user2": { "email": "[email protected]", "password": "madskillz"},
}

This seems quite strange to me, as every API that I've worked with up to this point asks for the API user to specify a single object when posting (Facebook, [Instagram](Instagram also expects single documents to be posted: http://instagram.com/developer/endpoints/comments/), Twitter etc.).

From http://stackoverflow.com/questions/411462/restful-way-to-create-multiple-items-in-one-request:

Although Bulk operations (batch create) are essential in many systems, [they are] not formally addressed by the RESTful architecture style.

From http://stackoverflow.com/a/1081601/1530754:

This is what it might look like: POST /user fname=John&lname=Doe&age=25 The server responds: 200 OK Location: /user/123

From http://stackoverflow.com/questions/7172784/how-to-post-json-data-with-curl-from-terminal-commandline-to-test-spring-rest:

curl -X POST -H "Content-Type: application/json" -d '{"username":"xyz","password":"xyz"}' http://localhost:3000/api/login

AngularJS posts a single object like this as well.

I'd like to POST or PUT an object like so:

POST /user
{ "email": "[email protected]", "password": "iRock"}

• Have collection of more than one item
• Make request to collection index and make note of time T
• Delete 1 item
• Make request to collection index with If-Modified-Since of time T
• Observe resulting HTTP 304

This is very similar to Bug #243, except that the collection is not empty afterwards.

In the short term, can you suggest a way to ignore If-Modified-Since headers on requests to collections?

While updating my Eve app to test #331, I started getting the following error:

[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]   File "/usr/local/lib/python2.7/dist-packages/Eve-0.4_dev-py2.7.egg/eve/flaskapp.py", line 151, in __init__
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]     self.register_resource(resource, settings)
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]   File "/usr/local/lib/python2.7/dist-packages/Eve-0.4_dev-py2.7.egg/eve/flaskapp.py", line 699, in register_resource
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]     self._set_resource_defaults(resource, settings)
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]   File "/usr/local/lib/python2.7/dist-packages/Eve-0.4_dev-py2.7.egg/eve/flaskapp.py", line 545, in _set_resource_defaults
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]     settings['defaults'] = build_defaults(schema)
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]   File "/usr/local/lib/python2.7/dist-packages/Eve-0.4_dev-py2.7.egg/eve/default_values.py", line 39, in build_defaults
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216]     if 'default' in value:
[Mon May 12 05:15:21 2014] [error] [client 10.1.41.216] TypeError: argument of type 'bool' is not iterable

This seems to be related to #327. The build_defaults() function is expecting something to be iterable, but it isn't in my resource schema. I haven't been able to track down in the last fifteen minutes. My best guess is that the function is chocking on my custom validator triggers (which should be valid syntax.) Any ideas, @nicolaiarocci / @graffic?

By the way, I know Eve has a change log, but it'd be really nice to start a migration guide... My app has broken more than once when updating to a new version and it always requires lots of a manual digging. A migration guide could really help with that.

It will be more convenient for composing eve app with existing flask app if eve app is a blueprint. Current eve flask app can be just a thin wrapper of the eve blueprint.

for example:

class EveBlueprint(Blueprint):
   pass

class Eve(Flask):
   register_blueprint(EveBlueprint)

Hi, I need to perform data aggregation, but variables must be passed in the url and not via arguments ( ?aggregate={}).

I have the following hook enabled:

def on_aggregate(endpoint, pipeline):
    print(f'endpoint: {endpoint}')
    print(f'pipeline: {pipeline}')

1. My standard config is:

media_por_user_id = {
    'url': 'views/userid/media',
    'datasource':{
        'source': 'media',
        'aggregation': {
            'pipeline':
                [
                    {
                        '$match': {
                            "user": "$user_id"
                        }
                    }, {
                        '$group': {
                            '_id': '$user',
                            'medias': {
                                '$push': {
                                    'media_filename': '$media_filename',
                                    'media_status': '$media_status'    
                                }
                            }
                        }
                    }
                ]
            }
    },
    'schema': schema_media,
}

Request: http://127.0.0.1/views/userid/media?aggregate={"$user_id":"639b51720ac10cbf4b0a3d43"}

Response:

{"_items": [{"_id": "639b51720ac10cbf4b0a3d43", "medias": [{"media_filename": "r", "media_status": "available"}, {"media_filename": "r1", "media_status": "available"}, {"media_filename": "r2", "media_status": "available"}]}], "_meta": {"page": 1, "max_results": 25, "total": 1}, "_links": {"parent": {"title": "home", "href": "/"}, "self": {"title": "views/userid/media", "href": "views/userid/media?aggregate={\"$user_id\":\"639b51720ac10cbf4b0a3d43\"}"}}}

Hook Output:

endpoint: media_por_user_id pipeline: [{'$match': {'user': '639b51720ac10cbf4b0a3d43'}}, {'$group': {'_id': '$user', 'medias': {'$push': {'media_filename': '$media_filename', 'media_status': '$media_status'}}}}]

Note how $user_id is captured correctly

2-. Trying to pass $user_id through the url:

media_por_user_id = {
    'url': 'views/<regex("[a-f0-9]{24}"):user_id>/media',
    'datasource':{
        'source': 'media',
        'aggregation': {
            'pipeline':
                [
                    {
                        '$match': {
                            "user": "$user_id"
                        }
                    }, {
                        '$group': {
                            '_id': '$user',
                            'medias': {
                                '$push': {
                                    'media_filename': '$media_filename',
                                    'media_status': '$media_status'    
                                }
                            }
                        }
                    }
                ]
            }
    },
    'schema': schema_media,
}

Request: http:/127.0.0.1/views/639b51720ac10cbf4b0a3d43/media

Response:

{"_items": [], "_meta": {"page": 1, "max_results": 25, "total": 0}, "_links": {"parent": {"title": "home", "href": "/"}, "self": {"title": "views/<regex(\"[a-f0-9]{24}\"):user_id>/media", "href": "views/639b51720ac10cbf4b0a3d43/media"}}}

Hook Output:

endpoint: media_por_user_id, pipeline: [{'$match': {'user': '$user_id'}}, {'$group': {'_id': '$user', 'medias': {'$push': {'media_filename': '$media_filename', 'media_status': '$media_status'}}}}]

in this case the value of $user_id is not captured.

I have tried many modifications, but without success. information available in this regard is scarce. So I'm evaluating sending an empty pipeline to the aggregation hook. and handle it at that level, using the following approach:

from flask import request

def on_aggregate(endpoint, pipeline):
    print(f'endpoint: {endpoint}')
    print(f'pipeline: {pipeline}')
    url = request.url
    # extract $user_id from url
    # contruct pipeline
    return True

But before going to that stage I would like to have clarification if it is a configuration error or Eve does not support this feature.

Thanks

Hi @nicolaiarocci,

Too many times I have needed an endpoint to be used as a configuration point for other applications, for example:

schema_transcoding_config = { 'encoding_enabled': {'type': 'boolean', 'default': True}, 'threads': {'type': 'int', 'default': 1}, }

endpoint only needs one item, and old ones are deleted periodically.

I suggest the following new feature: At the endpoint level, a new setting "one_document_endpoint" (for example): An endpoint configured with "one_document_endpoint", must ensure that only one document is kept, in case of Create/Update Document, old documents must be deleted. DELETE operations on Collection/Document not allowed, Bulk operation not allowed.

Please consider whether this feature is appropriate. I am a volunteer for its implementation.

Thanks!

This issue tracker is a tool to address bugs in Eve itself. Please use Stack Overflow for general questions about using Eve or issues not related to Eve (see http://python-eve.org/support).

If you'd like to report a bug in Eve, fill out the template below. Provide any any extra information that may be useful / related to your problem. Ideally, create an MCVE, which helps us understand the problem and helps check that it is not caused by something in your code.

Expected Behavior

When uuidRepresentation is not set in MONGO_OPTIONS generating etags should still be possible.

-- app.py
my_settings = {
    'MONGO_HOST': 'localhost',
    'MONGO_PORT': 27017,
    'MONGO_DBNAME': 'the_db_name',
    'DOMAIN': {'contacts': {}}
}

app = Eve(settings=my_settings)
app.run()


-- some_test.py
from eve import post_internal
post_internal('contacts', {'a_valid_contact': 'this must be'}, skip_validation=True)

Actual Behavior

Since no uuidRepresentation is present in MONGO_OPTIONS, the default option is selected, which is an int, not a string. A simple fix would be changing line 350 in eve/utils.py from config.MONGO_OPTIONS.get("uuidRepresentation", UuidRepresentation.STANDARD) to config.MONGO_OPTIONS.get("uuidRepresentation", "standard")

or add a field 4 to the uuid_map on line 343:

uuid_map = {
            "standard": UuidRepresentation.STANDARD,
            4: UuidRepresentation.STANDARD,
            "unspecified": UuidRepresentation.UNSPECIFIED,
            "pythonLegacy": UuidRepresentation.PYTHON_LEGACY,
            "csharpLegacy": UuidRepresentation.CSHARP_LEGACY,
            "javaLegacy": UuidRepresentation.JAVA_LEGACY,
        }

    def uuid_representation_as_string():
        uuid_map = {
            "standard": UuidRepresentation.STANDARD,
            "unspecified": UuidRepresentation.UNSPECIFIED,
            "pythonLegacy": UuidRepresentation.PYTHON_LEGACY,
            "csharpLegacy": UuidRepresentation.CSHARP_LEGACY,
            "javaLegacy": UuidRepresentation.JAVA_LEGACY,
        }
>       return uuid_map[
            config.MONGO_OPTIONS.get("uuidRepresentation", UuidRepresentation.STANDARD)
        ]
E       KeyError: 4

Environment

• Python version: 3.9.2
• Eve version: 2.0.1

Since Flask 2.2, all tests relying on app.add_url_rule are failing. Flask changelogfor v2.2 reports significant changes.

For the time being, I'm pinning Flask dependency to <2.2.

Anybody willing to take up this task?

Environment

• Python version: Python 3.9.2
• Eve version: Eve==2.0

EXTENDED_MEDIA_INFO = ['content_type', 'name', 'length'] RETURN_MEDIA_AS_BASE64_STRING = False RETURN_MEDIA_AS_URL = True MEDIA_ENDPOINT = 'media'

I'm trying to delete files on the media endpoint, with deleteitem_internal, but apparently deleteitem_internal does not consider the media endpoint when is enabled, the same could be happening for other internal methods. It will also be valuable if when deleting an item from media (with deleteitem_internal), related items in the fs.files and fs.chunks collections are also deleted

Example:

fs.files:

_id: 62fa8c38f4c4954b8fef7e71 filename: "pic.jpg" contentType: "image/jpg" chunkSize: 261120 length: 48640 uploadDate: 2022-08-15T18:11:04.278+00:00

fs.chunks: _id: 62fa8c38f4c4954b8fef7e72 files_id :62fa8c38f4c4954b8fef7e71 n: 0 data: BinData(0, '/9j/4AAQSkZJRgABAgAAAQABAAD//gAQTGF2YzU4LjkxLjEwMAD/2wBDAAgKCgsKCw0NDQ0NDRAPEBAQEBAQEBAQEBASEhIVFRUS…')

media_lookup = {'_id': 62fa8c38f4c4954b8fef7e71 } deleteitem_internal('media', concurrency_check=False, suppress_callbacks=True, original=None, **media_lookup)

File "/home/trz/pushpull/lib/python3.9/site-packages/eve/methods/delete.py", line 99, in deleteitem_internal resource_def = config.DOMAIN[resource] KeyError: 'media'

same error is triggered when trying : deleteitem_internal('fs.files', concurrency_check=False, suppress_callbacks=True, original=None, **media_lookup)

Regards

Actual Behavior

MONGO_URI's username, password, and authSource in uri are not parsed correctly

https://github.com/pyeve/eve/blob/master/eve/io/mongo/flask_pymongo.py#L76

It uses dbname as authSource, but the authSource should be parsed from options https://pymongo.readthedocs.io/en/stable/api/pymongo/uri_parser.html#pymongo.uri_parser.parse_uri

See URI format https://www.mongodb.com/docs/manual/reference/connection-string/

Environment

• Python version: 3.10
• Eve version: 2.0

I am willing to open a PR to resolve the issue. Thanks