Closes https://github.com/Scille/umongo/issues/129.

The failing test is due to an issue in marshmallow that was fixed since 3.0.0rc1 was released. It'll be fixed with next marshmallow release.

Update API to support v2.0.0 of motor. This involves changing tests that use the deprecated count() cursor method to use either count_documents() or to_list() instead, and removing the callback keyword argument from the to_list wrapper.

I'm afraid there's an issue with allow_none fields.

For both serialization and deserialization to/from Mongo, if the value is None, None should be returned and all _(de)serialize_from_mongo overridden methods should be skipped, as they don't deal with the allow_none case and either break (e.g. StrictDateTime) or return a nullish object (empty list, empty dict).

See attached fix proposal.

I didn't code any test as I'm in a bit of a rush and I'd rather have your feedback first.

Parameters of as_marshmallow_schema should be passed to nested Schemas.

In this implementation, I pass them to all fields, so that EmbeddedField gets it and pass it when calling as_marshmallow_schema.

Nothing really complicated here. I had to modify as_marshmallow_field's signature, hence the modifications in unrelated fields.

Note that I changed kwargs into field_kwargs in those fields to make it obvious to the reader it is not the same kwargs we're talking about. It would work without the rename, but it would look like a mistake at first sight:

    def as_marshmallow_field(self, params=None, mongo_world=False, **kwargs):
        # Oh, we're overwriting kwargs! Let's file a bug!
        kwargs = self._extract_marshmallow_field_params(mongo_world)

Another option was to use _ but I'm not sure it is so common for kwargs:

    def as_marshmallow_field(self, params=None, mongo_world=False, **_):
        kwargs = self._extract_marshmallow_field_params(mongo_world)

I preferred to rename the field kwargs inside the method:

    def as_marshmallow_field(self, params=None, mongo_world=False, **kwargs):
        field_kwargs = self._extract_marshmallow_field_params(mongo_world)

I could rework this if you have a better idea.

This PR does not address those two related issues I assigned myself a while ago: https://github.com/Scille/umongo/issues/64, https://github.com/Scille/umongo/issues/65...

Currently, when an unknown field is found in database using a strict DataProxy, the KeyError is not caught. When dealing with deeply nested documents, it can be a bit cumbersome to track the faulty EmbeddedDocument.

This PR creates a dedicated exception that prints the name of the data proxy to ease the debug procedure.

Implementation proposal for the reset_missings feature.

Removes value from document if loadable in the Schema but missing in input data.

user = User(name='Hannibal', team="A-Team")
user.update({'name': 'Looping'})
assert user.team == "A-Team"
user.update({'name': 'Chuck Norris'}, reset_missings=True)
assert user.team == None

I'm afraid there's an issue with embedded document inheritance when deserializing from DB: the _cls attribute is ignored. Looks like it only works when loading from outside world, not from DB.

This dirty fix seems to do the trick in my use case, but it is certainly not the right implementation. It is mostly a copy-paste from _deserialize to _deserialize_from_mongo. It should at least be factorized.

I'm not sure the instance should be created using to_use_cls(**value). It didn't work for me (complained about an unknown field name due to a dump_only field), so I randomly tried with build_from_mongo and it happened to work...

I think I had this case covered in my PR since I overrode build_from_mongo in EmbeddedDocumentImplementation to pick the right class. But I'm not sure this is relevant to your implementation. I basically copied inheritance related stuff from Document to EmbeddedDocument, while you addressed inheritance in EmbeddedField, AFAIU.

BTW, isn't there a way in EmbeddedField to get embedded_document_cls at init time (when declaring the field in the schema)?

Those attributes come from marshmallow, which is all about serialize/deserialize a document:

• missing is used when the field is not present in the dict to deserialize
• default is used when the field is not present in the document to serialize

However in umongo the focus shifts toward the document itself, so those terms seems a bit clunky

• the missing object is already used inside umongo's DataProxy to represent field not present in MongoDB. Hence the missing attribute should mean what value to return if the value is missing
• the default attribute sounds like the value to set by default to the field when creating a new document

In a nutshell the meaning of missing and default are reversed between mashmallow and umongo...

What I'm thinking to do:

• Remove the missing attribute (in fact just hide it from abstract.BaseField constructor and documentation)
• Only use the default attribute for both missing and default (in marshmallow's logic)
• Add methods to get from the umongo document an equivalent pure mashmallow Schema (see #34 )

The idea is to hide mashmallow logic to expose a more consistent API from the umongo user's point of view. Then provide a way to get back a pure Marshmallow Schema when needed to do all the custom.

example:

@instance.register
class Person(Document):
    name = fields.StrField(default='John Doe')

p = Person()
# Default is set
assert p.name == 'John Doe'
# Default value will be written in database
assert p._data.get('name') == 'John Doe'
# If we want more cunning behavior (e.g. only save in database non-default value)
# we should use a `MethodField` to define custom method for serialization/deseriazation

del p.name
assert p._data.get('name') == missing
# If not present on database, we switch back to default value as well
assert p.name == 'John Doe'

# Now it's customization time !
PersonSchema = p.get_marshmallow_schema()
class MyCustomSchema(PersonSchema):
    ...

# It could be also useful to provide method to get a specific field
class MyCustomSchema2(marshmallow.Schema)
    name = p.get_marshmallow_field('name')

@lafrech What do you think ?

Currently, the Schema of a Document can be obtained from Document.Schema. However, this Schema is made to [|de]serialize between "DB / data_proxy" and "OO world", not between "OO world" and "client/JSON world". (See diagram in the docs).

In other words, uMongo is made to be used like this:

    document.dump(schema=schema)
    # which is equivalent to
    schema.dump(document._data._data)
    # but not equivalent to
    schema.dump(document)

The difference being that the data_proxy does not behave like the document:

• It may have keys named differently is "attribute" is used
• document returns None if a key is missing
• ...

Therefore, using the Schema to serialize a Document may work but it currently has corner cases.

@touilleMan confirms that the ability to export a Marshmallow Schema without the uMongo specificities is in the scope of uMongo and is a feature we want to have.

The idea could be to add a method or attribute to the document to provide that "cleaned up" Schema.

I'm opening this issue to centralize the reflexions about that.

Currently, here are the issues I found:

• [ ] check_unknown_fields raises ValidationError if passed a dump_only field even if value is missing, which is an issue when validating a document before deserialization. (Bug report: https://github.com/Scille/umongo/issues/18, PR: https://github.com/Scille/umongo/pull/19)
• [x] Some uMongo fields fail if value is None. Maybe this one is unrelated but just happened to occur while calling `schema.dump(document) (PR: https://github.com/Scille/umongo/pull/32)
• [ ] Fields with "attribute" not being None will fail because the Schema tries to find the value in "attribute", while in the document, it is available at the field's name. To avoid this, we could set "attribute" to None in all the fields. (PR: https://github.com/Scille/umongo/pull/33)

https://github.com/Scille/umongo/pull/33 drafts a way of exporting the Marshmallow Schema from the document.

If you like to find embedded documents the order of the kv pairs in the embedded document is essential as 'find' will only match documents with the same order. As the mongo docs say (https://docs.mongodb.com/manual/tutorial/query-documents/#exact-match-on-the-embedded-document):

Equality matches on an embedded document require an exact match of the specified , including the field order.

pymongo thus supports python OrderedDict: http://stackoverflow.com/a/30787769/4273834

as does marshmallow: http://marshmallow.readthedocs.io/en/latest/quickstart.html#ordering-output

Is there a way to use OrderedDict in umongo?

I'm pulling my hair with datetime TZ awareness issues.

I initiate the connection to MongoDB with tz_aware = False. I'm no expert about this, and I had never thought much about it before now, but from what I gathered, it seems like a reasonable choice to make. Besides, it's pymongo's default (but flask-PyMongo's hardcodes tz_aware to True).

When a document is pulled from the DB, its DateTimeField attribute's TZ awareness depends only on MongoClient's tz_aware parameter (no Marshmallow schema involved):

tz_aware=True -> pymongo provides an aware datetime -> umongo returns an aware datetime tz_aware=False -> pymongo provides a naive datetime -> umongo returns a naive datetime

This is one more reason to set tz_aware = False, because I'm passing a document/object to a lib that expects a naive timezone. (OAuth2 lib expects expires timestamp to be naive and compares it to datetime.utcnow() (see code).) I suppose I could alternatively modify my getters to make all datetimes naive before returning tokens/grants, but on some use cases, it could get cumbersome.

Marshmallow, however, returns every datetime as TZ aware (doc). For this reason, umongo's DateTimeField's _deserialize method returns a TZ aware datetime. Since I'm using it (via webargs) to parse the inputs to my API, I'm getting TZ aware datetimes. Likewise, calling load() on a document will use _deserialize and result in a TZ aware datetime.

So if I load a date, it becomes TZ aware. Therefore, to compare it to a date from the database, this one needs to be aware as well.

Should I understand that umongo is meant to be used with tz_aware=True, so that dates fetched from the database can be compared to dates loaded thought Marshmallow schemas?

Could there be a flag/meta allowing to specify if a DateTimeField should return a naive datetime?

I made a quick and dirty patch to DateTime's _deserialize to remove the TZ from the returned output.

    dt = super()._deserialize(value, attr, data)
    return dt.replace(tzinfo=None)

This seems to work on my use case. However, the day we complete pure Marshmallow schema export, the exported schema I pass to webargs for API input parsing won't have that feature. Unless this is added to Marshmallow as well. (I asked there about it there: https://github.com/marshmallow-code/marshmallow/issues/520.)

Feedback welcome. Those TZ issues are new to me, so I may be totally misguided.

When I run unit tests involving umongo, I get the following warning message (actually, if I run the tests via pytest instead of unittest, I get hundreds of the same warning):

/Users/tiltowait/Library/Caches/pypoetry/virtualenvs/inconnu-qKm_l4La-py3.10/lib/python3.10/site-packages/umongo/data_proxy.py:160: RemovedInMarshmallow4Warning: The 'missing' attribute of fields is deprecated. Use 'load_default' instead.
   if callable(field.missing):
 /Users/tiltowait/Library/Caches/pypoetry/virtualenvs/inconnu-qKm_l4La-py3.10/lib/python3.10/site-packages/umongo/data_proxy.py:163: RemovedInMarshmallow4Warning: The 'missing' attribute of fields is deprecated. Use 'load_default' instead.
   self._data[mongo_name] = field.missing
 /Users/tiltowait/Library/Caches/pypoetry/virtualenvs/inconnu-qKm_l4La-py3.10/lib/python3.10/site-packages/umongo/data_proxy.py:161: RemovedInMarshmallow4Warning: The 'missing' attribute of fields is deprecated. Use 'load_default' instead.
   self._data[mongo_name] = field.missing()

Is there any plan to address this? It looks like a simple fix, but I'm not familiar with umongo's code to be confident in that assessment.

It seems txmongo is increasingly hard to support.

txmongo cannot be imported with pymongo >= 4 (see https://github.com/twisted/txmongo/issues/278)

Even if the import issues are fixed, txmongo internally uses an OP_QUERY command which is incompatible with Mongo server instances >= 6.0 (see https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/#footnote-op-query-footnote).

txmongo is not actively developed.

Should txmongo support remain, or just be deprecated?

I would like to reference binary blobs larger than 16 MB in size in my uMongo documents.

Seems like it should be possible to implement a sort of GridFS reference as a type of field. Has anybody tried this, or put any thought to how it might be done?

Relates to https://github.com/Scille/umongo/issues/37

I am trying to set an EmbeddedDocument as a value in a DictField, which according to #99 should work. However, when I do so, I receive the following error:

bson.errors.InvalidDocument: cannot encode object: <object EmbeddedDocument objects.items.MockItem({'uuid': UUID('cee1f505-1014-4bc8-9095-7047c869587c'), '_token_id': '1583', 'durability': 100})>, of type: <Implementation class 'objects.items.MockItem'>

The DictField's definition:

    _slots = fields.DictField(attribute="slots")

The MockItem EmbeddedDocument:

instance.register
class MockItem(EmbeddedDocument):
    """An item with associated durability."""

    uuid = fields.UUIDField(default=uuid.uuid4)
    _token_id = fields.StrField(required=True, attribute="tokenId")
    durability = fields.IntField(default=100)
    _item = None

    # Custom __getattr__ and __hash__ omitted

Stepping through the debugger, prior to attempting to commit(), the _slots dict has the following value:

<object umongo.data_objects.Dict({'base': None, 'pet': None, 'cloak': None, 'oh': None, 'body': None, 'hair': None, 'ears': None, 'face': None, 'legs': None, 'feet': None, 'chest': None, 'hands': None, 'waist': None, 'head': None, 'mh': <object EmbeddedDocument objects.items.MockItem({'uuid': UUID('d28a07c1-49b3-4099-be44-4733d04420cd'), '_token_id': '-104', 'durability': 100})>})>

In my AsyncioMotorClient, I am setting uuidRepresentation="standard", but I get the error with or without the UUID field present. What am I doing wrong?

as_marshmallow_schema returned the following message for the ConstantField attribute:

@instance.register
class Dog(Document):
    breed = fields.ConstantField("Mongrel")

DogMaSchema = Dog.schema.as_marshmallow_schema()

Traceback (most recent call last):
  File "...\AppData\Local\Programs\Python\Python38\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 5, in <module>
  File "...\.venv\lib\site-packages\umongo\abstract.py", line 66, in as_marshmallow_schema
    nmspc = {
  File "...\.venv\lib\site-packages\umongo\abstract.py", line 67, in <dictcomp>
    name: field.as_marshmallow_field()
  File "...\.venv\lib\site-packages\umongo\abstract.py", line 209, in as_marshmallow_field
    m_field = m_class(**field_kwargs, metadata=self.metadata)
TypeError: __init__() missing 1 required positional argument: 'constant'

All the examples show we can add the data entry to database one by one. But it is possible to use a method similar to insertMany to add many docs at once?