What does these changes do:

1. Fix bug with dereference function: it does not dereference refs in embedded documents from list ListField(EmbeddedDoc(ReferenceField(X))). It performs an additional call to db using dereference_id function while accessing the field.
   • Add test to reproduce.
   • Add fix.
2. Fix false positive test test.test_dereference.DereferenceTestCase.test_reference_not_found. It saves an empty Post model which has title as a pk. That forces to store post document with _id as an ObjectId. So post.delete() does not actually delete any documents from database.

Checked and validated type of related_field in EmbeddedDocumentField and EmbeddedDocumentListField. Raised Value error the same was as RelatedModelFieldsBase.

This PR introduces a new class pymodm.base.models.LazyDecoder to help with bookkeeping associated with tracking the encoding of MongoModel objects being retrieved from and written to the database.

https://jira.mongodb.org/browse/PYMODM-48

Disallowing unknown fields seemed like the right choice at first, but now there are complaints about this decision (including http://stackoverflow.com/q/41472478/3271558). In the future, we could make validating unexpected fields optional, by creating a new option on the class Meta. I'm going to hold of on doing so until there's a clear need for that, though.

At first I tried caching the default value in inst._data, but that broke a bunch of tests, because reading a default value of a field is not supposed to result in saving that default value to MongoDB.

So instead I added inst._defaults. This doesn't get saved to MongoDB, but __get__ reads from it instead of calling .get_default() again.

What do these changes do:

• Remove ReferenceField._is_instance property as it is useless (field is shared between model instances).
• Rewrite dereferencing to fix some issues:
  • Dereference unhashable ids (auto dereferencing still does not work for this case as reference id is treated as dereferenced document not as an id).
  • Set reference field value to None if there is no document with such id (as auto dereferencing does).
  • Fix bug when some fields reference to different collections but have the same id. So that only one of that field will be set with correct value as the document_map was {id --> document}. Example:

class User(MongoModel):
    name = fields.CharField(primary_key=True)
class Category(MongoModel):
    title = fields.CharField(primary_key=True)
class Post(MongoModel):
    user = fields.ReferenceField(User)
    category = fields.ReferenceField(Category)
user = User(name='Bob').save()
category = Category(title='Bob').save()
post = Post(user=user, category=category)
with no_auto_dereference(post):
    post.refresh_from_db()
    dereference(post)
    # one of the follow statements will fail as both post.user and post.category
    # will be User (or Category).
    assert isinstance(post.user, User)
    assert isinstance(post.category, Category)

• Dereference DBRefs in dict-like fields. As there is no field for DBRef the only way to store it is within dict-like fields (DictField, OrderedDictField). So dereference traverses such containers to find DBRef values and try to dererference them.

What do these changes do:

• Add auto_create_indexes option to control whether model's indexes should be created when model class definition is evaluated or not. The default value is True so that default behavior would be fully compatible with current one. With this option it is possible to control indexes creation and makes models classes definition independent from connect function call before any model class (mainly with indexes) is evaluated.
• Add QuerySet.create_indexes method to explicitly create indexes which are defined in model's Meta. As a bonus this method can be used with context_managers.
• Add tests to test indexes creation.

Sample code

import uuid
from pymodm import connect, MongoModel, fields


class User(MongoModel):
    _id = fields.UUIDField(primary_key=True)
    name = fields.CharField()
    job_title = fields.CharField()


class Asset(MongoModel):
    _id = fields.UUIDField(primary_key=True)
    user = fields.ReferenceField(User)
    comments = fields.CharField()


if __name__ == "__main__":
    connect('mongodb://*:*@<host>/test?authSource=admin')
    user = User(
        _id=uuid.uuid4(),
        name='Jack',
        job_title='Software Engineer',
    )
    user.save()
    asset_id = uuid.uuid4()
    Asset(
        _id=asset_id,
        user=user,
        comments='Old comments',
    ).save()
    asset = Asset.objects.get({'_id': asset_id})
    asset.comments = 'New comments'
    asset.save()
    # The ReferenceField will be dereferenced each time asset.user is used
    print(asset.user.name)
    print(asset.user.job_title)

Analysis

TopLevelMongoModel.save() calls MongoModelBase.full_clean() and MongoModelBase.to_son(). Both of them use no_auto_dereference context manager and get the values of the fields.

ReferenceField.__get__() calls ReferenceField.to_python()

    def __get__(self, inst, owner):
        value = super(ReferenceField, self).__get__(inst, owner)
        MongoModelBase = _import('pymodm.base.models.MongoModelBase')
        if inst is not None and isinstance(inst, MongoModelBase):
            return self.to_python(value)
        return self

ReferenceField.to_python() calls ReferenceField.dereference_if_needed()

    def to_python(self, value):
        # Default/blank values can be returned immediately.
        if self.is_blank(value):
            return value

        # Attempt casting to referenced model.
        if isinstance(value, dict):
            try:
                return self.related_model.from_document(value)
            except (ValueError, TypeError):
                pass

        return self.dereference_if_needed(value)

ReferenceField.dereference_if_needed() returns self.related_model._mongometa.pk.to_python(value) since no_auto_dereference context manager is used. The value of key 'user' in asset's _python_data is set to user's _id which is an uuid. So when asset.user is used later, because of there already is a key 'user' in asset's _python_data, super(ReferenceField, self).__get__(inst, owner) in ReferenceField.__get__() will return user's _id. asset.user will be dereferenced again and again.

    def dereference_if_needed(self, value):
        # Already dereferenced values can be returned immediately.
        if isinstance(value, self.related_model):
            return value

        # Attempt to dereference the value as an id.
        if self.model._mongometa._auto_dereference:
            dereference_id = _import('pymodm.dereference.dereference_id')
            return dereference_id(self.related_model, value)

        return self.related_model._mongometa.pk.to_python(value)

This issue will result in a critical performance issue if getting the value of a ReferenceField after saving the model or using any other methods which get the ReferenceField's value within no_auto_dereference context manager.

As per my understanding of your code, I have added the distinct method to return the distinct values for particular field. I have added the distinct method in queryset.py file

Here's one solution to PYMODM-68: instead of creating the indexes when the class is created, we could create them when the model's collection is first used. I think this is similar to what MongoEngine does.

Another option would be to give the user the option to disable automatic index creation, like MongoEngine's auto_create_index=False.

Fix for pymongo.errors.OperationFailure: the $sort key specification must be an object. This occurs when a queryset has been given an order_by then an aggregation is added. self._order_by is a list of tuples, but must be an object. The order of keys in order_by matters, so we must use SON.

I created PYMODM-119, expecting to_son to automatically dereference fields.

@ShaneHarvey explained why that cannot be, but agreed that this can be clarified in the docs.

Here is a one line addition to the docstring of to_son

It does not dereference fields.