Hi guys,

Now everything is fixed and is fully compatible for both PyMongo 2.X and 3.X

I'm awaiting some feedback for:

1. ~~The way I introduced differential behaviour based on PyMongo version~~
2. ~~The behaviour for save() when a document is created with a given primary key, source of 5 of remaining issues. Here particularly, I have removed the check for self._created since we used to modify this why changing/switching db on the fly. This whole behaviour has changed in PyMongo3 and IMHO doesn't make sense anymore.~~
3. ~~Your opinion if the test_hint failure could point to a bug in PyMongo3~~
4. ~~A hand on the other error on test_geo_indexes_recursion~~
5. ~~A problem with binary field used as primary index that was always there and only discovered now by a better test that became mandatory by the use of a work-around concerning a PyMongo 3.0 bug already fixed in the upcoming 3.0.1~~

Cheers, Matthieu

Mongoengine is tested on different PyMongo versions, different Python versions... This pull-request add a tox.ini file allowing to run the test suite on every supported version composition . This allow to do proper testing before submitting pull-requests without having to wait for TravisCI cross-versions testing.

To run the test in tox, simply run:

$ tox

It will create a virtualenv for each combination and run the same test suite (like TravisCI).

I threw together this patch as an idea to fix #934.

I haven't dug too deep into Mongoengine to know what else this might affect, but I used _from_son in BaseDocument object as my injection point. The idea being that if one tries to define in code a field that does not exist, an error is still raised, but if one pulls in data from a PyMongo object, we don't raise an error.

fix mark_as_changed: rm lower level changed fields otherwise may cause conflict update error this issue appears when retrieve from db

In [34]: class Foo(mongoengine.Document):

    bar = mongoengine.DictField()
   ....:     

In [35]: Foo.objects.delete()
Out[35]: 1

In [36]: foo = Foo()

In [37]: foo.save()
Out[37]: <Foo: Foo object>

In [38]: foo = Foo.objects.first()

In [39]: foo.bar['what'] = 'ever'

In [40]: foo.bar['other'] = {}

In [41]: foo._changed_fields
Out[41]: ['bar.what', 'bar.other']

In [42]: foo.bar['other']['what'] = 'ever'

In [43]: foo._changed_fields
Out[43]: ['bar.what', 'bar.other', 'bar.other.what']

In [44]: foo.save()
---------------------------------------------------------------------------
OperationError                            Traceback (most recent call last)
<ipython-input-44-e6645f54a3c0> in <module>()
----> 1 foo.save()

.../mongoengine/document.pyc in save(self, force_insert, validate, clean, write_concern, cascade, cascade_kwargs, _refs, save_condition, **kwargs)
    365                 message = u'Tried to save duplicate unique keys (%s)'
    366                 raise NotUniqueError(message % unicode(err))
--> 367             raise OperationError(message % unicode(err))
    368         id_field = self._meta['id_field']
    369         if created or id_field not in self._meta.get('shard_key', []):

OperationError: Could not save document (Cannot update 'bar.other' and 'bar.other.what' at the same time)

When id field presented in Embedded document, changes inside this document was ignored in method _get_changed_fields because it's id was added to inspected array before processing it fields. Also modify unittests for _delta method, which fails without code modification.

I use connect(host='mongodb://username:password@localhost:port/database') in my django settings.py. Run server then got a report:

mongoengine.connection.ConnectionError: Cannot connect to database default :
False is not a read preference.

Is this a bug or not? Write in python 3.4.3 with django 1.8 and mongoengine 0.9.0

I'm trying to delete a document with self.delete() but it raises and error: File "/Users/garito/TimeFounder/App/0.3/env/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 467, in delete return result["n"] TypeError: 'NoneType' object has no attribute 'getitem'

I've checked the result with print result and it is None The deletion is made ok but the try to return result["n"] raises and error because result is None

Did I missed something or did I find a bug?

Thanks!

If an undefined field is supplied to a document instance, a FieldDoesNotExist Exception will be raised. This way, you will be notified when you provide a field that does not exist.

ATM when you do this:

MyDoc(undefined_field="test")

It will create a document instance for MyDoc. However, if undefined_field does not exist, the value test won't end up anywhere. You'll think the object is successfully created with the provided values, but actually it isn't.

To remedy this situation, an exception will be raised. This way you'll be noticed about your mistake in development. If the document instance is created dynamically on bases of an API call data for example, then you can catch the exeption and notify the API user that they have provided an undefined field.

I am new to mongoengine, but this doesn't make any sense to me, that when I call my my_update() function, the user's updated_at field doesnt get updated but other fields do. here is my model:

class User(db.Document):
    username = db.StringField(required=True, unique=True, max_length=20)
    created_at = db.DateTimeField(default=datetime.datetime.utcnow())
    updated_at = db.DateTimeField()
    friend_list = ListField(StringField(max_length=100))

when I do a save, it saves the new friend_list correctly but it keeps the old updated_at field, and that one will never get updated again.

def my_update(user_id):
    form = UserForm()
    user = User.objects.get_or_404(id=user_id)
    user.friend_list = insert_random_data()
    user.updated_at = datetime.datetime.utcnow()
    user.save()
    return users = User.objects.order_by('-updated_at', '-created_at')

so if I run my_update a few times, it will update the friend_list each time, but the update_at field keeps staying the same !!! and I have no idea. I am really curious why is it behaving like this !

Hi guys,

Several times I ran into some cases where I came across these init() method returning a value, or some unused statements or variables, as well as many pep8 issues. It is never great to make a big change, but since we are now cleaning so many PRs and there will be anyway a big PR with #946 for next version, I thought it could be a good time to clean generally some code.

I took some time to really read once (quickly) all the code line by line. It allowed me also to find some issues or strange behaviour and to add some TODOs.

I would also like to take the opportunity to tackle one other point. We want to be PEP8 conform and this is very good. However one point is at least vague: line length. We have in our code some pieces which are strictly limited to 79 characters and some others to 119.

PEP8 recommends 79 characters, however Django makes some exceptions there: https://docs.djangoproject.com/en/1.8/internals/contributing/writing-code/coding-style/

Should we include the same point in our contribution guidelines? Some PRs have a lot of code doing only line breaks clean-up (which I tried to avoid in this cleaning effort to avoid unnecessary changes). And at the same time, I see a lot of places in the code where we have line breaks that do make the code significantly harder to read. @rozza @DavidBord @thedrow @seglberg @yograterol can I have your opinion on that point?

Fix for #931 issue. When model restored from database and there is a nested MapField there is an error in wrong key for changes when trying to add/change item in lower-level MapField.

Here is the class hierarchy:

class DataItem(EmbeddedDocument):
    key = StringField(required=True)
    value = StringField(required=True)

class MyDoc(Document):
    data = MapField(
        field=EmbeddedDocumentField(DataItem)
    )
    

And the data in db:

{
    "_id" : ObjectId("63a87cb374d6bc3efd9ea570"),
    "data" : {
        "key" : {
            "key" : "1",
            "value" : "test"
        }
    }
}
{
    "_id" : ObjectId("63a87cb374d6bc3efd9ea571"),
    "data" : {
        "a" : {
            "key" : "2",
            "value" : "test"
        }
    }
}

The query MyDoc.objects(data__a__value="test").first() succeeds while the query MyDoc.objects(data__key__value="test").first() fails with mongoengine.errors.InvalidQueryError: Cannot resolve field "value"

QuerySet class in MongoEngine does not have .model key as the in the QuerySet class as defined in django.db.models.query.QuerySet.

adding this field to queryset on initialization could allow for better compatibility with DRF and related packages.

We've run into an issue where reload, save, and update throw an error when they dereference the abstract field from a ReferenceField.

It works fine when you initially save the object or get it from the DB but fails if you modify and then save it.

It also works if you use setattr to set the attribute.

Since we use an abstract document, it doesn't exist in the database, so dereference method fails.

Traceback

>>> demo()
Company
456
changed to the second title
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/code/issue.py", line 55, in demo
    issue.reload()  # works
  File "/app/venv/lib/python3.10/site-packages/mongoengine/document.py", line 745, in reload
    self._qs.read_preference(ReadPreference.PRIMARY)
  File "/app/venv/lib/python3.10/site-packages/mongoengine/queryset/base.py", line 844, in select_related
    return queryset._dereference(queryset, max_depth=max_depth)
  File "/app/venv/lib/python3.10/site-packages/mongoengine/dereference.py", line 102, in __call__
    self.object_map = self._fetch_objects(doc_type=doc_type)
  File "/app/venv/lib/python3.10/site-packages/mongoengine/dereference.py", line 197, in _fetch_objects
    references = get_db()[collection].find({"_id": {"$in": refs}})
  File "/app/venv/lib/python3.10/site-packages/pymongo/database.py", line 237, in __getitem__
    return Collection(self, name)
  File "/app/venv/lib/python3.10/site-packages/pymongo/collection.py", line 207, in __init__
    raise TypeError("name must be an instance of str")
TypeError: name must be an instance of str

Example

from mongoengine import StringField, Document, ReferenceField


class Owner(Document):
    meta = {"abstract": True}
    name = StringField()


class Company(Owner):
    # meta = {"strict": False}

    key = StringField()


class Issue(Document):

    title = StringField(required=True)
    owner = ReferenceField(Owner)  # since Owner is the abstract class; it is not registered in the database


def demo():
    company = Company(name="Company", key="456")
    company.save()
    issue = Issue(title="image", owner=company).save()

    print(issue.owner.name)  # prints "Company"
    print(issue.owner.key)  # prints "456"

    setattr(issue, "title", "second title")
    issue.save()
    print("changed to second title")

    issue.reload()  # fails

    # also fails
    issue.text = "another text"
    issue.save()  # fails

Calling len() on a QuerySet object will fetch and cache all results from the database, returning the length of the cached results. If len() is called during iteration, it will cause the iteration to end prematurely. An example:

function test_len(run_len):
    # A collection with enough documents that the cursor will perform multiple db fetches (N > 100).
    docs = SomeDocument.objects()
    run_len = False
    i = 0
    for doc in docs:
        i = i + 1
        _ = run_len and len(docs)  # This causes the issue.

    assert i == docs.count(), i  # When run_len is True, i == 100 < docs.count().

test_len(False)  # Works as expected.
test_len(True)   # Demonstrates issue.

This issue was introduced in commit https://github.com/MongoEngine/mongoengine/commit/9251ce312bc0545d3b86224e35a913029a86695e while resolving issue https://github.com/MongoEngine/mongoengine/issues/247.

save() not working in an EmbeddedDocumentList in nested embedded document, as _instance is an EmbeddedDocument instead of Document

Reproducer:

class TestK2(EmbeddedDocument):
  values = EmbeddedDocumentListField(StringField, default=[])

class TestK1(EmbeddedDocument): 
  k2 = EmbeddedDocumentField(TestK2)

class Main(Document):
  part = EmbeddedDocumentField(TestK1)

def main(): 
  main = Part()
  main.part.k1.k2.values.save()  # AttributeError: TestK1 has no attribute 'save'

I would assume that nested EmbeddedDocuments should have _instance pointing to the Document instance and _parent pointing to the parent doc (embedded or actual document)