Mongita is to MongoDB as SQLite is to SQL

This is a really awesome lib. I wanted to see if I am able to start a db, and use it with mongoengine. I'm not sure if it requires an actual connection.

While it is possible to use MongoEngine 0.22.1 with mongita-1.1.0 MongitaClientMemory:

import pymongo
import mongita
#  This works
pymongo.MongoClient = mongita.MongitaClientMemory
import mongoengine
mongoengine.connect(host='c:/temp/mongita')

It does not work with MongitaClientDisk:

import pymongo
import mongita
# This fails
pymongo.MongoClient = mongita.MongitaClientDisk
import mongoengine
mongoengine.connect(host='c:/temp/mongita')

The MongitaClientDisk constructor always fails.

Not clear if this is just a version mis-match with mongoengine or not. The error also affects the unit tests for mongita.

The problem seems to be in mongita_client.py in the MongitaClientDisk constructor where it invokes:

disk_engine.DiskEngine.create(host)

This is because according to: https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html the host parameter to the MongoClient() method is a list, not a string. Since the DiskEngine.create() factory method is expecting a string, it reports the error: "unhashable type: list"

The simple fix is to test in the MongoClientDisk constructor in mongita_client.py to see if the parameter is a list and if so pluck the first element. Since the default value for mongoengine is 'localhost' this is worth checking for and replacing with DEFAULT_STORAGE_DIR.

While we are at it, a check for the existence of the parent directory where the database is to be located is worthwhile. This leaves us with:

    def __init__(self, host=DEFAULT_STORAGE_DIR, **kwargs):
        host = host or DEFAULT_STORAGE_DIR
        if host == 'localhost':
            host = DEFAULT_STORAGE_DIR
 
         if os.path.exists(os.path.dirname(host)):
            raise NotADirectoryError(os.path.dirname(host))
 
        self.engine = disk_engine.DiskEngine.create(host)

I'd be happy to generate a pull request, but I'd like to know that the pre-existing unit tests work on some system before requesting a pull for a fix that may be out of date.

According to https://docs.mongodb.com/manual/reference/operator/query/in/#mongodb-query-op.-in

If the field holds an array, then the $in operator selects the documents whose field holds an array that contains at least one element that matches a value in the specified array (for example, <value1>, <value2>, and so on).

mongita raises an error in this case:

Traceback (most recent call last):
  File "test.py", line 57, in test_mongita
    result = list(col.find({"names: {"$in": ["asd", "qwe"]}}))
  File "Python38\lib\site-packages\mongita\cursor.py", line 56, in __iter__
    for el in self._gen():
  File "Python38\lib\site-packages\mongita\collection.py", line 870, in __find
    for doc_id in self.__find_ids(filter, sort, limit, metadata=metadata):
  File "Python38\lib\site-packages\mongita\collection.py", line 845, in __find_ids
    if doc and _doc_matches_slow_filters(doc, slow_filters):
  File "Python38\lib\site-packages\mongita\collection.py", line 193, in _doc_matches_slow_filters
    if _doc_matches_agg(doc_v, query_ops):
  File "Python38\lib\site-packages\mongita\collection.py", line 143, in _doc_matches_agg
    if doc_v not in query_val:
TypeError: unhashable type: 'list'

First off, I love this library. It's one of the main reasons I made the jump from NodeJS with NEDB to Python. However, I'm trying to implement pagination on my Flask site, and the cursor.skip method would make life a little easier for me. Thank you for taking the time for implementing this feature. I appreciate all you do.

Hi, Very nice work with the mongita project!

I have just started experimenting with it. I have found that if i write to a collection in one process and reads from it in another, the values when reading will not be updated without reinstance the MongitaClient. From behaviour it seems like the full db is loaded to memory? is there a function I can use to refresh?

I was testing this with MongitaClientDisk.

If I open two instances of the same db and collection in the same process the changes to the collection is reflected immediately.

Best regards.

@scottrogowski, this is a really nice project. The name is awesome too!

I have not had the chance yet to really git it a spin, however I think the benchmarks can be improved a bit. I believe SQLite performance comparision is can be improved if you compared insertion of a dict into JSON.

I think this is where most of the CPU cycles in the row insertion are consumed, which make SQLite look so bad ...

def _to_sqlite_row(doc):
    doc['_id'] = str(doc['_id'])
    return (doc['_id'], doc['name'], doc['dt'], doc['count'], doc['city'],
            doc['content'], doc['percent'],
            json.dumps(doc['dict'], default=json_util.default))

Thanks for publishing this nifty little project!

First of all, thank you for this library. I love it.

Would love to see $push implemented.

https://docs.mongodb.com/manual/reference/operator/update/push/

If I have a chance I'll make an attempt at it this week. What modules would I need to touch for this?

I am trying to install mongita on Python 3.9 on a Windows machine. I am getting the following error. Any ideas?:

$ pip install mongita Collecting mongita Using cached mongita-1.0.0.tar.gz (33 kB) ERROR: Command errored out with exit status 3221225477: command: 'D:\anaconda\envs\py39\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\len_w\AppData\Local\Temp\pip-install-ee8rj6s8\mongita\setup.py'"'"'; file='"'"'C:\Users\len_w\AppData\Local\Temp\pip-install-ee8rj6s8\mongita\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn' cwd: C:\Users\len_w\AppData\Local\Temp\pip-install-ee8rj6s8\mongita
Complete output (11 lines): running egg_info creating C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info writing C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\PKG-INFO writing dependency_links to C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\dependency_links.txt writing requirements to C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\requires.txt writing top-level names to C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\top_level.txt writing manifest file 'C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\SOURCES.txt' reading manifest file 'C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'LICENSE,' writing manifest file 'C:\Users\len_w\AppData\Local\Temp\pip-pip-egg-info-m1bl3ksn\mongita.egg-info\SOURCES.txt' ---------------------------------------- ERROR: Command errored out with exit status 3221225477: python setup.py egg_info Check the logs for full command output.

Segmentation fault

I see that it is written in the docs:

It is not process-safe

But the particular case is not clear. What if to create one client to work with one collection?. Example with MongoDB. Is it a safe approach?

Hi!

I am not sure if it's against the design goals of the library, but it would be very useful to be able to provide custom Python callback functions to the _doc_matches_slow_filters code path of the find() method. That means a custom function which takes a single document and returns True or False if it should be included in the output. As fast as I understand this should be easy to implement as that's what the non-indexed code path is basically doing anyway.

Using that it would also be easy to work around operators which are not implemented yet.

I'm experiencing weird issues,

the api will throw a key error exception when accessing a dict. And after that, the db will be erased.

It seems to be a thread safety issue? read/write can't happen at the same time.

When trying to create more than one index via create_indexes, a MongitaNotImplementedError occurs. I understand that implementing index intersection is a difficult task, but I really hope that it will be in Mongita someday.

mongita.errors.InvalidName: Collection cannot be named 'Nerve_Tibial.v8.egenes_ann_query_res.vcf'.

It seems to me that the prohibition of presence of non-letter symbols in the collection name is superfluous. By the way, MongoDB does not have this restriction.

It is not clear whether collections are compressed. If so, what algorithm is used by default? Can the user specify a preferred algorithm and compression level?

An example from the MongoDB world: create_collection(name, storageEngine={'wiredTiger': {'configString': 'block_compressor=zstd'}})

I've dumped my Mongo Database in local folder now working with it mongitaDB.

PyMongo has such functionality where I can retrieve documents from the Database with specific fields. For an example:

mongo_client.db.col.find({},{"_id":1})

This line returns the cursor from where I will get the "_id" field. Then, I've tried something similar with mongitaDB.

mongita_client.db.col.find({},{"_id":1})

It raises following error:

mongita.errors.MongitaError: Unsupported sort parameter format. See the docs.

That's an awesome tools, and really very close to MongoDB. Is there any feature equivalent to mongodump and mongorestore (or mongoexport and mongoimport)? If not, it would be a useful addition to the module. Thanks.