…tion.

This implements: https://github.com/zopefoundation/ZODB/issues/50

Rather than watching invalidations, simply use 1 + the storages lastTransaction, which is equivalent to but much simpler than waiting for the first invalidation after a transaction starts.

More importantly, it means we can always use loadBefore and get away from load. We no longer have to worry about ordering of invalidations and load() results.

Much thanks to NEO for pointing the way toward this simplification!

Implementing this initially caused a deadlock, because DB.open() called Connection.open() while holding a database lock and Connection.open() now calls IStotage.lastTransaction(), which acquires a storage lock. (It's not clear that lastTransaction() really needs a storage lock.) Meanwhile, IStotage.tpc_finish() calls a DB function that requires the DB lock while holding the storage lock. Fixing this required moving the call to Connection.open() outside the region where the DB lock was held.

To debug the problem above, I greatly improved lock-debugging support. Now all of the ZODB code imports Lock, RLock and Condition from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value, then these are wrapped in such a way that debugging information is printed as they are used. This made spotting the nature of the deadlock easier.

Of course, a change this basic broke lots of tests. Most of the breakage arises from the fact that connections now call lastTransaction on storages at transaction boundaries. Lots of tests didn't clean up databases and connections properly. I fixed many tests, but ultimately gave up and added some extra cleanup code that violated transaction-manager underware (and the underware's privates) to clear transaction synchonizers in test setup and tear-down. I plan to add a transaction manager API for this and to use it in a subsequent PR.

This tests makes database and connection hygiene a bit more important, especially for tests, because a connection will continue to interact with storages if it isn't properly closed, which can lead to errors if the storage is closed. I chose not to swallow these errors in Connection, choosing rather to clean up tests.

The thread debugging and test changes make this PR larger than I would have liked. Apologies in advance to the reviewers.

http://winbot.zope.org/builders/ZODB_dev%20py_270_win32/builds/2116/steps/test/logs/stdio

(I plan to update builders to the currently supported platforms)

(also, @jimfulton you need failure emails from winbot?)

Add test that exercises open vs invalidation race condition that, if happen, leads to data corruption. We are seeing such race happening on storage level in ZEO (https://github.com/zopefoundation/ZEO/issues/166), and previously we've seen it also to happen on Connection level (https://github.com/zopefoundation/ZODB/issues/290). By adding this test to be exercised wrt all storages we make sure that all storages stay free from this race.

And it payed out. Besides catching original problems from https://github.com/zopefoundation/ZODB/issues/290 and https://github.com/zopefoundation/ZEO/issues/166 , this test also discovered a concurrency bug in MVCCMappingStorage:

Failure in test check_race_open_vs_invalidate (ZODB.tests.testMVCCMappingStorage.MVCCMappingStorageTests)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/BasicStorage.py", line 492, in check_race_open_vs_invalidate
    self.fail(failure[0])
  File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
    raise self.failureException(msg)
AssertionError: T1: obj1.value (24)  !=  obj2.value (23)

The problem with MVCCMappingStorage was that instance.poll_invalidations was correctly taking main_lock with intention to make sure main data is not mutated during analysis, but instance.tpc_finish and instance.tpc_abort did not taken main lock, which was leading to committed data to be propagating into main storage in non-atomic way.

This bug was also observable if both obj1 and obj2 in the added test were always loaded from the storage (added obj2._p_invalidate after obj1._p_invalidate).

-> Fix MVCCMappingStorage by correctly locking main MVCCMappingStorage instance when processing transaction completion.

/cc @d-maurer, @jamadden, @jmuchemb

In the early days, before MVCC was introduced, ZODB used to raise ReadConflictError on access to object that was simultaneously changed by another client in concurrent transaction. However, as doc/articles/ZODB-overview.rst says

Since Zope 2.8 ZODB has implemented **Multi Version Concurrency Control**.
This means no more ReadConflictErrors, each transaction is guaranteed to be
able to load any object as it was when the transaction begun.

So today the only way to get a ReadConflictError should be at commit time for an object that was requested to stay unchanged via checkCurrentSerialInTransaction.

However MVCCAdapterInstance.load(), instead of reporting "no data", was still raising ReadConflictError for a deleted or not-yet-created object. If an object is deleted and later requested to be loaded, it should be "key not found in database", i.e. POSKeyError, not ReadConflictError. Fix it.

Adjust docstring of ReadConflictError accordingly to explicitly describe that this error can only happen at commit time for objects requested to be current.

There were also leftover code, comment and test bits in Connection, interfaces, transact, testmvcc and testZODB, that are corrected/removed correspondingly. testZODB actually had ReadConflictTests that was completely deactivated: commit b0f992fd ("Removed the mvcc option..."; 2007) moved read-conflict-on-access related tests out of ZODBTests, but did not activated moved parts at all, because as that commit says when MVCC is always on unconditionally, there is no on-access conflicts:

Removed the mvcc option.  Everybody wants mvcc and removing us lets us
simplify the code a little. (We'll be able to simplify more when we
stop supporting versions.)

Today, if I try to manually activate that ReadConflictTests via

@@ -637,6 +637,7 @@ def __init__(self, poisonedjar):
 def test_suite():
     return unittest.TestSuite((
         unittest.makeSuite(ZODBTests, 'check'),
+        unittest.makeSuite(ReadConflictTests, 'check'),
         ))

 if __name__ == "__main__":

it fails in dumb way showing that this tests were unmaintained for ages:

Error in test checkReadConflict (ZODB.tests.testZODB.ReadConflictTests)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 320, in run
    self.setUp()
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/testZODB.py", line 451, in setUp
    ZODB.tests.utils.TestCase.setUp(self)
AttributeError: 'module' object has no attribute 'utils'

Since today ZODB always uses MVCC and there is no way to get ReadConflictError on access, those tests should be also gone together with old pre-MVCC way of handling concurrency.

/cc @jimfulton

Hello up there,

For wendelin.core v2 I need a way to know at which particular database state application-level ZODB connection is viewing the database. I was looking for that in Connection.open / MVCCAdapter code and noticed concurrency bug that leads to wrong data returned by ZODB to application:

---- 8< ---- (zopenrace.py)

#!/usr/bin/env python
"""Program zopenrace.py demonstrates concurrency bug in ZODB Connection.open()
that leads to stale live cache and wrong data provided by database to users.

The bug is that when a connection is opened, it syncs to storage and processes
invalidations received from the storage in two _separate_ steps, potentially
leading to situation where invalidations for transactions _past_ opened
connection's view of the database are included into opened connection's cache
invalidation. This leads to stale connection cache and old data provided by
ZODB.Connection when it is reopened next time.

That in turn can lead to loose of Consistency of the database if mix of current
and old data is used to process a transaction. A classic example would be bank
accounts A, B and C with A<-B and A<-C transfer transactions. If transaction
that handles A<-C sees stale data for A when starting its processing, it
results in A loosing what it should have received from B.

Below is timing diagram on how the bug happens on ZODB5:

    Client1 or Thread1                                  Client2 or Thread2

    # T1 begins transaction and opens zodb connection
    newTransaction():
        # implementation in Connection.py[1]
        ._storage.sync()
        invalidated = ._storage.poll_invalidations():
            # implementation in MVCCAdapterInstance [2]

            # T1 settles on as of which particular database state it will be
            # viewing the database.
            ._storage._start = ._storage._storage.lastTrasaction() + 1:
                s = ._storage._storage
                s._lock.acquire()
                head = s._ltid
                s._lock.release()
                return head
                                                        # T2 commits here.
                                                        # Time goes by and storage server sends
                                                        # corresponding invalidation message to T1,
                                                        # which T1 queues in its _storage._invalidations

            # T1 retrieves queued invalidations which _includes_
            # invalidation for transaction that T2 just has committed past @head.
            ._storage._lock.acquire()
                r = _storage._invalidations
            ._storage._lock.release()
            return r

        # T1 processes invalidations for [... head] _and_ invalidations for past-@head transaction.
        # T1 thus will _not_ process invalidations for that next transaction when
        # opening zconn _next_ time. The next opened zconn will thus see _stale_ data.
        ._cache.invalidate(invalidated)


The program simulates two clients: one (T2) constantly modifies two integer
objects preserving invariant that their values stay equal. The other client
(T1) constantly opens the database and verifies the invariant. T1 forces access
to one of the object to always go through loading from the database, and this
way if live cache becomes stale the bug is observed as invariant breakage.

Here is example failure:

    $ taskset -c 1,2 ./zopenrace.py
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "./zopenrace.py", line 136, in T1
        t1()
      File "./zopenrace.py", line 130, in t1
        raise AssertionError("t1: obj1.i (%d)  !=  obj2.i (%d)" % (i1, i2))
    AssertionError: t1: obj1.i (147)  !=  obj2.i (146)

    Traceback (most recent call last):
      File "./zopenrace.py", line 179, in <module>
        main()
      File "./zopenrace.py", line 174, in main
        raise AssertionError('FAIL')
    AssertionError: FAIL

NOTE ZODB4 and ZODB3 do not have this particular open vs invalidation race.

[1] https://github.com/zopefoundation/ZODB/blob/5.5.1-29-g0b3db5aee/src/ZODB/Connection.py#L734-L742
[2] https://github.com/zopefoundation/ZODB/blob/5.5.1-29-g0b3db5aee/src/ZODB/mvccadapter.py#L130-L139
"""

from __future__ import print_function
from ZODB import DB
from ZODB.MappingStorage import MappingStorage
import transaction
from persistent import Persistent

# don't depend on pygolang
# ( but it is more easy and structured with sync.WorkGroup
#   https://pypi.org/project/pygolang/#concurrency )
#from golang import sync, context
import threading
def go(f, *argv, **kw):
    t = threading.Thread(target=f, args=argv, kwargs=kw)
    t.start()
    return t


# PInt is persistent integer.
class PInt(Persistent):
    def __init__(self, i):
        self.i = i


def main():
    zstor = MappingStorage()
    db = DB(zstor)


    # init initializes the database with two integer objects - obj1/obj2 that are set to 0.
    def init():
        transaction.begin()
        zconn = db.open()

        root = zconn.root()
        root['obj1'] = PInt(0)
        root['obj2'] = PInt(0)

        transaction.commit()
        zconn.close()


    okv = [False, False]

    # T1 accesses obj1/obj2 in a loop and verifies that obj1.i == obj2.i
    #
    # access to obj1 is organized to always trigger loading from zstor.
    # access to obj2 goes through zconn cache and so verifies whether the cache is not stale.
    def T1(N):
        def t1():
            transaction.begin()
            zconn = db.open()

            root = zconn.root()
            obj1 = root['obj1']
            obj2 = root['obj2']

            # obj1 - reload it from zstor
            # obj2 - get it from zconn cache
            obj1._p_invalidate()

            # both objects must have the same values
            i1 = obj1.i
            i2 = obj2.i
            if i1 != i2:
                raise AssertionError("T1: obj1.i (%d)  !=  obj2.i (%d)" % (i1, i2))

            transaction.abort() # we did not changed anything; also fails with commit
            zconn.close()

        for i in range(N):
            #print('T1.%d' % i)
            t1()
        okv[0] = True


    # T2 changes obj1/obj2 in a loop by doing `objX.i += 1`.
    #
    # Since both objects start from 0, the invariant that `obj1.i == obj2.i` is always preserved.
    def T2(N):
        def t2():
            transaction.begin()
            zconn = db.open()

            root = zconn.root()
            obj1 = root['obj1']
            obj2 = root['obj2']
            obj1.i += 1
            obj2.i += 1
            assert obj1.i == obj2.i

            transaction.commit()
            zconn.close()

        for i in range(N):
            #print('T2.%d' % i)
            t2()
        okv[1] = True


    # run T1 and T2 concurrently. As of 20191210, due to race condition in
    # Connection.open, it triggers the bug where T1 sees stale obj2 with obj1.i != obj2.i
    init()

    N = 1000
    t1 = go(T1, N)
    t2 = go(T2, N)
    t1.join()
    t2.join()

    if not all(okv):
        raise AssertionError('FAIL')
    print('OK')


if __name__ == '__main__':
    main()

Thanks beforehand, Kirill

/cc @jimfulton

P.S. It would be nice to provide ZODB.Connection.at() explicitly similarly to https://godoc.org/lab.nexedi.com/kirr/neo/go/zodb#Connection

ZODB now translates transaction meta data, user and description from text to bytes before passing them to storages, and converts them back to text when retrieving them from storages in the history, undoLog and undoInfo methods.

The IDatabase interface was updated to reflect that history, undoLog and undoInfo are available on database objects. (They were always available, but not documented in the interface.)

Previously protocol 1 was used. The higher protocol is more efficient for new-style classes (all persistent objects are new-style), according to the docs, at the cost of being very slightly less space efficient for old-style classes.

In tests of a persistent object with two trivial numeric attributes, the higher protocol was 12 bytes smaller, and serialized and deserialized 1us faster. Introducing a reference to another new-style class (with a small dict and list of strings for attributes) for a more realistic test made the higher protocol twice as fast to serialize (20.5 vs 10.3us), almost half the size (215 vs 142 bytes), and it deserialized 30% faster (6.5 vs 4.6us).

On Python 2, this will now allow open file objects to be pickled (loading the object will result in a closed file); previously this would result in a TypeError (as does under Python 3). We had tests that you couldn't do that with a BlobFile so I had to update it to still make that true.

I wouldn't recommend serializing arbitrary open files under Python 2 (for one thing, they can't trivially be deserialized in Python 3), but I didn't take any steps to prevent it either. Since this hasn't been possible, there shouldn't be code in the wild that is trying to do it---and it wouldn't be forward compatible with Python 3 either.

UPDATE: see this comment for an up to date description: https://github.com/zopefoundation/ZODB/issues/208#issuecomment-426468941

When running tests (in plone.restapi) we sometimes get the following KeyError. It is related to timing issues - inserting a time.sleep(2) in the test setup causes the problem to occur more frequently.

It can also be reproduced outside of a testing environment, when zope is shutting down in the middle of ongoing requests. See a comment below for a method of easy reproduction.

ZODB 5.4.0 It does not happen with ZODB 5.3.0

It is related to this commit: https://github.com/zopefoundation/ZODB/commit/1b9475d413705e510ff2f19ee6449688ffb04d46

In line 948 we see: c.close(False)

This is part of the traceback: File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 948, in _release_resources c.close(False) In 5.3.0 the corresponding line just sets the transaction manager to None: c.transaction_manager = None

  Set up plone.restapi.testing.PloneRestApiDXLayer:Functional in 0.000 seconds.
  Running:
    1/77 (1.3%) test_comments_add_root (plone.restapi.tests.test_documentation.TestCommenting)/work/buildout_cache/eggs/ZServer-4.0b1-py2.7.egg/ZServer/PubCore/ZServerPublisher.py:22: DeprecationWarning: publish_module is deprecated. Please import from ZServer.ZPublisher.Publish.
  from ZPublisher.Publish import publish_module
    8/77 (10.4%) test_documentation_expansion (plone.restapi.tests.test_documentation.TestCommenting)/work/buildout_cache/eggs/zope.deprecation-4.3.0-py2.7.egg/zope/deprecation/deprecation.py:88: DeprecationWarning: isDefaultPage is deprecated. Import from Products.CMFPlone.defaultpage instead
  name)
    22/77 (28.6%) test_documentation_file (plone.restapi.tests.test_documentation.TestDocumentation)/work/buildout_cache/eggs/Products.PortalTransforms-3.1.3-py2.7.egg/Products/PortalTransforms/libtransforms/commandtransform.py:105: DeprecationWarning: os.popen4 is deprecated.  Use the subprocess module.
  cin, couterr = os.popen4(command, 'b')
    32/77 (41.6%) test_documentation_jwt_login (plone.restapi.tests.test_documentation.TestDocumentation) (1.199 s)Traceback (most recent call last):
  File "./bin/test", line 442, in <module>
    '--test-path', '/work/buildout_cache/eggs/repoze.xmliter-0.6-py2.7.egg',
  File "/work/buildout_cache/eggs/collective.xmltestreport-1.3.4-py2.7.egg/collective/xmltestreport/runner.py", line 60, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/work/buildout_cache/eggs/collective.xmltestreport-1.3.4-py2.7.egg/collective/xmltestreport/runner.py", line 73, in run_internal
    runner.run()
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 189, in run
    self.run_tests()
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 291, in run_tests
    self.skipped, self.import_errors)
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 460, in run_layer
    import_errors)
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 383, in run_tests
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 370, in run
    result.stopTest(self)
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 897, in stopTest
    self.testTearDown()
  File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 821, in testTearDown
    layer.testTearDown()
  File "/work/playground/plone/plone52devel/src/plone.app.testing/plone/app/testing/layers.py", line 288, in testTearDown
    super(PloneTestLifecycle, self).testTearDown()
  File "/work/playground/plone/plone52devel/src/plone.testing/src/plone/testing/z2.py", line 945, in testTearDown
    self['zodbDB'].close()
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 651, in close
    @self._connectionMap
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 518, in _connectionMap
    self.pool.map(f)
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 213, in map
    self.all.map(f)
  File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/weakset.py", line 62, in map
    f(elt)
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 659, in _
    conn._release_resources()
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 948, in _release_resources
    c.close(False)
  File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 304, in close
    self.transaction_manager.unregisterSynch(self)
  File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/_manager.py", line 106, in unregisterSynch
    self._synchs.remove(synch)
  File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/weakset.py", line 51, in remove
    del self.data[id(obj)]
  File "/usr/lib/python2.7/weakref.py", line 86, in __delitem__
    del self.data[key]
KeyError: 139902031255440

Because of ZODB wrappers like ZEO, I can't touch FileStorage & DemoStorage without breaking compatibility. So what I suggest is to first develop the 2 API in the 4.x branch, with deprecation warnings. Once done, we merge into master branch and (I prefer) force to use the new API.

Currently when client has IStorageTransactionMetaData instance (e.g. on storage iteration) it is possible to read transaction's .user and .description in raw form, but .extension is always returned unpickled.

This creates several problems:

• tools like zodb dump [1] cannot dump data exactly as stored on a storage. This makes database potentially not bit-to-bit identical to its original after restoring from such dump.

• zodb dump output could be changing from run to run on the same database. This comes from the fact that e.g. python dictionaries are unordered and so when pickling a dict back to bytes the result could be not the same as original.

  ( this problem can be worked-around partly to work reliably for e.g. dict with str keys - by always emitting items in key sorted order, but it is hard to make it work reliably for arbitrary types )

Both issues make it hard to verify integrity of database at the lowest possible level after restoration, and make it hard to verify bit-to-bit compatibility with non-python ZODB implementations.

To fix we provide a way to retrieve raw extension from transaction metadata. This is done in backward-compatible way by introducing new interface

IStorageTransactionInformationRaw

which can be optionally provided by an IStorageTransactionInformation to indicate that transaction metadata can be also read in raw from.

Then BaseStorage.TransactionRecord ctor is extended with optional raw_extension argument which if != None is remembered and makes IStorageTransactionInformationRaw to be also provided by constructed TransactionRecord instance.

We also adapt FileStorage & friends to provide/use new functionality.

Similarly to e.g. undo, storages should be indicating they will return iterated transactions provided with raw metadata via implementing

supportsTransactionInformationRaw() -> True

[1] https://lab.nexedi.com/nexedi/zodbtools

Add test for Bug2 described in https://github.com/zopefoundation/ZEO/issues/209 + modernize race testing infrastructure. We add this test here, so that it can be run automatically for all storages - e.g. both ZEO and NEO.

ZEO currently can corrupt data if client disconnects simultaneously to another client performing tpc_finish. It fails e.g. as follows:

(z-dev) kirr@deca:~/src/wendelin/z/ZEO5$ ZEO_MTACCEPTOR=1 zope-testrunner -fvvvx --test-path=src -t check_race_external_invalidate_vs_disconnect
/home/kirr/src/wendelin/venv/z-dev/bin/zope-testrunner traceio=True
/home/kirr/src/wendelin/z/ZEO5/src/ZEO/StorageServer.py:51: DeprecationWarning: The mtacceptor module is deprecated and will be removed in ZEO version 6.
  'in ZEO version 6.', DeprecationWarning)
Running tests at level 1
Running .BlobAdaptedFileStorageTests tests:
  Set up .BlobAdaptedFileStorageTests in 0.000 seconds.
  Running:
 check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests) (1.889 s)

Failure in test check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 357, in check_race_external_invalidate_vs_disconnect
    T2ObjectsInc2Phase())
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/util.py", line 400, in _
    return f(*argv, **kw)
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 446, in _check_race_xxx_vs_external_disconnect
    self.fail('\n\n'.join([_ for _ in failure if _]))
  File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
    raise self.failureException(msg)
AssertionError: T15: obj1 (6) - obj2(4) != phase (1)
obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
zstor._cache.clear()
zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: 0x03ea4cc506104155
zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: 0x03ea4cc506104155

T51: obj1 (6) - obj2(4) != phase (1)
obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc503413799  next_serial: None
zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
zstor._cache.clear()
zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None

Please see individual patches for details.

I am familiar with Python and willing to contribute for ZODB and its related python project.

But feel difficult to start.

Is there any resource or documentation which can help me understand the mechanism inside ZODB in details?

If so, it would be awesome.

I know that there is little development but it might be useful for someone if using ZODB in 2022.

when trying to store changes to persistent objects inside a ZODB.DB.transaction with block. they are not stored, and no error is raised. while doing the same between transaction.begin() and transaction.commit() calls does work.

It seems to me that is the only way to currently use a with block is to change objects directly through conn.root(), that means all persistent objects must know full path from root to themselves, which is impractical.

there is also another weird behavior. where after storing an object for the first time, and retrieving it returns the same object, while the 2nd call and up will return a different object. this trips tests trying to check if something is stored successfully, as it only happens once.

the following code tries to store attributes in a 2 level persistent hierarchy (represents actual code)

import ZODB
import ZODB.FileStorage
from persistent.mapping import PersistentMapping
import transaction

store=ZODB.FileStorage.FileStorage("temp1.db")
db=ZODB.DB(store)

def get_init(name, obj):
    with db.transaction(f"creating root[{name}]") as conn:
        try:
            return conn.root()[name]
        except KeyError:
            conn.root()[name] = obj()
            return conn.root()[name]

class A:
    def __init__(self):
        self.cfg = PersistentMapping()

    def __setitem__(self, key, value) -> None:
        transaction.begin()
        self.cfg[key+", inside block"] = value
        transaction.commit()
        with db.transaction():
            self.cfg[key+", inside with"] = value #does not work
        #these should be equivalent, no?

    def __iter__(self):
        return iter(self.cfg)

class Manager:
    def __init__(self):
        self.a1=get_init("testing", PersistentMapping) # set up the db, should only happen once

    def __setitem__(self, name, obj) -> None:
        """Registers in persistent storage"""
        with db.transaction(f"Adding testing:{name}") as conn:
            if name in conn.root()["testing"]:
                print(f"testing with same name {name} already exists in storage")
                return
            conn.root()["testing"][name] = obj

    def __getitem__(self, name: str):
        return db.open().root()["testing"][name]

dm=Manager()

initial=A() #only relevant for forst run
dm['a']=initial #only relevant for forst run

fromdb1= dm['a']
fromdb2= dm['a']

with db.transaction() as conn:
    fromdb1.cfg['updated from outer txn, directly'] = 1 #doed not work
    conn.root()['testing']['a'].cfg['updated from outer txn,through conn'] = 1
    #this should be equivalent but only the second one works

# transaction.begin()
# transaction.commit()
initial['new txn updated on initial'] = 1
fromdb1['new txn updated on retrieved 1']= 1
fromdb2['new txn updated on retrieved 2']= 1
# db.open().root()['testing']['a']['new txn updated on new db']=1

print(f"initial obj - {initial.cfg}")
print(f"from db obj 1- {fromdb1.cfg}")
print(f"from db obj 2- {fromdb2.cfg}")
print(f"\nnew from db obj- {dm['a'].cfg}")

print(f"\nis the initial obj and the first obj from db the same: {initial is fromdb1}")
print(f"is the initial obj and the second obj from db the same: {initial is fromdb2}")

The expected result is for all writing methods to work similarly, or the docs to reflect those limitations.

Thanks!!

What version of Python and Zope/Addons I am using:

Python 3.9.2, ZODB3, Win1064bit

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

For modern python library which involved path location, it would be nice to accept Union[str, os.PathLike[str]], instead of just str.

I have a bit of a dual question: 1st. Is there a better place I should be asking these non-issue clarification questions that the docs alone were unable to answer for me? 2nd: What are a some best practices for using ZODB in a ZEO configuration with a handful of users that will be operating(r/w) mostly in their own BTree persistences? specifically I am trying to employee best practices when trying to read from the database such that I don't overwrite work by attempting to refresh the current view of the 'live' data.

Any assistance would be much appreciated as I am very new to ZODB and have chosen it as we need a near embedded database instance that can be easily traversed to find groupings of applicable data where the structure can be vastly different between the entries beyond core 'queryable' tags that are used in the keys of the BTree structures mostly.

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zodb-5.7.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zodb-5.7.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
==================================================================== test session starts =====================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/ZODB-5.7.0
collected 96 items

src/ZODB/scripts/tests/test_doc.py .                                                                                                                   [  1%]
src/ZODB/scripts/tests/test_fsdump_fsstats.py ..                                                                                                       [  3%]
src/ZODB/scripts/tests/test_fstest.py ..                                                                                                               [  5%]
src/ZODB/scripts/tests/test_repozo.py ........................................................................                                         [ 80%]
src/ZODB/tests/testConnectionSavepoint.txt .                                                                                                           [ 81%]
src/ZODB/tests/test_TransactionMetaData.py .......                                                                                                     [ 88%]
src/ZODB/tests/test_cache.py .                                                                                                                         [ 89%]
src/ZODB/tests/test_doctest_files.py .                                                                                                                 [ 90%]
src/ZODB/tests/test_fsdump.py .                                                                                                                        [ 91%]
src/ZODB/tests/test_mvccadapter.py ...                                                                                                                 [ 94%]
src/ZODB/tests/test_prefetch.py ....                                                                                                                   [ 98%]
src/ZODB/tests/test_storage.py .                                                                                                                       [100%]

====================================================================== warnings summary ======================================================================
src/ZODB/scripts/tests/test_fsdump_fsstats.py::FsdumpFsstatsTests::test_fsstats
  /usr/lib64/python3.8/runpy.py:127: RuntimeWarning: 'ZODB.scripts.fsstats' found in sys.modules after import of package 'ZODB.scripts', but prior to execution of 'ZODB.scripts.fsstats'; this may result in unpredictable behaviour
    warn(RuntimeWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================== 96 passed, 1 warning in 1.19s ================================================================

Also looks like ZODB/test/ content is installed. I think that this warning may be related to fact that ZODB/tests is installed.

Is there a best practice or some first hand experience for which file sizes it makes sense to use Blob storage?

I'm using the blob storage to store files and images, but I wonder if it's maybe better (performance-wise?) to keep small files in the ZODB.