johnny cache django caching framework

In order to achieve johnny-cache 1.5 goals, I made the following:

• I added Travis CI to smoothly move from 1.4 to 1.5, but discovered that johnny-cache was not able to run with Django 1.1 (and Python 2.4 & 2.5 are not available on Travis CI)
• so I wanted to speed things a bit. I implemented full Python 3 support and decided to drop Django<1.3 & Python<2.6 support (as specified in the 1.5 goals).
• in tests, I changed old failUnless etc to assertTrue, assertFalse, assertEqual and assertNotEqual, all supported from Python 2.6
• I started working on Django 1.6 support, and only pushed a backward-compatible version of #36. I still have lots of pending changes on my computer about this, but the test suite still fails so I'll stick to this bunch of changes for today.

Hi,

I'm relatively new to Johnny Cache and memcached so maybe it's a simple problem, but I have a report app that is giving me problems. For 2011, for example, I have 14 teams on the landing page. Clicking on most work fine, but one always has this error:

MemcachedError at /nai/reports/annual-reports/2011/gsfc/
error 37 from memcached_set: SYSTEM ERROR(Resource temporarily unavailable), host: 127.0.0.1:11211 -> libmemcached/io.cc:358

I'm wondering if it's something to do with my version of memcached on my server.

I'm running Ubuntu. Can you tell me what the best version of memcached to use?

Also, I'm using pylibmc==1.2.3

Thanks.

The docs explain that this is a tricky part: http://pythonhosted.org/johnny-cache/queryset_cache.html#transactions

But I am struggling to understand if autocommit causes any issues for Johnny? Or does it make Johnny's life easier?

We're using johnny cache in a multi-web server environment and we're running into some strange issues in which query caches aren't being invalidated. Unfortunately, we haven't been able to figure out how to reproduce it though it does happen consistently.

We're set up at EC2 and we have 2 web servers set up using nginx + uwsgi. We're only using one db (pooled using pgbouncer on each webserver) and have johnny cache configured to use the memcached backend with ElastiCache.

Most of our code uses Django's default autocommit mode but we do have some methods where we use the @transaction.commit_on_success decorator.

We also have some cron jobs (set up using django-cronjobs) which we run and which need to invalidate caches. In the methods for those cron jobs, we call johhn.cache.enable() to set up johnny cache before we do any other work.

The scenario below works sometimes but then stops working after some time. Once it stops working, we can manually invalidate the object's cache and it starts working again or we can cycle uwsgi on the webservers which also gets things working again.

1. Create a new MyObject (via a form that is saved)
2. In the cronjob, get a list of all MyObjects

Step 2 is what stops working in the sense that new MyObjects don't show up in the list.

We've spent hours trying to track this down so any help you can provide would be great!

Hi,

Using Django 1.4 and in admin I have a model that I can't see the detail view.

MemcachedError at /admin/nims/article/1141/
error 37 from memcached_set: SYSTEM ERROR(Resource temporarily unavailable), host: 127.0.0.1:11211 -> libmemcached/io.cc:358
Request Method: GET
Request URL:    http://127.0.0.1:8007/admin/nims/article/1141/
Django Version: 1.4
Exception Type: MemcachedError
Exception Value:    
error 37 from memcached_set: SYSTEM ERROR(Resource temporarily unavailable), host: 127.0.0.1:11211 -> libmemcached/io.cc:358
Exception Location: /home/django/.virtualenvs/astrobiology-alpha/local/lib/python2.7/site-packages/django/core/cache/backends/memcached.py in set, line 64
Python Executable:  /home/django/.virtualenvs/astrobiology-alpha/bin/python
Python Version: 2.7.1

Most of my admin works, my guess is this article model is quite a big bigger than the others..

Please let me know how to debug this and if I can send you anything else that would be useful.

Thanks, Shige

AttributeError: 'module' object has no attribute 'empty_iter'

Solution:

import sys
try:
    if sys.argv[1] != 'syncdb':
        raise Exception
except:
    from johnny.cache import enable
    enable()

Fixed bug that caused KeyError in johnny.transaction.TransactionManager._commit_all_savepoints - issues #26 and #30 on Github (#69 on Bitbucket)

This patch is fixing cause of the problem, not hiding it as pull requests #26 and #30.

To reproduce cause of the problem you should:

1. Enable QueryCacheMiddleware
2. Work in unmanaged transaction state (it means autocommit mode, i.e. without django.middleware.transaction.TransactionMiddleware and not within transaction.enter_transaction_management() block)
3. Create a savepoint and then try to save some object (it is done by django.db.models.query.QuerySet#get_or_create or by django.contrib.sessions.backends.db.SessionStore#save)

The essence of the problem is the fact that Django ORM widely uses a django/db/transaction.py#commit_unless_managed (which is unpatched by Johny Cache) function instead of django/db/transaction.py#commit (which is patched by Johny Cache). So, when we working in unmanaged transaction state Johny Cache will still register savepoints upon creation but will be unable to handle commits and rollbacks that wast performed by 'unless_managed' functions. Since Django is using dumb algorithm for savepoint names generation (they reuse same names within thread once they were commited) it will lead to a situation when Johny Cache will register several savepoints with same name within one transaction and fail when it will try to commit or rollback them.

Proposed solution adds patching to django/db/transaction.py#commit_unless_managed and django/db/transaction.py#rollback_unless_managed functions in the same way as it was done for django/db/transaction.py#commit and django/db/transaction.py#rollback functions with one difference - additional check whether current transaction is unmanaged.

I use some manage.py command to parse and upload some data to my database. johnny-cache cached my data once and after updates my data cache doesn't update. In database I have actual data, but my site show old data from cache.

I have to restart my memcached server to solve that problem after each data update. Anyone has similar problem?

@jmoiron it seems that you've published your SECRET_KEY in settings.py.

Here are the details:

https://www.quantifiedcode.com/app#/project/gh:jmoiron:johnny-cache?groups=code_patterns%3A1Wit6MHp%3Af0

johnny.cache.get_tables_for_query_pre_16: use a set as the base tables collection before entering recursion, thus avoiding a MemoryError should the WHERE descendant list go too deep.

This fully fixes #45 by preventing the top-most table list from growing more than it has to by being a set.

We ran into this problem because we had a bug that built a bad query. The query was valid, but the app broke when johnny-cache was in use.

I see the johnny.utils.celery_enable_all function, but it is not clear where it should be called from.

I see this same question posted on the (old?) bitbucket repo too in issue 47 but it was never answered.

Any clarification would be much appreciated.

Bumps django from 1.4 to 2.2.24.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I get the following error when trying to deserialize django data and only with cache enabled. (Johnny cache 1.4, Django 1.4.18)

Traceback (most recent call last):
  File "/opt/tangram/tangramdt/tangram_base/src/tangram/apps/api/deploy.py", line 285, in import_process_def
    transaction.commit()
  File "/opt/tangram/tangramdt/buildout/eggs/johnny_cache-1.4-py2.7.egg/johnny/transaction.py", line 145, in newfun
    self._flush(commit=commit, using=using)
  File "/opt/tangram/tangramdt/buildout/eggs/johnny_cache-1.4-py2.7.egg/johnny/transaction.py", line 133, in _flush
    self.cache_backend.set(key, value, self.timeout)
  File "/opt/tangram/tangramdt/buildout/eggs/Django-1.4.18-py2.7.egg/django/core/cache/backends/memcached.py", line 64, in set
    self._cache.set(key, value, self._get_memcache_timeout(timeout))
  File "/opt/tangram/tangramdt/buildout/eggs/python_memcached-1.48-py2.7.egg/memcache.py", line 565, in set
    return self._set("set", key, val, time, min_compress_len)
  File "/opt/tangram/tangramdt/buildout/eggs/python_memcached-1.48-py2.7.egg/memcache.py", line 802, in _set
    return _unsafe_set()
  File "/opt/tangram/tangramdt/buildout/eggs/python_memcached-1.48-py2.7.egg/memcache.py", line 780, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "/opt/tangram/tangramdt/buildout/eggs/python_memcached-1.48-py2.7.egg/memcache.py", line 751, in _val_to_store_info
    pickler.dump(val)
DatabaseError: ORA-22922: nonexistent LOB value

As a workaround I surround it with a cache.disable() / cache.enable()

Django 1.8 no longer has the class SqlDateCompiler, so we might have to modify the code to accommodate that.

https://github.com/jmoiron/johnny-cache/blob/master/johnny/cache.py#L412