A pickled object field for Django

I've also dropped tox testing for unsupported django and python versions, while adding in python 3.6 testing. I'm not terribly concerned about the supported versions, but figured I'd do some cleanup while I was there. I can remove that specific commit if you'd prefer to leave the versions alone.

Since release v0.3.2, tests for my own project with Django 1.8.4 on Python 2.7 and 3.4 fail. Tests with 1.7.10 seem unaffected.

I will have to look at the code, but from my test log it looks like all results return the integer 1, instead of the actual field value.

Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0–3.7.

Not sure it dropping PY2 is on the cards now that it's EOL but, if so I think it would be fair to bump the default protocol version.

WIP because they'll be lots that breaks on tox/travis. I will check back on that later.

Thank for this project.

I needed to use the joblib pickle methods for numpy/pandas/scikit-learn. As explained here, this method is more efficient for this kind of object.

So:

• I added a JobLibPickledObjectField
• I created BasePickledObjectFieldwhich is the parent class of PickledObjectField and JobLibPickledObjectField
• I subclassed the existing PickledObjectFieldCheckTests to create JobLibPickledObjectFieldCheckTests

In few words, the existing API isn't changed, and the new field has the same behavior than the old one.

For some reason after this commit in Django 3 I can no longer use default values for PickledObjectField in MySQL, I get the following error:

django.db.utils.OperationalError: (1101, "BLOB, TEXT, GEOMETRY or JSON column 'my_field' can't have a default value")

The error only occurs in MySQL 8.0.13+ (as mentioned in the commit) but the thing is that defaults has worked for me in Django 1.11-2.2 with MySQL 5.6-8.0.19 so I'm not sure why it has been constrained in Django 3.

Has anyone else had any experience with this or know a way around it?

(Maybe this is an issue more suited for Django itself but I thought I would start here since this is what I'm using for the fields that fails)

Currently, picklefield requires six for singe one-line function with_metaclass.

I'm not sure whether such change is desirable, but I think removing nearly-unnecessary dependency might be convenient.

trying to pickle a Query object. it seems like the default values is not being serialized:

DB is postgres 10.1 django 3.0.3 python 3.7.4

eligible_team_query = PickledObjectField(default=Team.objects.all().query)

  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.SyntaxError: subquery must return only one column
LINE 1: ...query", "eligible_team_query") VALUES ('u17', '', (SELECT "p...

Originally posted by @flokain in https://github.com/gintas/django-picklefield/issues/47#issuecomment-588190697

There seems to be issues if a mutable default value is used. I'm not sure the best way to handle this, though.

Example:

class MyModel(models.Model):
    pickle = PickledObjectField(default={})

x = MyModel()
x.pickle['a'] = 3
x.pickle  #  {'a': 3}

y = MyModel()
y.pickle   # {'a': 3}

Hi, i use django-picklefield in a project of mine and noticed that it's quite slow compared to pickling an object using jsonpickle (https://github.com/jsonpickle/jsonpickle), storing the result in a TextField, and unpickling the TextField back into a Python object. Do you know what's up with that?

I don't have any benchmarks to show for at present, but i'll make some when i get a chance. Thought i'd raise the issue now, though.

Hi, I had added ppc64le(Linux on Power) architecture support on travis-ci in the branch and looks like its been successfully added. I believe it is ready for the final review and merge. The travis ci build logs can be verified from the link below. https://travis-ci.com/github/kishorkunal-raj/django-picklefield/builds/188998356

Reason behind running tests on ppc64le: This package is included in the ppc64le versions of RHEL and Ubuntu - this allows the top of tree to be tested continuously as it is for Intel, making it easier to catch any possible regressions on ppc64le before the distros begin their clones and builds. This reduces the work in integrating this package into future versions of RHEL/Ubuntu.

Please have a look.

Regards, Kishor Kunal Raj

@carltongibson I replaced the DEFAULT_PROTOCOL bump by a new setting instead as this bump breaks backward compatibility of __exact lookups since the string representation of pickles changes.

We could have documented it as a backward incompatible change but I feared some folks might blindly upgrade and end up with a database that is a mix of protocol 2 and 3 pickles which would break __exact and friends.

By introducing this new PICKLEFIELD_DEFAULT_PROTOCOL setting it should be possible for users to upgrade to the version they seem for fit for their use case (even > 3) without breaking backward compatibility.

Django 3.2 is emitting additional warnings, which is causing two tests to fail.

[   28s] ======================================================================
[   28s] FAIL: test_mutable_default_check (tests.tests.PickledObjectFieldCheckTests)
[   28s] ----------------------------------------------------------------------
[   28s] Traceback (most recent call last):
[   28s]   File "/home/abuild/rpmbuild/BUILD/django-picklefield-3.0.1/tests/tests.py", line 243, in test_mutable_default_check
[   28s]     id='picklefield.E001',
[   28s] AssertionError: Lists differ: [<War[905 chars]001'>, <Warning: level=30, msg="Auto-created p[378 chars]42'>] != [<War[905 chars]001'>]
[   28s] 
[   28s] First list contains 1 additional elements.
[   28s] First extra element 3:
[   28s] <Warning: level=30, msg="Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.", hint="Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.", obj=<class 'tests.tests.PickledObjectFieldCheckTests.test_mutable_default_check.<locals>.Model'>, id='models.W042'>
[   28s] 
[   28s] Diff is 2280 characters long. Set self.maxDiff to None to see it.
[   28s] 
[   28s] ======================================================================
[   28s] FAIL: test_non_mutable_default_check (tests.tests.PickledObjectFieldCheckTests)
[   28s] ----------------------------------------------------------------------
[   28s] Traceback (most recent call last):
[   28s]   File "/home/abuild/rpmbuild/BUILD/django-picklefield-3.0.1/tests/tests.py", line 253, in test_non_mutable_default_check
[   28s]     self.assertEqual(Model().check(), [])
[   28s] AssertionError: Lists differ: [<Warning: level=30, msg="Auto-created pri[380 chars]42'>] != []
[   28s] 
[   28s] First list contains 1 additional elements.
[   28s] First extra element 0:
[   28s] <Warning: level=30, msg="Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.", hint="Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.", obj=<class 'tests.tests.PickledObjectFieldCheckTests.test_non_mutable_default_check.<locals>.Model'>, id='models.W042'>
[   28s] 
[   28s] - [<Warning: level=30, msg="Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.", hint="Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.", obj=<class 'tests.tests.PickledObjectFieldCheckTests.test_non_mutable_default_check.<locals>.Model'>, id='models.W042'>]
[   28s] + []

This allows me to customize the loads and dumps via settings to use dill instead of the default pickle. If no setting exists, defaults to pickle.loads and pickle.dumps

eg.

# settings.py
PICKLEFIELD_DUMPS = 'dill.dumps'
PICKLEFIELD_LOADS = 'dill.loads'

On systems where no binary build is available, django-picklefield cannot be installed in a clean virtualenv, because it relies on Django being installed at build time

To reproduce:

$ mkvirtualenv -p python3.8 test-django-picklefield-build
$ workon test-django-picklefield-build
$ pip install --no-binary django-picklefield django-picklefield
Collecting django-picklefield
  Using cached django-picklefield-3.0.1.tar.gz (9.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-m4z3m_ay
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The obvious fix is to maintain the version differently, e.g. https://github.com/niklasb/django-picklefield/commit/dc2979840cdb0862ac22ca4b0ab3dc41684720f3

This is such a particular case that I do not expect it to be solved... and actually one could argue that the bug is in the Django library.

Django specifies that if you want to pickle a QuerySet, you must pickle the query https://docs.djangoproject.com/en/3.2/ref/models/querysets/#pickling-querysets

The problem is that a query has a method "resolve_expression", and in that case, Django skip the call to get_db_prep_save. https://github.com/django/django/blob/3.0.14/django/db/models/sql/compiler.py#L1257

get_db_prep_save in turn is supposed to call get_db_prep_value responsible for the pickling of the object https://github.com/django/django/blob/3.0.14/django/db/models/fields/init.py#L819

As a result the query object is not pickled, and Django throws an error because it creates a sub select in the query, etc.

I edit PickledObjectField's in admin using editable=True flag where the assigned widget is a textbox. So I have to add the following line to my admin class: formfield_overrides = {PickledObjectField: {'widget': widgets.AdminTextareaWidget}, } I think it'd be nice if this was the default behaviour.