Title says it all.

A quick check of requirements.txt with caniusepython3 claims these dependencies needed to be Python 3 compatible first:

• js.jquery (which is blocking js.bootstrap)
• js.chosen
• js.jquery_form (which is blocking js.deform)
• js.jquery_maskedinput (which is blocking js.deform)
• js.jquery_maskmoney (which is blocking js.deform)
• js.jquery_sortable (which is blocking js.deform)
• js.jquery_timepicker_addon (which is blocking js.deform)
• js.jqueryui (which is blocking js.deform)
• js.modernizr (which is blocking js.deform)
• js.tinymce (which is blocking js.deform)
• js.fineuploader
• js.html5shiv
• js.jquery_tablednd
• js.jqueryui_tagit
• kotti_tinymce
• plone.scale
• pytz
• repoze.zcml (which is blocking repoze.workflow)
• shutilwhich
• yurl

Some (most?) of them are false positives, because they do officially support Python 3 and are just lacking the corresponding trove classifiers. All they js.* should work with no or very little modification. shutilwhich is a Python 2 backport of a module from the Python 3 standard library.

This might change until we start working on this ourselves, but even if it won't and we'd need to add Python 3 support to some dependencies ourselves, this seems absolutely doable.

What do you think about providing by default a simple but nice not found view for Kotti instead of the standard NotFound white page? I'd like to hear your opinion

@tiberiuichim writes: Open issues:

• [x] No work done on UploadedFile.public_url integration, this means that filedepot's middleware is not usable right now - because no links are generated that would use it. I have created the UploadedFileResponse class which can be used to stream files to the browser. I don't think that we should make the middleware a default for Kotti, that would mean all files are openly available, without any security. An option would be to use the UploadedFileResponse to send a redirection to the public_url, where it is set by the depot that stores the files. The problem is, the proper filedepot way to get a file's url is to call DepotManager.url_for(path), which calls DepotManager.get_middleware().url_for(path) where path is 'depotname/file_id'.
• [x] We need a kotti.depot_default configuration option. I'm not sattisfied with the way things are now: we're adding the default 'fs0' depot, but once in use, you can't change its backend (or those files would stop working without a migration). Because I've added that option to kotti's conf_defaults, you can't really get rid of it anyway - you can only change its backend class by adding another kotti.depot.fs0.backend option in the .ini file. I have assumed that the default for multiple configured backends could be controlled by the order in which they are specified in the .ini files, but it seems that pyramid likes to sort the settings, so that can't be relied.
• [x] Maybe the File.data column should be renamed to File._data. This way we can have a setter for it and do the conversion to cgi.FieldStorage for the incoming data
• The following documentation sections need changes and updates:
• [x] Configuration http://diskotti.readthedocs.org/en/feature-rtd_theme_339/developing/basic/configuration.html
• [x] Blob-Storage http://diskotti.readthedocs.org/en/feature-rtd_theme_339/developing/advanced/blobstorage.html
• [x] Maybe a separate entry in the Developer manual, detailing how to create and serve new blob fields.

We need to get rid off deprecation warnings; Kotti needs to use pytest funcargs everywhere. Currently, a lot of deprecation warnings appear when we run tests.

We also need to allow add-ons to use funcargs. Also, docs need to be written on what funcargs exist with Kotti and how to use them. A link to these docs should be added to the deprecation warning.

Deliverables:

• [x] stop using unittest.TestCase derived test classes from kotti.testing
• [x] fix all deprecation warning regarding the above
• [x] expose Kotti's custom pytest fixtures so add-on can reuse them
• [x] add doc strings to all fixtures
• [ ] add documentation about recommended test style
• [ ] add link to documentation to deprecation warnings

Kotti is apparently not thread safe. For example, if two kotti instances run in the same Python process, things get really messed up:

[app:kotti2]
use = egg:kotti
---snip---

[app:kotti]
use = egg:kotti
---different DB, configuration, ...---
---snip---

[filter:fanstatic]
use = egg:fanstatic#fanstatic

[server:main]
use = egg:waitress#main
host = localhost
port = 5000

[composite:main]
use = egg:Paste#urlmap
/one = kotti2
/two = kotti

The problem seems to be the use of globals in Kotti, e,g, kotti.views.events uses globals to register event listeners. A solution could be to use thread-local data structures. Or don't use the globals at all but pass them around, maybe using pyramid's registry in some way?

IMHO kotti should be thread safe. It is some extra work, but worth it.

At the moment, class File stores contents in the database. This does not scale well and makes database backups a burden in case there are a large number of files stored. Ideally, File should allow contents to be stored externally to the database, via pluggable storage providers, making it possible to store data in the filesystem or in the cloud, for example.

Implementation Overview

The idea would be create an additional field in the database which contains information specific to external storage providers. In case this field is left blank, the current behaviour is not altered, and File contents are kept in the database. In case this field is filled with some sort of URL, an external provider will be selected accordingly, allowing File contents to be stored and retrieved accordingly.

Current test cases must continue to pass. Additional test cases need to be created so that the additional field (let's call it storage) is employed.

Implementation Alternatives

At first glance, SQLAlchemy-ImageAttach ( http://sqlalchemy-imageattach.readthedocs.org/ ) appears to be an excellent candidate for speeding up the implementation of this new feature. Unfortunately, the API proposed by SQLAlchemy-ImageAttach is specifically designed with images in mind, as opposed to any kind of attachment. Browsing the source code of SQLAlchemy-ImageAttach it becomes apparent that this dependency appears in several places, which does not seem to be convenient to us.

A second alternative would be employing OFS ( https://pypi.python.org/pypi/ofs ), which is BLOB oriented and presents a generic API "documented" here: https://github.com/okfn/ofs/blob/master/ofs/base.py . There are concrete implementations employing local data storage in this directory ( https://github.com/okfn/ofs/tree/master/ofs/local ) and remote implementations employing S3 among others in this directory ( https://github.com/okfn/ofs/tree/master/ofs/remote ). Despite OFS documentation is far from great, browsing the source code demonstrated to be simple and clean enough.

Despite OFS looks sufficiently simple, extensible and mature, i would like to allow even more flexibility, namely: do not depend explicitly of OFS, since OFS maybe sometimes tightly dependent on CKAN... and I had already my fair share of frustration with CKAN. Anyway, the idea is allowing OFS to be one (and possibly the most important and popular) alternative for external data storage in Kotti, but not the only one. More details about this topic are discussed later below.

Proposal of implementation

When Kotti starts, a singleton class called Storage is instantiated and stored in settings. This singleton class retrieves information from the configuration and initializes itself accordingly to the OFS implementation chosen, either local or remote. There's also possibility that nothing is specified in the configuration which, in this case, means that Storage is None, which means that the database should be employed by File as it is at the moment. Everytime a File is stored or retrieved, the [singleton] Storage is obtained from settings and employed accordingly, in case it is not None.

When a File is constructed, it's __init__ method should accept an optional argument storage which, in case it is specified, allows that specific instance of File to employ a specific instance of Storage. This is helpful in case storage migration or data reorganization is needed, allowing multiple "Storages" and not the only one provided by the singleton Storage. More details about this are presented later below.

When a File record is stored in the database, an additional field called storage is also persisted, making it possible to select an appropriate storage provider later, when that File requires data retrieval which, by the way, must be deferred in the same way it is deferred at the moment when internal data storage in the database is employed.

Details of Implementation

At the moment, I have an implementation "already working" but not integrated to File as proposed here. So, the next step would exactly this sort of integration we are talking about here.

I'm revealing some ideas below in order to allow more experient Kotti developers have their voice as soon as possible, so that we can speed up the implementation as fast as we can.

Below we have a draft of specification which eventually could be reorganized to a better formalized thing by someone, if desired.

Storage

1. class Storage implements a marking interface IStorage.

2. At the moment, Storage recognizes configuration entries started by storage.ofs, which makes OFS a first class citizen for external data storage, but not the only one. In future, other libraries or implementation would be possible to be plugged in, not only OFS.

3. A system-wide (singleton) Storage instance is saved into settings. In case no configuration items are found, a null Storage reference is created and saved into settings.

4. OFS itself permits several storage implementations, all of them having their own specific details in regards to initialization. For PTOFS ( https://github.com/okfn/ofs/blob/master/ofs/local/pairtreestore.py ), for example, it would be enough these two configuration entries:

   storage.ofs.class=ofs.local.PTOFS storage.ofs.ptofs.directory=/path/to/storage/directory

5. At this point, I'm supporting solely PTOFS. In future Storage would be able to recognize several configuration settings, being able to initialize not only PTOFS, but also other specific OFS implementations. NOTE (5.a) : This is only a limitation of scope for this initial implementation, since I have very limited time for dedicate for it.

6. At this point, our IStorage interface allows operations: store, retrieve and remove. NOTE (6.a) We are not supporting changes in the metadata at this point. See "Example Usage" at https://pypi.python.org/pypi/ofs for more clarification about this.

Example

This is a very brief example about how it look like:

    storage = settings['storage']
    url = storage.store('This is some data', filename='test.txt')
    with storage.retrieve(url) as f:
        for line in f:
            data += line
    assert(data == 'This is some data')
    storage.remove(url)

File

1. StoredFile extends File
2. StoredFile does not implement any additional interfaces, but it employs composition, emplying the already mentioned [singleton] Storage mentioned above.
3. StoredFile also allows an optional initialization parameter called storage which permits that a specific instance of StoredFile employs a distinct Storage instance from the system-wide [singleton] Storage. NOTE (3.a): Now it becomes clear that Storage is a singleton only in general. Certain applications, like storage reorganization or storage migration would employ multiple Storage instances, but this would be exceptions to the general rule.

When using pytest-pep8-1.0.3 we get the following error

local/lib/python2.7/site-packages/pytest_pep8-1.0.3-py2.7.egg/pytest_pep8.py:37: in pytest_collect_file
>               return Pep8Item(path, parent, pep8ignore, config._max_line_length)
local/lib/python2.7/site-packages/pytest_pep8-1.0.3-py2.7.egg/pytest_pep8.py:54: in __init__
>       self.keywords["pep8"] = pytest.mark.pep8
local/lib/python2.7/site-packages/pytest-2.3.2-py2.7.egg/_pytest/mark.py:122: in __getattr__
>           self._check(name)
local/lib/python2.7/site-packages/pytest-2.3.2-py2.7.egg/_pytest/mark.py:137: in _check
>           raise AttributeError("%r not a registered marker" % (name,))
E           AttributeError: 'pep8' not a registered marker

When running tests against SQLAlchemy 0.8 (as opposed to the currently pinned 0.7.9), we get these warnings from SQLA:

x86_64.egg/sqlalchemy/orm/unitofwork.py:39:
SAWarning: Usage of the 'collection append' operation is not currently
supported within the execution stage of the flush process. Results may
not be consistent.  Consider using alternative event listeners or
connection-level operations instead.
  sess._flush_warning("collection append")
/home/daniel/co/Kotti/local/lib/python2.7/site-packages/SQLAlchemy-0.8.0-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py:1390:
SAWarning: Usage of the 'Session.add()' operation is not currently
supported within the execution stage of the flush process. Results may
not be consistent.  Consider using alternative event listeners or
connection-level operations instead.
  self._flush_warning("Session.add()")

See this thread for details.

Hey Disko,

This branch removes your |version| variable in the Installation docs, and uses the github master branch instead. This allows someone to follow the Installation and Tutorial together.

I'm not sure if you want/need this right now. We need to figure out what to do long-term though.

Do with it what you wish :)

Mike

Currently it is not possible to delete a user. The easiest way to provide this is IMHO adding a button to the setup-user view. E.g. like this:

diff --git a/kotti/views/users.py b/kotti/views/users.py
index 884670c..eeb814f 100644s/--- a/kotti/views/users.py
+++ b/kotti/views/users.py
@@ -404,6 +404,10 @@ def users_manage(context, request):


 class UserEditFormView(EditFormView):
+    buttons = (Button('save', _(u'Save')),
+               Button('cancel', _(u'Cancel')),
+               Button('delete', _(u'Delete')))
+
     @property
     def success_url(self):
         return self.request.url
@@ -411,6 +415,17 @@ class UserEditFormView(EditFormView):
     def schema_factory(self):
         return user_schema(PrincipalBasic())

+    def delete_success(self, appstruct):
+        principals = get_principals()
+        user = principals.search(email=appstruct['email']).first()
+        if user is not None:
+            principals.__delitem__(user.name)
+            self.request.session.flash(_(u'User deleted.'), 'info')
+        else:
+            self.request.session.flash(_(u"Can't delete user."), 'error')
+        location = "%s/@@setup-users" % self.request.application_url
+        return HTTPFound(location=location)

What I don't like is the that it not feel really comfortable to have the buttons Save, Cancel and Delete in one row. Any likes or other ideas for an implementation?

Don't know if it is a kotti specific compatibility issue (full logs here https://pastebin.com/i6nPDH3Y):

-> return self.pkg_resources.resource_isdir(self.pkg_name, self.path)
  (Pdb) self.pkg_name
  'kotti'
  (Pdb) self.path
  'locale/'
  (Pdb) self.pkg_resources.resource_isdir(self.pkg_name, self.path)
  *** NotImplementedError: Can't perform this operation for unregistered loader type

Requirements:

• https://pastebin.com/nawCcEbj

Python version:

• 3.6.8

Anyone already saw a similar failure in kotti? Any hint about some compatibility issues in my dependencies versions?

Thanks in advance

Bumps mako from 1.2.0 to 1.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

https://github.com/disko/kotti_media is not being maintained, and I have requested to be a maintainer myself now.

I strongly suspect that the code is no longer viable, and so I may use the vlc media player instead.

Bumps waitress from 2.1.1 to 2.1.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

When a node's name or parent changes, Node.path needs to be recalculated recursively for all children of that node. The current implementation (a SQLAlchemy event handler) is very expensive and should be replaced by a pure SQL operation.

A CTE that could be used instead exists as a comment in https://github.com/Kotti/Kotti/blob/master/kotti/traversal.py#L183.

Also see:

• http://www.jeffwidman.com/blog/847/using-sqlalchemy-to-create-and-manage-postgresql-materialized-views/

• http://www.jeffwidman.com/blog/827/traversing-a-tree-stored-as-an-adjacency-list-using-a-recursive-cte-built-in-sqlalchemy/

This PR should be merged after a Kotti release, as it introduces changes to .po files.

I've introduced a kotti.util.translate_pluralize, maybe we should rename that to pluralize ?