Fixes:

#6714

Proposed fixes:

I am integrating Flask-login into CKAN, and I have to mention that this is not the final code, so I will put PR on draft so we can discuss further changes, if you have an idea how to improve the code, do not hesitate to say.

Features:

• [x] includes tests covering changes
• [ ] includes updated documentation
• [ ] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply

“This PR is Financed by Salsa Digital www.salsadigital.com.au. Find all Services at www.salsadigital.com.au/services”

Draft coding standards doc here: http://docs.ckan.org/en/1117-start-new-test-suite/testing-coding-standards.html

After our discussion at the OKF summit about a new test suite for CKAN we started working on some example logic tests. These need to be finished up, example tests for other packages written, and newing testing coding standards written up.

May only do part of this in this PR and end up making some new issues for the rest...

was WebTest released version 2.0, which breaks CKAN

We don't set a explicit version to WebTest, and they just released version 2.0. This version requires WebOb >= 1.2, but we have WebOb==1.0.8 in https://github.com/okfn/ckan/blob/master/pip-requirements.txt#L19.

This is why Travis is failing like https://travis-ci.org/okfn/ckan/jobs/5073907.

First of four pull requests. required extensibility for personalization extension (ckanext-semantic)

[1] https://github.com/okfn/ckan/pull/184 [2] https://github.com/okfn/ckan/pull/185

Users need a quick way to get to their organization admin tab as well as to their groups and harvest sources (plus error reports etc).

I suggest this is all from the dashboard

This is not ready for review yet I'm just opening the pull request now so @johnmartin and I can discuss it. Please review!

Also I still need to add some backend refactoring and testing before merging this. Done

Fixes #3484

• Revision UI removed (as warned in CKAN 2.7.0 changelog):

  • /revision/list - List of which datasets changed. Not easy to page through 200 at a time. Not used or useful as far as we can tell.
  • /revision/{id} - Detail of a revision - displays timestamp and the dataset - no use. Problematic when there are billions of revisions you have billions of these pages that get indexed.
  • /revision/edit/{id} - to undelete a revision. No-one uses this method. Better ways to achieve this.
  • /revision/diff/{id} - diffs different versions of a dataset. Replaced by /dataset/changes/

• History page removed - /dataset/{id}/history now redirects to Activity Stream. The History page was not supported or linked to from anywhere (on default templates) but it was useful for admins. The functionality of seeing the past versions is fully replaced by the Activity Stream now. (The Atom feed at the same URL is also removed: /dataset/{id}/history?format=atom, again deprecated. But there is a feed of datasets ordered by metadata_modified at /feed/dataset.atom, and the changes are in the package_activity_list API which could be made an Atom feed if there is interest.)

• Activity Stream saves the full dictized dataset, rather than just the package table. Migration script: migrate_package_activity.py (discussed below)

• Activity Stream for a dataset now has 2 new links against each activity item: "View this version" to view the dataset as it was and "Changes" - the diff since the previous version: The changes (diff) page looks like this: ... This isn't the clearest - changes are not highlighted - you have to look in the margin for a letter. But it's a pretty good start and can be improved.

  • NB The "View this version" and "Changes" pages can viewed by admins and optionally others too - depending on new option ckan.auth.public_activity_stream_detail. The default is False, so that sites upgrading don't suddenly splurge this detail without taking the conscious act of changing it, although we encourage it in the changelog. New sites will have it set to True by default, because it is added that way in deployment.ini_templ.

• Viewing old versions of datasets is done with a different parameter name. Uses ?activity_id=<id> rather than @<revision_id> or @<date>. (This breaking change is ok because the only links to these were from the history page which was not linked to itself.)

• Activity Stream model is simplified - we use just Activity and remove ActivityDetail. Previously when you update a resource, it'd create an Activity('changed package') and also a connected ActivityDetail('resource changed'). This just seems too complex in the model. Now if you change just an extra field or a resource field, it'll simple say 'X updated the dataset Y', rather than 'X added the resource Z to the dataset Y'. Before: After: Since we store the pkg_dict on every commit, we could in future write logic that could analyze the diff to describe the change in more detail, like before. But it's worth it for the simpler model.

  • ActivityDetail is no longer used and is deprecated. It remains in the model for now, in case someone has some custom details they want to keep. But the user is asked to agree to deletion and then the table is emptied by when they run migrate_package_activity.py.
  • activity_detail_list logic function is removed

• New action functions for viewing an activity: activity_show and very similar: activity_data_show making command-line diffs possible e.g.:

diff <(ckanapi action activity_data_show id=484f01d7-d9c8-4a4d-bb42-00da6b8525f7 object_type=package -c $CKAN_INI) <(ckanapi action activity_data_show id=293a87ed-7ff9-4191-9d92-cb523763b8dd object_type=package -c $CKAN_INI)
15c15
<   "metadata_modified": "2018-01-02T16:29:44.571621",
---
>   "metadata_modified": "2018-01-02T16:29:44.834676",
80c80
<       "revision_id": "1e6a3a25-2099-4318-891c-a2dd78b173de",
---
>       "revision_id": "9e779436-fe43-400d-92ad-9f1583a60a60",
91,92c91,92
<   "revision_id": "9eddc9f4-a29b-4feb-aff6-bfcea87f2139",
<   "state": "draft",
---
>   "revision_id": "9e779436-fe43-400d-92ad-9f1583a60a60",
>   "state": "active",

• Diffing between package versions is also possible with a new action function activity_diff.
• migrate_package_activity.py - migrates activity stream to store the full package dict, using the package revision tables.
  • Package revision tables (package_revision/package_extra_revision/package_tag_revision/resource_revision) are not removed and are still being populated, but are not used for anything. (We keep them just in case: There is a chance that there are packages have been create/changed/deleted but there are not corresponding activity stream objects (due to disabling ckan.activity_streams_enabled, or for changes while running ckan before 1.6, or due to a bug certain changes are not recorded as activity). Regenerating activity stream objects for these would probably be possible from the revisions, so why not leave them.)
  • This does not migrate old versions of groups/organizations/member/package_relationship/system_info so those revision tables remain, to be tackled in future. Eventually we'll get rid of vdm.
  • This migration takes a while to run (could be hours for the biggest sites), and the effect is only noticeable to admins by default, so rather than include it in the normal paster db upgrade migrations, it is a standalone script. Ideally you download it and run it against any version of CKAN (I've tested it 2.3-2.8) before you upgrade to this version of CKAN. But if you don't, that is fine - you can upgrade to this or a future CKAN version, and run this migration if you want - the only downside of not migrating is that the "View this version" and "Changes" links will 404 until you do run the migration (and by default only admins can see Changes page anyway). (An earlier version of this PR prevented you upgrading until the migratino was done, but is no longer the case.)

I think we are agreed that it is ok to remove Revision UI and History page without prior deprecation because they are long unlinked and unsupported.

This PR removes a chunk of code to do with 'revisions'. However migrate_package_activity.py requires the revisions still in the model. A future PR (looking like https://github.com/ckan/ckan/pull/4664/files#diff-67c809480fe3487c995fb33d7d134d84) can remove the rest:

• tables: group_revision/group_extra_revision/member_revision/package_relationship_revision/system_info_revision - maybe we need a similar way to view changes to groups at least? maybe the rest can go.
• model code
• vdm

This PR follows on from https://github.com/ckan/ckan/pull/4627: Activity Stream is displayed with standard templates, rather than via action functions that returned HTML.

This PR took over from the original work on this issue that was #3485

Todo:

• [x] Go through review comments https://github.com/ckan/ckan/pull/3485
• [x] diffing - see if it is poss using the api and diff.
• [x] Check: _filter_activity_by_user commenting out
• [x] Remove ActivityDetail in a migration script
• [x] write test for migrate_package_activity.py (ensure it doesn't break in future)
• [x] move the revision stuff out of model_dictize code into migrate_revisions.py (including calling package_dictize with the (context['revision_id'] option)
• [x] config option for everyone to see diffs
• [x] test against old/real activity streams
• [x] Really old versions of CKAN included activity event types that no longer exist, and trying to render those activity events fails. We need to add a fallback template to render these without exception. This will be more important once this PR goes in since it will be much easier to add arbitrary event types. (TkTech: One important test we should have is to ensure activity doesn't fail to render when the database includes an activity type or objects no longer exist. There have been 4-5 issues about this on the mailing list from what I've seen.)
• [x] Rendering org/group events when the org/group has been deleted cause crashes on some templates.
• [x] check templates-bs2 are in sync

Features:

• [x] includes tests covering changes
• [x] includes updated documentation
• [x] includes user-visible changes
• [x] includes API changes
• [ ] includes bugfix for possible backport

https://datatables.net/ is a very nice alternative to the recline grid view we have now. This is my first attempt at a resource view plugin, be gentle :smile:

I followed the Docker installation instructions, and while the Solr and Postgresql containers are running, CKAN fails to build.

$ docker ps -a
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS                              PORTS               NAMES
596bbfb621f2        ckan/solr:latest         "java -jar start.jar   5 days ago          Up 38 minutes                       8983/tcp            solr
b3af69f91938        ckan/postgresql:latest   "/usr/local/bin/run"   5 days ago          Up 42 minutes                       5432/tcp            db

Supporting services are running, but then (running it interactively to see logs):

$ docker run -it --rm -p 8000:8000 --link db:db --link solr:solr ckan/ckan
*** Running /etc/my_init.d/50_configure...
Distribution already installed:
  ckan 2.4a from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
  /etc/ckan/default/ckan.ini
Edited option sqlalchemy.url = "postgresql://ckan_default:pass@localhost/ckan_default"->"postgresql://ckan:[email protected]:5432/ckan" (section "app:main")
Option uncommented and set solr_url = "http://172.17.0.41:8983/solr/ckan" (section "app:main")
Option uncommented and set ckan.storage_path = "/var/lib/ckan" (section "app:main")
Edited option email_to = "[email protected]"->"[email protected]" (section "app:main")
Edited option error_email_from = "paste@localhost"->"ckan@5c6d3e2d141c" (section "app:main")
*** Running /etc/my_init.d/70_initdb...
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 204, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 144, in _load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 232, in load_environment
    p.load_all(config)
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 124, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 181, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 209, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 270, in update_config
    search.check_solr_schema_version()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 306, in check_solr_schema_version
    % (version, ', '.join(SUPPORTED_SCHEMA_VERSIONS)))
ckan.lib.search.common.SearchError: SOLR schema version not supported: 2.0. Supported versions are [2.3]
*** /etc/my_init.d/70_initdb failed with status 1

*** Killing all processes...

Clearly there's a version conflict or a version string set incorrectly, but I'm not familiar enough with CKAN to understand how I might go about resolving this.

What is this?

This PR adds a Dockerfile and support files (including nginx and postfix configuration) for a binary CKAN 2.2 docker image. Specifically, this allows you to build a docker image including CKAN 2.2, running behind nginx and gunicorn, by running

docker build .

For example, to build an image called ckan/ckan tagged at version 2.2, you might run

docker build -t ckan/ckan:2.2 .

The resulting image contains only CKAN, with a nearly vanilla configuration. In order to use it, you must do one of two things. You can either use the vanilla configuration as-is, and this requires that you specify the location of a Postgres database and a Solr core on startup:

docker run -i -t -p 80:80 \
  -e DATABASE_URL=postgres://user:pass@hostname/db \
  -e SOLR_URL=http://hostname:8983/solr/ckan_default \
  ckan/ckan:2.2

This will run CKAN, connect to the database, and initialise it if need be. Configuring Solr will have to be done separately. There are a couple of other environment variables you can use to customise the deployment, including GUNICORN_NUM_WORKERS and ERROR_EMAIL, which do what you might expect.

Alternatively, and perhaps more realistically you can use this image as a base for extension. If a configuration file is injected to /etc/ckan/default.ini, the image will use that and ignore the DATABASE_URL, SOLR_URL, and ERROR_EMAIL environment variables. (GUNICORN_NUM_WORKERS is still used.)

A minimal Dockerfile that uses this as a base might look something like this:

FROM ckan/ckan:2.2

ADD ./mycustomconfig.ini /etc/ckan/default.ini

Lastly, by default the CKAN file store is at /var/lib/ckan, and in a production environment you would almost certainly mount this data volume outside the running container:

docker run ... -v /mnt/storage:/var/lib/ckan ...

Why should I care?

I'm of the opinion that deploying CKAN at the moment is too complicated. The package installation makes certain assumptions (Postgres and Solr on the same server; only one CKAN per machine) which seem unrealistically restrictive (not to say unwise) for production environments.

This setup allows you to trivially run multiple CKAN instances on a single machine, all pointing to Postgres and Solr by URL (either local or remote, it doesn't matter), without having to resort to the 10-step source install. It also makes it trivial to use LXC to memory constrain individual instances (docker -m 1g ...) which will be important in high-density deployments.

Perhaps more excitingly, this can be used as a binary build base for more complicated deployments, including those which need additional extensions and configuration. You can simply start from this image and add extensions/additional services as necessary. The image builds on baseimage-docker which makes running additional supervised services very easy.

What's not right yet?

1. Perhaps the most important omission from this PR is documentation updates. I'd like to have the discussion with people about whether this is something they want to see in the main CKAN repository before I commit to writing up the docs.
2. CKAN is currently autoconfigured using some questionable techniques. Perhaps we should instead permit configuring certain key config properties using environment variables? Perhaps some of what is currently configuration (site_title, site_logo, site_description...) should move into the database?
3. I don't know. You guys know far more about what a sensible CKAN deployment does than I do. Tell me what's missing.

What else should I know?

• I've taken the liberty of reserving the "ckan" username at the Docker index, so if we want to push this as suggested, to ckan/ckan:2.2 we can.
• For the time being, you can try out the nickstenning/ckan:2.2 image.
• NB: This PR merges to the release-v2.2 branch.

We should aim to support Ubuntu 14.04, probably by next release.

I'd expect the main issues would be on installation instructions and libraries / requirements compatibility.

Things to check: http://packages.ubuntu.com/trusty

• [x] Postgres 9.3 - SQL commands (eg datastore)
• [x] Solr 3.6 - Solr 4 has been widely used across CKAN instances, so I wouldn't expect problems here
• [x] Libraries versions (see eg #1635)
• ...

CKAN version 2.9.7

Describe the bug I tried to purge a dataset with the CLI. However, that fails due to a foreign key constraint on the package_relationship_revision table. See stracktrace below.

It seems that the purge command does not remove entries from the package_relationship_revision before it tries to delete entries from the package_relationship table.

Steps to reproduce

ckan dataset purge cb577ca0-d22c-48c2-bf57-58b9ba9c63e9

Expected behavior The purge command should remove all db entries related to the specified dataset.

Additional details

Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1243, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ForeignKeyViolation: update or delete on table "package_relationship" violates foreign key constraint "package_relationship_revision_continuity_id_fkey" on table "package_relationship_revision"
DETAIL:  Key (id)=(9b247c29-2b42-4dd1-9b39-36e03c19bd88) is still referenced from table "package_relationship_revision".

Fixes #5621

Proposed fixes:

Features:

• [ ] includes tests covering changes
• [ ] includes updated documentation
• [ ] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply

Fixes #7195

Proposed fixes:

Fix color contrast in 'add dataset' page.

Features:

• [ ] includes tests covering changes
• [ ] includes updated documentation
• [X] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply

Fixes #7194

Proposed fixes:

Make skip content visible

Features:

• [ ] includes tests covering changes
• [ ] includes updated documentation
• [X] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply

Fixes #7187

Proposed fixes:

Add title attribute to iframes.

Features:

• [ ] includes tests covering changes
• [ ] includes updated documentation
• [X] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply

Fixes #7186

Proposed fixes:

Fix bug report template to make heading tags semantic

Features:

• [ ] includes tests covering changes
• [ ] includes updated documentation
• [X] includes user-visible changes
• [ ] includes API changes
• [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply