This PR adds a new ISiteSearch to allow users to hook specific logic before/after each type of search.

This is inspired by before_package_search and after_package_search methods implemented in IPackageController:

This is a Work in Progress:

While working against master this Blueprint test fails with a CSRF error.

I think it is related to changes made in https://github.com/ckan/ckan/pull/7096/files so I have updated the test to use Authorization instead of REMOTE_USER.

Not sure what the strategy will be to update this tests.

This PR address the scenario to update an organization package_count field when a package is created via the UI. To do this we need to detect when package_update action is updating the package's state from draft to active

More Info:

Chaining package_create only covers API or CLI scenarios. (Meaning that we are updating an org package_count when adding a new dataset)

However, while creating a package using the UI package_create contains a draft dataset. This means that the call to reindex the owner_org will not update the package_count attribute since draft datasets are not counted.

The change of the package state from draft to active happens latter in the process when submiting a resource and calling to package_update.

Following #3 this PR covers the case when the Package's organization is updated.

We should trigger a rebuild both in the new org and the old org to properly reflect the package_count in the index.

When executing the command ckan sitesearch rebuild pages the current implementation throws a NameError:

Traceback (most recent call last):
  File "/usr/lib/ckan/venv/bin/ckan", line 8, in <module>
    sys.exit(ckan())
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/ckanext/sitesearch/cli.py", line 56, in rebuild
    rebuild_pages(defer_commit, force, quiet, entity_id)
  File "/usr/lib/ckan/venv/lib/python3.8/site-packages/ckanext/sitesearch/lib/rebuild.py", line 91, in rebuild_pages
    pages = Page.pages()
NameError: name 'Page' is not defined

This is because this condition evaluate to false when loading the module:

    if plugin_loaded("pages"):
        from ckanext.pages.db import Page

Moving this import inside the method works properly although I'm not sure why or how CKAN's cli manages imports and plugins load.

I cannot understand why this import worked when it was under the cli.py module, but not under lib/rebuild.py. 🤔

Issue

Currently when a package is created/deleted, the organization/group index is not refreshed. This means that any call to organization_search/group_search will not return an up-to-date package_count attribute.

Solution

• Refactored all the rebuild logic used in the cli command into a new lib called rebuild
• Created package_create chained action that calls a rebuild on the organization if the new dataset actually belongs to an Organization.
• Created package_delete chained action that iterates though all the package's groups and rebuild it's index.
• Created member_create chained action to rebuild the groups index if the action if for actually adding a package to a group.

Notes

I like the idea of a rebuild module since it isolates the responsibility to update the index to the own entity. Instead of calling an organization_show in the package_create we simply delegate to the organization itself the action of rebuilding it's own index.

Testing Notes

CKAN's fixtures are designed for each test to ensure the database is cleaned before running (instead of the canonical way of cleaning the database after running). This makes that the current implementation of fixtures to fail when doing several test runs because TestGroupOrOrgSearch will still have leftovers of the last test in the previous run. (Adding a "clean_db" fixture clashes with "group_search_fixtures")

Also, CKAN's fixtures are not conistent in the way they add/remove tables to the database. This can cause clean_db to fail cleaning the pages tables since it is found in the model metadata but not in the database itself (or vice versa). This scenario can happen when tests are executed unitarily. To "fix" it, just execute the whole suite of tests. (pytest ckanext/sitesearch)

To fix this we force a teardown_class method cleaning the index and database.

This is a simple implementation of a table to log all the search terms.

Due to initial requirements, it's necessary to log in the user_id so I'm not sure it's something we want to implement here. (I made it optional just in case).

Due to the current implementation of site_search calling other searches, we need to track if it's a site_search so we do not log it twice. I'm using flask's global object to track if the request is_site_search.