Separation in / configuration of Docker containers for separation of services and easier deployment on (mutiple) server(s).

After more usage of/migration to REST-APIs / Microservices like https://github.com/opensemanticsearch/open-semantic-search-apps/issues/19 and implementation of Open Semantic Entity Search API separation in docker container should be easier, since generation/update of configs can be done on different/distributed servers over network.

Hello, newbie in OSS.

Trying to delete all the tags and facets author, person, message to... etc, from OSS Desktop, but didnt work. Tried it over django web UI, deleting all facets, but didnt work. Is it possible to do? Thats a lot of unnecessary information for us. And the tags some times increase too much. I would like to make a tag only with the folder where the file is in. is it possible?

Thanks

Status page showing count of now yet fully processed documents / running ETL tasks and occured ETL errors like failed plugins or access rights errors.

I have an open-semantic-search server running since March on an up-to-date debian 9.6 VM (KVM) with more than 4 millions documents successfully indexed. Everything was running smoothly until a few days ago, but now indexing new files always results in an error (and the files are not indexed anymore):

 Traceback (most recent call last):
  File "/usr/bin/opensemanticsearch-index-dir", line 274, in <module>
    connector.commit()
  File "/usr/lib/python3/dist-packages/opensemanticetl/etl.py", line 295, in commit
    self.exporter.commit()
  File "/usr/lib/python3/dist-packages/opensemanticetl/export_solr.py", line 220, in commit
    request = urllib.request.urlopen( uri )
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Server Error

I have no idea what is wrong with my machine and what to look at. Adding more disk space as suggested in issue #115 did not solve the problem (I have 238GB free and 154GB used). The machine has plenty of RAM (15GB, but never uses more than 10) and 8 cores available from the host, and everything was running well for almost a year.

Thanks in advance for your help. I can provide more information if you need.

Hi

I have the desktop version installed on my system with the intention to be able to tag a large database of word documents and find reference across the whole database. So far so good but as I'm running tests with a subsection of documents (600-700) I'm finding that some of them won't be properly indexed as the search engine can't access the text body. On search these documents merely return results in the document title but nothing of the text body and the preview tab is empty. However I am able to open them directly through Libre Office, so there can't be a problem with permissions.

It works fine for the majority of documents so I'm at a loss what the difference is.

Any help would be greatly appreciated

I am trying to install Open Semantic Search Server on an AWS vm with Ubuntu 16.04.3 LTS (Xenial Xerus) by using ALL IN ONE Package: https://www.opensemanticsearch.org/download/open-semantic-search_18.02.02.deb.

Below are some of the errors I am getting in installation traces, any help on it would be great:

apache2_reload: Your configuration is broken. Not reloading Apache 2 apache2_reload: apache2: Syntax error on line 216 of /etc/apache2/apache2.conf: Could not open configuration file /etc/apache2/conf-enabled/solr-php-ui.conf: No such file or directory

However, I checked and I can see solr-php-ui.conf in the path /etc/apache2/conf-enabled.

_ImportError: No module named 'opensemanticetl' chown: cannot access '/var/lib/opensemanticsearch/db.sqlite3': No such file or directory chmod: cannot access '/var/lib/opensemanticsearch/db.sqlite3': No such file or director_y

Also once the installation completes, I get 500 error on clicking effectively on all the menu items. And in error logs of apache2, I am getting numerous wsgi:error like:

[Wed Feb 07 10:32:01.849612 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] mod_wsgi (pid=10969): Target WSGI script '/var/lib/opensemanticsearch/opensemanticsearch/wsgi.py' cannot be loaded as Python module. [Wed Feb 07 10:32:01.849660 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] mod_wsgi (pid=10969): Exception occurred processing WSGI script '/var/lib/opensemanticsearch/opensemanticsearch/wsgi.py'. [Wed Feb 07 10:32:01.849764 2018] [wsgi:error] [pid 9645] [client 127.0.0.1:38168] mod_wsgi (pid=9645): Target WSGI script '/var/lib/opensemanticsearch/opensemanticsearch/wsgi.py' cannot be loaded as Python module. [Wed Feb 07 10:32:01.849814 2018] [wsgi:error] [pid 9645] [client 127.0.0.1:38168] mod_wsgi (pid=9645): Exception occurred processing WSGI script '/var/lib/opensemanticsearch/opensemanticsearch/wsgi.py'. [Wed Feb 07 10:32:01.849817 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] Traceback (most recent call last): [Wed Feb 07 10:32:01.849864 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] File "/var/lib/opensemanticsearch/opensemanticsearch/wsgi.py", line 14, in [Wed Feb 07 10:32:01.849870 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] application = get_wsgi_application() [Wed Feb 07 10:32:01.849880 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] File "/usr/lib/python3/dist-packages/django/core/wsgi.py", line 14, in get_wsgi_application [Wed Feb 07 10:32:01.849885 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] django.setup() [Wed Feb 07 10:32:01.849893 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] File "/usr/lib/python3/dist-packages/django/init.py", line 18, in setup [Wed Feb 07 10:32:01.849898 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] apps.populate(settings.INSTALLED_APPS) [Wed Feb 07 10:32:01.849907 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] File "/usr/lib/python3/dist-packages/django/apps/registry.py", line 78, in populate [Wed Feb 07 10:32:01.849911 2018] [wsgi:error] [pid 10969] [client 127.0.0.1:38166] raise RuntimeError("populate() isn't reentrant")

Since thanks to SKOS thesaurus concepts or entities can now have relations like broader or narrower or a tree structure: Support this taxonomies/hierarchies and RDF (sub)classes in search UI for faceted search.

I'm trying to install the filesystem monitoring *.debs on Ubuntu Xenial, but I receive the following error message:

opensemanticsearch-trigger-filemonitoring : Depends: opensemanticsearch-connector-files (>= 0)
but it is not installable

What am I'm doing wrong?

FWIW here are all setup steps done on this server:

sudo apt install ./open-semantic-search-server-ubuntu_xenial_16.10.10.deb
sudo apt-get install unoconv
sudo apt install ./opensemanticsearch-trigger-filemonitoring_15.06.26_all.deb
...
opensemanticsearch-trigger-filemonitoring : Depends: opensemanticsearch-connector-files (>= 0)
but it is not installable

Enhance Named Entities Manager towards Thesaurus Manager:

• Consolidate models from tagging app to use names or concepts from thesaurus manager.
• Since concepts or entities can now have relations like broader or narrower or a tree structure: Support this taxonomies/hierarchies in search UI for faceted search
• Use the alternate labels, aliases, synonyms and hidden labels / misspellings not only for faceted search but for full text search, too. I.e. by exporting to synonyms.txt for Solr or enrich search query
• Add UI element for choice of language for multilingual thesaurus (to set/change field "lang" in models)

on a fresh debian stretch: there is no export of entities to neo4j.

Version: open-semantic-search_18.12.23

already installed: py2neo + export_neo4j, and activated by config!

no logs, something missing?

I am using Open Semantic Search Server from https://www.opensemanticsearch.org/download/open-semantic-search_18.02.07.deb and trying to upload an user defined ontology from Manage Structure -> Ontologies -> Add New Ontology.

After clicking on Save, I am being navigated to a Django Error Page (in attachment) that states - PermissionError at /ontologies/create [Errno 13] Permission denied: '/var/solr/data/core1/conf/named_entities/tmp_ocr_dictionary.txt' -> '/etc/opensemanticsearch/ocr/dictionary.txt'

On disabling Django Debug, all I am getting is 500 error - nothing much in logs that I have.

My basic aim here is to use the ontology for tagging documents which is not happening at present. PermissionError at _ontologies_create.zip

Any assistance or leads on it would be great :)

Hey,

first of all, amazing piece of work that you've done! While indexing our server we've probably encountered a bug regarding the mapping of file paths:

Our current mapping ist defind in /etc/opensemanticsearch/connector-files, with the aim to make search results directly accessible through a apache server: config['mappings'] = { "/mnt/server/": "http://192.168.2.20/server/" } Additionally we made a ln -s /mnt/server /var/www/html/server This works great so far for nearly every file.

But for any file (like .pdf) inside E-Mails (.msg) or E-Mail archives (*.pst), which means for every E-Mail attachment, the mapping results in: http://192.168.2.20/mnt/server/... instead of: http://192.168.2.20/server/...

So, the question is, did we make any mistake regarding the configuration or is there a bug regarding attachments?

Best regards Josef

Hi, I installed the latest Opemsemanticsearch Version as deb-Package in my Ubuntu 22 LTS Hyper-V machine. I'd like to use OSS for our about 1700 docx documentations of non standard Feature of our Software. The indexing of the docs worked without any problems.

My Problem is: By Default all docx are tagged with multiple default tags, I think the came from the 'Apache solr"!?! Here some examples:

Is it possible to deactived this tags in the apache solr? If tried the following, which didn't worked:

• Delete / Clear the index of OSS
• Edit the {{{/var/solr/data/opensemanticsearch/conf/_schema_analysis_synonyms_skos.json}}} and delete all entries instead of one
• Reload the 2 OMS Cores with {{{curl http://localhost:8983/solr/admin/cores?action=RELOAD&core=opensemanticsearch}}} {{{curl http://localhost:8983/solr/admin/cores?action=RELOAD&core=opensemanticsearch-entities}}}
• After reindex my docx, the default tags are still there
• I have looked inside the '/var/opensemanticsearch/db' sqllite DB, too, but didn't find something useful

Did anyone has a hint, to get rid of the default tags?

Hello,

I try to crawl a webpage (full domain) but never will be crawled more than the startpage. In the Datasources UI I tried http and https, with www and without, with trailing slash and without. It never works. I would expect that the crawler will follow the links found in the startpage. I have no idea why it does not work as expected.

(The whole installation was made on bullseye with "one command" as documented in https://opensemanticsearch.org/doc/admin/install/search_server/ )

Generic ingestor for JSON, so existing Open Semantic ETL plugins like Named Entity Extraction by Thesaurus and Ontologies, Stemming dependent on language detection and settings and indexing to Open Semantic Search schemas will/can be be used automatically for many additional datasources without developing an specialized importer.

I am trying to set up a dev environment with round trip from source to build on a server. I'm most of the way there, and the main menu looks pretty good. But when I press the Config button I get:

OperationalError at /setup/
no such table: setup_setup

Exception Location: | /usr/local/lib/python3.9/dist-packages/django/db/backends/sqlite3/base.py, line 423, in execute

Any advice? thanks!