Let's add a CI action in GitHub to update a container image in DockerHub

• contact me (@ceteri) to get @DerwenAI credentials/secrets set up as needed
• update the .github/workflows/ci.yml file, see https://docs.github.com/en/actions/publishing-packages/publishing-docker-images
• add a build for the docker/Dockerfile container

This will help people who want to run the our notebook-based tutorial from DockerHub, i.e., what we use for the conference workshops.

@ceteri Paco, So I have some time to spend working with some schema.org based data from Hydroshare and exploring using kglabs to explore it. I'm having issues applying it and I hope in resolving them I might be able to help somehow in the docs and such.

Hopefully this isn't just me being stupid in graph space, but is of some help back to the project. Happy to share.

Working with the same data from Issue 24 and now trying the NetworkX area I got a specific error.

So this code:

import networkx as nx

sparql3 = """
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?subject ?object
WHERE { 
  ?subject a <https://schema.org/Dataset> .
  ?subject <https://schema.org/creator> ?creator .
  ?creator rdf:first ?o .
  ?o <https://schema.org/name> ?object
}
  """

subgraph = kglab.SubgraphMatrix(kg, sparql3)
nx_graph = subgraph.build_nx_graph(nx.DiGraph(), bipartite=True)

results in this error

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_3293779/2456468873.py in <module>
     13 
     14 subgraph = kglab.SubgraphMatrix(kg, sparql3)
---> 15 nx_graph = subgraph.build_nx_graph(nx.DiGraph(), bipartite=True)

~/.conda/envs/kglab/lib/python3.8/site-packages/kglab/subg.py in build_nx_graph(self, nx_graph, bipartite)
    250         """
    251         if self.kg.use_gpus:
--> 252             df = self.build_df()
    253             nx_graph.from_cudf_edgelist(df, source="src", destination="dst")
    254         else:

~/.conda/envs/kglab/lib/python3.8/site-packages/kglab/subg.py in build_df(self, show_symbols)
    223 
    224         if self.kg.use_gpus:
--> 225             df = cudf.DataFrame(rows_list, columns=col_names)
    226         else:
    227             df = pd.DataFrame(rows_list, columns=col_names)

~/.conda/envs/kglab/lib/python3.8/contextlib.py in inner(*args, **kwds)
     73         def inner(*args, **kwds):
     74             with self._recreate_cm():
---> 75                 return func(*args, **kwds)
     76         return inner
     77 

~/.conda/envs/kglab/lib/python3.8/site-packages/cudf/core/dataframe.py in __init__(self, data, index, columns, dtype)
    257                     )
    258                 else:
--> 259                     self._init_from_list_like(
    260                         data, index=index, columns=columns
    261                     )

~/.conda/envs/kglab/lib/python3.8/site-packages/cudf/core/dataframe.py in _init_from_list_like(self, data, index, columns)
    397         if columns is not None:
    398             if len(columns) != len(data):
--> 399                 raise ValueError(
    400                     f"Shape of passed values is ({len(index)}, {len(data)}), "
    401                     f"indices imply ({len(index)}, {len(columns)})."

ValueError: Shape of passed values is (5293, 5293), indices imply (5293, 2).

The results of that SPARQL on the graph should be like:

subject,object
https://www.hydroshare.org/resource/aefabd0a6d7d47ebaa32e2fb293c9f8a#schemaorg,Courtney G Flint
https://www.hydroshare.org/resource/f94ac7f8d8a048cdbd2610dfa7cd315b#schemaorg,Zhiyu (Drew) Li
https://www.hydroshare.org/resource/f9a75c0b289649aa844e84c24f9f5780#schemaorg,Young-Don Choi
https://www.hydroshare.org/resource/173875a936f14c22a5ba19c721adfb86#schemaorg,Remi Dupas
https://www.hydroshare.org/resource/f1116211202a4c069919797272023e62#schemaorg,Nathan Swain
https://www.hydroshare.org/resource/6d80e4bd00244b5dabaff34074cd3102#schemaorg,Garrick Stephenson
https://www.hydroshare.org/resource/25133b13a1fc4fca9187c2d4e272d4e8#schemaorg,Jessie Myers
https://www.hydroshare.org/resource/ca0f2f0f28ba40018ae64b973e2bb35a#schemaorg,Ruth B. MacNeille
https://doi.org/10.4211/hs.88454dae8c604009b684bfa136e5f7f4#schemaorg,Celray James CHAWANDA
https://doi.org/10.4211/hs.1c6034be6886412ba59970ab1157fa7e#schemaorg,Bethany Neilson
for 5293 lines

Currently KnowledgeGraph class is implemented in ~1000 lines of code. To improve design, readability and accessibility we should consider refactoring. I would suggest considering composition by grouping the different methods in components classes like:

class Loaders:
    ...
    def load_rdf(...):
        ...
    def load_jsonld(...):
       ...

class KnowledgeGraph:
    def __init__(...):
        self.loaders = Loaders()
    ...

So that the methods can be moved to different files like kglab/loaders.py and have a clearer layout of the code.

After refactoring the calls would look like:

import kglab

kg = KnowledgeGraph()
kg.loaders.load_rdf(...)

Please try pytest --nbmake examples/ex6_2.ipynb

New expected behaviour

see https://github.com/DerwenAI/kglab/issues/108#issuecomment-1048445124

Change logs

Add the materialized() method to kglab.py. Add an example for it at ex6_2.ipynb

Add docstring at the top of kglab.py

Following up after some work with Trino authors yesterday, there are needs ahead for better metadata modeling based on inference techniques, semantic technologies, etc., for example in Iceberg connectors.

Just found this about Morph-RDB :

• http://www.semantic-web-journal.net/system/files/swj2619.pdf
• https://morph.oeg.fi.upm.es/tool/morph-rdb
• https://github.com/oeg-upm/morph-rdb/

Probably a good thing to have a spike toward: how to integrate with Trino and Morph-RDB. Will check with @dachafra, et al.

Could be good to discuss with @dvsrepo and Asun about this, too? It could become an integration point for Recognai?

I'm submitting a

• [X] bug report (CRITICAL)

This subsumes #275

Currently when using either pyproject.toml or setup.py approaches to build source and wheel, the wheel for a release is ~400Kb while the source distribution is ~300Mb, which is too large for PyPi.

Note that the former was moved to wip/pyproject.toml since its use with Flit appeared to be bugging. Perhaps Poetry works better?

See the MANIFEST.in file, where there are both include and exclude statements getting used. For some reason, these rules are followed when building a wheel, but not when building a source distribution (sdist)

@Mec-iS @tomaarsen @jake-aft - do you have any suggestions about the library packaging here could be structured better for avoid this issue?

I'm submitting a bug report.

• I get a "AttributeError: partially initialized module 'kglab'" error when trying to run the kglab example on the readme

Current Behaviour:

When I run the code I get: AttributeError: partially initialized module 'kglab' has no attribute 'KnowledgeGraph' (most likely due to a circular import)

Expected Behaviour:

That it would work per the repo readme file

Steps to reproduce:

• Create a new Python project
• Install kglab
• Create new file test.py
• Copy the code from the readme file
• Run the code

Snapshot:

Environment:

• Python version 3.10.7
• pip version 22.2.2 (using pipenv)
• OS details MacOS 12.6
• IDE PyCharm 2022.2.2

Installed packages

aiobotocore              2.4.0
aiohttp                  3.8.3
aioitertools             0.11.0
aiosignal                1.2.0
appnope                  0.1.3
asgiref                  3.5.2
asttokens                2.0.8
async-timeout            4.0.2
attrs                    22.1.0
backcall                 0.2.0
botocore                 1.27.59
cachetools               5.2.0
certifi                  2022.9.24
cffi                     1.15.1
charset-normalizer       2.1.1
chocolate                0.0.2
colorama                 0.4.5
cryptography             38.0.1
csvwlib                  0.3.2
decorator                5.1.1
Django                   4.1.2
elementpath              3.0.2
executing                1.1.1
falcon                   3.1.0
frozenlist               1.3.1
fsspec                   2022.8.2
gcsfs                    2022.8.2
google-api-core          2.10.2
google-auth              2.12.0
google-auth-oauthlib     0.5.3
google-cloud-core        2.3.2
google-cloud-storage     2.5.0
google-crc32c            1.5.0
google-resumable-media   2.4.0
googleapis-common-protos 1.56.4
greenlet                 1.1.3.post0
html5lib                 1.1
icecream                 2.1.3
idna                     3.4
ipython                  8.5.0
isodate                  0.6.1
jedi                     0.18.1
Jinja2                   3.1.2
jmespath                 1.0.1
jsonpath-python          1.0.6
jsonpickle               2.2.0
kglab                    0.6.1
language-tags            1.1.0
MarkupSafe               2.1.1
matplotlib-inline        0.1.6
morph-kgc                2.2.0
multidict                6.0.2
networkx                 2.8.7
numpy                    1.23.3
oauthlib                 3.2.1
owlrl                    6.0.2
oxrdflib                 0.3.2
packaging                21.3
pandas                   1.5.0
parso                    0.8.3
patsy                    0.5.3
pexpect                  4.8.0
pickleshare              0.7.5
pip                      22.2.2
prettytable              2.5.0
prompt-toolkit           3.0.31
protobuf                 4.21.7
pslpython                2.3.0
ptyprocess               0.7.0
pure-eval                0.2.2
pyarrow                  9.0.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pycparser                2.21
Pygments                 2.13.0
pynvml                   11.4.1
pyoxigraph               0.3.6
pyparsing                3.0.9
pyshacl                  0.20.0
python-dateutil          2.8.2
pytz                     2022.4
pyvis                    0.3.0
PyYAML                   6.0
rdflib                   6.2.0
rdflib-jsonld            0.6.2
requests                 2.28.1
requests-oauthlib        1.3.1
rsa                      4.9
s3fs                     2022.8.2
scipy                    1.9.2
setuptools               65.4.1
six                      1.16.0
sql-metadata             2.6.0
SQLAlchemy               1.4.41
sqlparse                 0.4.3
stack-data               0.5.1
statsmodels              0.13.2
tqdm                     4.64.1
traitlets                5.4.0
uritemplate              4.1.1
urllib3                  1.26.12
urlpath                  1.2.0
wcwidth                  0.2.5
webencodings             0.5.1
wheel                    0.37.1
wrapt                    1.14.1
yarl                     1.8.1

I noticed some warnings in the CI runs for #263:

=============================== warnings summary ===============================
kglab/query/sparql.py:195
  /opt/kglab/kglab/query/sparql.py:195: DeprecationWarning: invalid escape sequence \?
    pattern = re.compile("(\?" + var + ")(\W)")  # pylint: disable=W1401

kglab/query/sparql.py:195
  /opt/kglab/kglab/query/sparql.py:195: DeprecationWarning: invalid escape sequence \W
    pattern = re.compile("(\?" + var + ")(\W)")  # pylint: disable=W1401

So I've updated this line to use the r string prefix instead.

• Tom Aarsen

From #248

This is the first draft for implementing automated RDF tests. The first batch of test from RDF-tests (basic) can be run with pytest tests/rdf_tests/test_rdf_basic.py -k test_rdf_runner -s from the project directory; tests are in tests/rdf_tests/dat. For the moment the only assertion implemented is a check on the length of returned results. This is already interesting as some kg query outputs return more or less bindings compared to the expected ones.

The script should work also on oxigraph-tests but it doesn't, it seems the read_manifest file provided with rdflib cannot parse the oxigraph manifest, maybe a discrepancy in the XML structure? @Tpt please provide some feedback.

Currently the tests are cut-pasted.

Tests with anomalies:

• Running basic/Non-matching triple pattern ERROR: Non-matching triple pattern resulted in object of type 'NoneType' has no len()
• Running basic/Basic - Prefix/Base 1 Basic - Prefix/Base 1 resulted in False
• Running basic/Basic - Var 1 Basic - Var 1 resulted in False
• Running basic/Basic - Var 2 Basic - Var 2 resulted in False
• Running basic/Basic - Term 6 Basic - Term 6 resulted in False
• Running basic/Basic - Term 7 ERROR: Basic - Term 7 resulted in Expected {SelectQuery | ConstructQuery | DescribeQuery | AskQuery}, found '.' (at char 140), (line:5, col:23)

The ones resulting in "False" mean length discrepancies between expected and actual output, the one with errors are exceptions raised.

After successfully following the instruction for the installation I got, over and over, the following error. I have installed it in a Conda environment.

Add support for W3C CSV on the Web (CSWV).

• integrate use of csvwlib
• validate with CSVW Test suites

See also:

• https://w3c.github.io/csvw/
• https://www.w3.org/community/csvw/

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements.txt

pyvis 0.1.9 requires networkx, which is not installed.
owlrl 6.0.2 requires rdflib, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3180412 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

Current behaviour

pyproject.toml uses flint to build

New expected behaviour

Issue: Move setup.py functionality to poetry for pre-check and package build and release - https://github.com/DerwenAI/kglab/issues/275

Build and publish the package with poetry in the pyproject.toml

I'd like to recommend that we put this on a branch to further test and validate it is doing everything we want/need it to do

Unsure of the Makefile that builds the docs and how it might be integrated into pyproject.toml with poetry Need to review some of the pre-commit-config options

I am working on poetry build / publish instructions these will need to be integrated with the docs build instructions for whomever will be building the package not sure if this is something we need in the repo?

The pyproject.toml needs to have the version changed to the version that will be correct at the time of merge

Change logs

added exclude to pyproject.toml of the tests/rdf_tests to not be included in the package

Updated pre-commit-config to check the poetry additions Updated pyproject.toml to use poetry build and dependency layouts

Size of the package is now ~50 K for both whl and zip

Current behaviour

No instructions to install kglab with pipenv or poetry Issue: Improve installation- https://github.com/DerwenAI/kglab/issues/272

New expected behaviour

Instructions to install kglab with pipenv or poetry

Change logs

Instructions to install kglab with pipenv This is not for building the package or development work on kglab project.

Instructions to install kglab with poetry This is not for building the package or development work on kglab project.

Readme updated with Instructions to install kglab with pipenv or poetry Created new directory to test the install envs

Removed and decoupled the Poetry Build changes from the previous PR. These will follow in a future PR

Current behaviour

Subgraph operations are wrapped in Subgraph #273

New expected behaviour

Frame is used in place of Subgraph

Change logs

• no more Subgraph class, welcome Frame class
• subg.py renamed to frame.py

Please double-check that documentation files are rendered correctly according to new naming.

P.S. I had to rerun all the notebooks.

I'm submitting a

[x] feature idea to research and discuss

This is split from https://github.com/DerwenAI/kglab/issues/272

Current Behaviour:

Investigate and review the implication of changing the existing use of setup.py and using poetry instead for pre-check, package build, and release.

One thing I am not at all clear on is the documentation building for KGLab, and how that will need to be managed in a poetry env / process.

SubGraphMatrix (and in perspective SubGraphTensor) is the reference class for graph algebra and network analysis. Would it be better to rename the subg.py module and classes therein for encompassing a more general approach?

For example:

• subg.py -> ?
• SubGraphMatrix: keeping the fact that a SPARQL query is needed (so the subgraph naming), it would be better from a data scientist point-of-view to have this class to follow some more popular convention like for example GraphFrame or DataGraph or NetFrame (just throw a die with the right naming permutations, I though about this names: graph, frame, net, datagram, data, table, ...)
• SubGraphTensor -> as above for future applications

cc: @ceteri @tomaarsen