Match-making API for OpenSanctions

Last update: Dec 15, 2022

Overview

OpenSanctions Match-making API

This directory contains code and a Docker image for running an API to match data against OpenSanctions. It is intended to be run on-premises in KYC contexts so that no customer data leaves the deployment context.

Demo instance

See https://api.opensanctions.org

Usage

In order to use the OpenSanctions API, we recommend running an on-premises instance on your own servers or in a data center. Updated images of the API with current data are built nightly and can be pulled from Docker hub:

mkdir -p yente && cd yente
wget https://github.com/opensanctions/yente/blob/main/docker-compose.yml
docker-compose up

This will make the matching API available on Port 8000 of the local machine.

If you run the container in a cluster management system like Kubernetes, you may want to find a way to pull a fresh container every night so that a new image with updated data will be pulled from the Docker registry. You will then also need to re-run the indexer, the equivalent of the last line in the example above.

Please contact the OpenSanctions team if you are interested in exploring a hosted solution for running the API.

Settings

The API server has a few settings, which are passed as environment variables. The settings include:

YENTE_ENDPOINT_URL the URL which should be used to generate external links back to the API server, e.g. https://osapi.mycompany.com.
YENTE_CACHED can be set to "true" in order to load all data to memory on startup. This will make the API incredibly fast, but consume 3-4GB of RAM.
YENTE_SCOPE_DATASET can be used to define the main dataset being used. This is usually default, but can be set e.g. to sanctions to load a more specific set of data.

Development

For development, install package like this:

pip install -e .

Finally, you can run an auto-reloading web server like this:

uvicorn yente.app:app --reload

Comments

Fuzzy matching not working
Hi,

The fuzzy matching parameter has no effect:

I tried to return results for https://api.opensanctions.org/search/default?q=Barrrack%20Obama&fuzzy=true and it should return a result since there's only 1 letter changing

I checked in the code https://github.com/opensanctions/yente/blob/main/yente/search/queries.py#L85 and in Elastic Search documentation, it should work but as a matter of fact, it does not.

Searching on Google returns results linked to a wrong mapping but I could not find any problem in the ES mapping either. I ended up updating the text_query function to this:

def text_query( dataset: Dataset, schema: Schema, query: str, filters: FilterDict = {}, fuzzy: bool = False, ): if not len(query.strip()): should = {"match_all": {}} elif fuzzy and query.find('~') == -1: should = { "match": { "text": { "query": query, "fuzziness": "AUTO", "lenient": True, "operator":"AND" } } } else: should = { "query_string": { "query": query, "fields": ["names^3", "text"], "default_operator": "and", } } return filter_query([should], dataset=dataset, schema=schema, filters=filters)

The reason for this line fuzzy and query.find('~') == -1 is to not mix fuzziness and ~ operator. If query contains ~, the fuzzy parameter is just ignored

@pudo any comment on this ?

I can open a pull request if needed
opened by skrafft 6
index ready time

Hello

Running on a 8core 8gb ram system... since 30min ago

curl http://localhost:8000/readyz {"detail":"Index not ready."}

anyone knows aproximatley how much time it takes to finish it?

Thank you

opened by AndreiD 4
Is there a way to use /data/datasets/index.json instead of https://data.opensanctions.org/datasets/latest/index.json?
In manifest.yml am I on the right track to use the local /data/datasets/ generated by opensanctions/opensanctions/ instead of the index available at https://data.opensanctions.org/datasets/latest/index.json?

Something like this: /app/manifests/manifest.yml (??)

schedule: "*/30 * * * *" catalogs: - path: /data/datasets/index.json scope: all

When I try this nothing seems to happen when running yente. After looking at manifest.py it seems that url: is required here. If I use the default configuration it works and populates elasticsearch but not with the custom one above. With manifests.yml above it just starts and sits there with no fetching/indexing.

TLDR; I guess what I'm asking is how does one use the local datasets/ created by a locally running https://github.com/opensanctions/opensanctions instead of fetching all the data from OpenSanctions.org?

I'm running it like this (docker swarm):

yente: image: ghcr.io/opensanctions/yente:latest environment: YENTE_ENDPOINT_URL: https://<url> YENTE_MANIFEST: /app/manifests/manifest.yml YENTE_ELASTICSEARCH_URL: http://elasticsearch:9200 YENTE_STATEMENT_API: "false" YENTE_UPDATE_TOKEN: <randomstuff> volumes: - /mnt/gfs/OpenSanctions/data:/data - /mnt/gfs/OpenSanctions/manifest.yml:/app/manifests/manifest.yml networks: - traefik_public deploy: mode: replicated replicas: 1 placement: constraints: [node.role == manager] restart_policy: condition: on-failure labels: - ...
opened by LandonStewart 2
Bump nomenklatura from 2.5.2 to 2.5.3
Bumps nomenklatura from 2.5.2 to 2.5.3.

Commits

47f08fb Bump version: 2.5.2 → 2.5.3

be6707f Adopt upstream helper

f0e8e03 add a senzing exporter

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 2
Bump followthemoney from 2.9.6 to 2.9.7
Bumps followthemoney from 2.9.6 to 2.9.7.

Commits

f6ad4dd Bump version: 2.9.6 → 2.9.7

df33089 Update files in preperation for a new FtM release

51cf89f upgrade countrynames to restore python 3.8 compatibility

09ed21e Merge pull request #716 from alephdata/translations_followthemoney-translatio...

6797556 Bump rollup-plugin-ts from 3.0.1 to 3.0.2 in /js (#725)

b5fb4b2 Bump @typescript-eslint/eslint-plugin from 5.27.0 to 5.27.1 in /js (#723)

5bd2b52 Bump lint-staged from 12.4.3 to 13.0.1 in /js (#722)

389773d Bump @types/node from 17.0.40 to 17.0.42 in /js (#724)

6bd3a13 Bump jest-config from 28.1.0 to 28.1.1 in /js (#721)

59858f2 Bump @typescript-eslint/parser from 5.27.0 to 5.27.1 in /js (#720)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 2
Allow Aiohttp to use proxy

We have to use a proxy within our organisation to connect to data.opensanctions.org. For most Lunix applications it suffices to set the environmental variables HTTP_PROXY and HTTPS_PROXY but Aiohttp requires the flag 'trust_env=True' to read from environmental variables; https://docs.aiohttp.org/en/stable/client_advanced.html - Proxy support.

For example: async with aiohttp.ClientSession(trust_env=True) as session: async with session.get("http://python.org") as resp: print(resp.status)

Could this, or another way of specifying a proxy, be added to Yente?

opened by hotdog809 2
Bump nomenklatura from 2.2.8 to 2.3.0
Bumps nomenklatura from 2.2.8 to 2.3.0.

Commits

2981e17 Bump version: 2.2.8 → 2.3.0

2f8f549 upgrade scikit-learn

c49d181 Copy in the other enrichers

8435c90 pre-historic ftm-enrich code

87e1651 introduce a URL normalizer utility function

20a917e Working yente MVP

f37f320 requests type stubs

121ab7b Sketch out enrichment system

203a78c Fix typing

95917c8 Database backed cache with sqlite, code from OpenSanctions

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot will merge this PR once CI passes on it, as requested by @pudo.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 2
Bump aiocsv from 1.2.1 to 1.2.2
Bumps aiocsv from 1.2.1 to 1.2.2.

Commits

947b72b 1.2.2 version bump

d9964d8 Remove 3.10 from CI, cause it broken

d01ab3c Travis: run on latest Python 3.7 and 3.10?

75ab895 Ensure latest setuptools and wheel run on CI

d0190c4 Add typing data according to PEP561

488c70f Update documentation for 1.2

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 2
Allow indexing/including unrelated datasets
We want to be able to load offshoreleaks and other stuff like this into the default collection at run-time. Probably means detaching our definition of the datasets from the index.json spec a bit at some point.

In order to do this, I want to introduce a manifest.yml to describe all the datasets in the system. This would a) reference the OpenSanctions index and how often to fetch that, b) be able to add more sources that are not part of OpenSanctions.

Here's a proposed format:

opensanctions: index: https://data.opensanctions.org/datasets/latest/index.json scope: default schedule: "*/30 * * * *" sources: icij_offshoreleaks: title: ICIJ OffshoreLeaks entities_url: https://data.opensanctions.org/contrib/icij-offshoreleaks/full-oldb.json schedule: null collections: - all - offshore local_dataset1: title: My local fraudsters schedule: "* 30 1 * * *" # Apply an FtM namespace: namespace: true collections: - all - fraud queries: csv_url: file:///home/pudo/data/fraudsters.csv entities: (see https://docs.alephdata.org/developers/mappings)

This would have the following effects:

a) Load all OpenSanctions data inside the default dataset, checking for updates every 30 minutes b) Load the ICIJ OffshoreLeaks database once and include those entities in search results for the collections all and offshore. c) Generate FtM objects from a local CSV file and load those entities into a new dataset once per night.
enhancement
opened by pudo 2
Bump nomenklatura from 2.7.5 to 2.7.7
Bumps nomenklatura from 2.7.5 to 2.7.7.

Commits

f9396f5 Bump version: 2.7.6 → 2.7.7

c71d3c9 slim down objects before sending them to yente

578d806 Bump version: 2.7.5 → 2.7.6

2b2cc6b properly implement yente api keys

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 1
Bump elasticsearch[async] from 8.5.2 to 8.5.3
Bumps elasticsearch[async] from 8.5.2 to 8.5.3.

Release notes

Sourced from elasticsearch[async]'s releases.

8.5.3

Client is compatible with Elasticsearch 8.5.3

Commits

14fcea0 Bumps 8.5 to 8.5.3

adc3f00 [8.5] add release notes for 8.5.1

2b3d6c5 [8.5] Upgrade actions/checkout to v3

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 1
Explore migrating to opensearch-py

They're having a little shit-flinging battle on the backs of every open source project using their products:

https://github.com/spring-projects/spring-data-elasticsearch/issues/1880

Looks like OpenSearch-py will continue to work with ElasticSearch, but probably trail ES by a few versions. Need to explore how up-to-date its async support is.
help wanted dependencies python

opened by pudo 0
Re-instate deep nesting tests

After migrating OpenSanctions to use externals, the wd_curated dataset ended up being empty. That's what all tests for yente were written against. So I've now switched it over to eu_fsf, but eu_fsf doesn't have family ties. Need to find that somewhere to test deep nesting of entities again.

opened by pudo 0

Releases(v3.1.2)

v3.1.2(Dec 31, 2022)

This release includes numerous bug fixes related to the stability of the software, and fixes one backend issue with the matching API.

Full Changelog: https://github.com/opensanctions/yente/compare/v3.0.0..v3.1.2
Source code(tar.gz)
Source code(zip)
v3.0.0(Dec 9, 2022)
This release revamps the way we handle dataset metadata in yente in order to make it easier to load and update non-OpenSanctions datasets. It is a breaking change because it modifies the format of the manifest file (manifest.yml). If you're running yente with the default manifest file, the changes will not affect you.

Updating your manifest.yml

In order to update your custom manifest for yente 3.0.0, you need to change the external section into a catalogs section like this:

schedule: null # old: external: url: "https://data.opensanctions.org/datasets/latest/index.json" scope: all # new: catalogs: - url: "https://data.opensanctions.org/datasets/latest/index.json" scope: all resource_name: entities.ftm.json datasets: []

The cool thing about this is that you can now add extra catalogs. Try putting in https://data.opensanctions.org/contrib/graph/catalog.json and it'll index a bunch of non-OpenSanctions data :)

What's Changed

Revamp metadata architecture by @pudo in https://github.com/opensanctions/yente/pull/188

Bump structlog from 22.1.0 to 22.3.0 by @dependabot in https://github.com/opensanctions/yente/pull/184

Bump orjson from 3.8.2 to 3.8.3 by @dependabot in https://github.com/opensanctions/yente/pull/189

Bump nomenklatura from 2.7.4 to 2.7.5 by @dependabot in https://github.com/opensanctions/yente/pull/191

Bump elasticsearch[async] from 8.5.2 to 8.5.3 by @dependabot in https://github.com/opensanctions/yente/pull/192

Full Changelog: https://github.com/opensanctions/yente/compare/v2.2.1...v3.0.0
Source code(tar.gz)
Source code(zip)
v2.2.0(Nov 3, 2022)
This release includes an updated/improved entity matching model and will produce different scores than the previous release.

What's Changed

Bump fastapi from 0.85.0 to 0.85.1 by @dependabot in https://github.com/opensanctions/yente/pull/158

Bump orjson from 3.8.0 to 3.8.1 by @dependabot in https://github.com/opensanctions/yente/pull/160

Bump aiocsv from 1.2.2 to 1.2.3 by @dependabot in https://github.com/opensanctions/yente/pull/161

Bump followthemoney from 3.0.8 to 3.0.10 by @dependabot in https://github.com/opensanctions/yente/pull/166

Bump elasticsearch[async] from 8.4.3 to 8.5.0 by @dependabot in https://github.com/opensanctions/yente/pull/167

Full Changelog: https://github.com/opensanctions/yente/compare/v2.1.0...v2.2.0
Source code(tar.gz)
Source code(zip)
v2.1.0(Oct 14, 2022)

This release changes the way that redirects for merged entities are performed. This should lead to a smaller index size and faster indexing.
Source code(tar.gz)
Source code(zip)
v2.0.0(Oct 14, 2022)

Yente 2.0 removes the statement API (used only to power the web site) and simplifies the overall codebase. It also significantly improves the documentation of the request endpoints in the OpenAPI docs.
Source code(tar.gz)
Source code(zip)
v1.8.0(Jul 11, 2022)
This release includes the following:

Stability improvements to the API and indexer, improving the performance and stability especially for large indexing jobs (30GB+).

Updated scoring model for the matching API, including new model features and better precision.

Updated data model (FollowTheMoney 3.0), with Address entities becoming first-class search targets.

Various dependency updates and performance improvements.

Source code(tar.gz)
Source code(zip)