Concept set can be created (edit: Joe: In the future when they have added this feature) via a bulk import process and by pass the concept set editor for importing a bulk of externally authorized concept sets. Specify detail specification to handle this process.

Tables needed to generate (edit: Joe: Added links):

• [x] 1. concept set container: (concept_set_container_edited table)
• [x] 2. concept set version: (code_sets table)
• [x] 3. concept set metadata: (concept_set_version_item_rv_edited table)

More details will be uploaded. This is a place holder issues for Joe and Steph.

Description

(@stephanieshong Please feel free to edit this and change this text if I got anything wrong.) There are some edge cases where we have value sets with the same name. And sometimes there is a single OID which represents a grouping of those value sets. And, in a smaller subset of those edge cases, there is no such OID / value set on UMLS.

In those cases, we should create a new grouping value set:

• Create to be uploaded in enclave. We don't need to make in UMLS.
• Will not have an OID
• Members of the concept set should be all the members of all of the value sets with the same name.
• Provenance should include the OIDs of those value sets.

Tasks

• [x] 1. Update our code to revert the changes we made (the appending of code systems to the value set names).
• [ ] 2. Handle collision cases
  • [ ] 2.a. Use existing grouping sets that exist.
  • [ ] 2.b. Handle cases where there isn't an existing grouping set.

Task 2 Options

Option (d) seems to be the best, based on our discussion at the 2022/02/17 BIDS data meeting.

~a. Merge the value sets~

I think this assumes that the value sets that would be merged are of entirely different code systems. I suppose we can work with that assumption for now, but if that isn't the case, we'll need to think about what to do. This one seems preferred.

~b. Give the value sets different names~

This is what we've already done. We decided to append code system names to the value set. But it seems this is not preferred.

~c. Only upload 1 of the value sets~

I this is the case, we'd need to store information about the value set collisions to some data storage layer (file or DB). Then wed need a curation process to determine which set is best.

d. Create a new set (when grouping not available)

1. Optional: Create new set in VSAC and get OID: https://www.nlm.nih.gov/vsac/support/authorguidelines/createvs.html (edit: We decided this doesn't need to be done)
2. Optional: Add new OID to cset.csv (only if we do (1))
3. Update code to programmatically create new sets in these collision cases where no grouping already exists.
4. Upload to enclave
5. Optional: Register IDs? (options: a, b) (only if we do (1))

• Davera mentioned this, but I'm not sure it's necessary, as VSAC autogenerates new OIDs (step 1 above). HL7 also charges $500 to register an OID.

e. Use previously created combination value sets / OIDs (when grouping set available)

A "grouping set" is when a value set is defined as a "grouping" in UMLS VSAC, and its members are OIDs that point to its composite value sets.

Apparently, for some (or perhaps all?) instances of value sets which have more than 1 instance in VSAC, there exist other value sets which have grouped these multiple instances together. I believe this is mainly to account for different code systems. So, for example, there may be a value sets for blood transfusion. There may be a set that is (i) in one code system (e.g. SNOMED) and another that uses (ii) a different code system (e.g. ICD10CM). Then, there may be a (iii) 3rd value set which combines the codes with both. In this case, what we we want to do is to use and upload (iii) to the enclave, and not upload (i) and (ii).

We should use this option (e) as much as possible, and only use option (d) (creating a new value set) if VSAC doesn't already have such a grouped value set.

Description

Allow vsac_wrangler to fetch from the Lisa2 VSAC GRAVITY sheet in this GoogleSheets spreadsheet: https://docs.google.com/spreadsheets/d/1jzGrVELQz5L4B_-DqPflPIcpBaTfJOUTrVJT5nS_j18/edit#gid=1272275514

Related

#42

Oh, @DaveraGabriel, you're talking about VSAC access, not JHU OMOP access. We should move that to a different issue. I already made the mistake of confusing the two in this thread.

Originally posted by @Sigfried in https://github.com/HOT-Ecosystem/ValueSet-Tools/issues/14#issuecomment-1011233079

I'm getting a new error while trying to run enclave_wrangler for Lisa6, which has a single oid. I put the error message in the commit comment for https://github.com/HOT-Ecosystem/ValueSet-Tools/commit/fdfe3cfc0e47a2ba0e662797857e6ce7c4e8a213

Overview

We want front-end experience to be fast, and we don't want to make repeated, unnecessary requests.

Options

Can do one or all of the following

• [ ] 1. Global state for a given user session
• [ ] 2. Browser caching (for multiple user sessions on same machine)

Getting errors with codeset_id 419757429, which is currently included in the Example Comparison set of cset ids. Need to figure out why. This is occurring in both the develop branch and the db branch. I don't remember it being an issue before we switched to working on the db branch, so maybe the problem is arising from some change in the data rather than a problem with code. Or maybe if we go to an older commit on the development branch the problem will disappear.

Updates

    Backend
    - Finished off hierarchy portion of route cr-hierarchy
    - Added functions for building hierarchy to utils
    - Misc: Codestyle updates

User experience & code flow description

Updating an existing cset

1. Frontend: User clicks on header / button to edit concept set.
2. Frontend: reacts to click ~a. Frontend: They get a page which shows: (i) list of concept names~ (can do this in the future) b. Frontend: A new column.
3. Frontend: The label should say (draft) instead of (v#). The user can check/uncheck the concepts they want / don't want
4. Frontend: They click a button to 'commit changes' a. Will need their user RID in order to set the created_by fields for concept_set_container and code_set b. What other metadata do we need (for MVP)?
5. Backend: Hit route (new route?)
6. Backend: Persist changes (multiple files? which files?) a. Persist by writing to prepped files? yes...and maybe use git diff and patch
7. Backend: Push to enclave?
8. Frontend: User sees confirmation message that commit succeeded?

Creating a new cset

TODO

Concerns

1. How to handle production server updates to termhub-csets vs local deveopment?

• Maybe have them commit to different branches, e.g. if env variable is production, commit to main, else develop.

Having problems with missing concepts and missing links between concepts... can explain later.

What I think we should do is expand the subset of concepts in prepped files to: all of the concepts appear in the concept_ancestor table where either the ancestor_concept_id or the descendant_concept_id is included in the concept_set_members table. Right

Updates

WIP:
- Update: remove "Unnamed: x" columns from datasets
- Update: move jupyter transform code over to enclave_wrangler
- Update: datasets: filter concept sets that have no container

Overview

Check token validity early and often.

Sub-tasks

• [x] 1. If enclave_wrangler request fails, check if token is dead, and print err about that if so.
• [x] 2. If token will expire soon (e.g. 2 weeks), print warning.
• [ ] 3. Check on server start
• [ ] 4. Optional: Check on schedule (e.g. daily) (harder)

Additional details

Here's how:

➜ curl -XGET https://unite.nih.gov/multipass/api/me -H "Authorization: Bearer $PALANTIR_ENCLAVE_AUTHENTICATION_BEARER_TOKEN"
{
  "id": "6387db50-9f12-48d2-b7dc-e8e88fdf51e3",
  "username": "[email protected]",
  "attributes": {
    "multipass:organization": [
      "NIH"
    ],
    "multipass:email:primary": [
      "[email protected]"
    ],
    "multipass:organization-rid": [
      "ri.multipass..organization.73f45502-dee1-46e9-ab49-64a738b13971"
    ],
    "upn": [
      "[email protected]"
    ],
    "multipass:realm": [
      "nih-adfs"
    ],
    "multipass:realm-name": [
      "NIH Auth"
    ]
  }
}

➜ curl -XGET https://unite.nih.gov/multipass/api/token/ttl -H "Authorization: Bearer $PALANTIR_ENCLAVE_AUTHENTICATION_BEARER_TOKEN"
12654423

That last number is time-to-live in seconds.

I'm not sure where to run these checks. When the time is getting close, we need to ask Mariam Deacy to generate a new one for us.

Overview

We would like to allow users to log in using NCATS unified authentication (example), or something like it.

We met w/ the JHU cloud infra team today, and Patrick Le mentioned to us that Hopkins is part of some federated network. It uses "shibboleth" along w/ SAML. There is a different team that can help us work through this, but this would require some programming on our end.

Overview

Set up RDBMS to serve data to the REST API, rather than loading datasets as global variables. Hopefully PostgreSQL. But given that the JHU hosting team cannot provide special services for that, we may consider another option.

Subtask list

• [x] 1. Complete local setup
• [ ] 2. Setup documentation
• [ ] 3. Complete setup on server, and update documentation if changes
• [x] 4. Optimize schema: e.g. TEXT --> VARCHAR(n)
• [x] 5. Load dataset data
• [ ] 6. Create pipeline for loading/refreshing dataset data
• [ ] #186
• [ ] 8. Build functions or whatever to populate mySQL tables with data from N3C ontology API. (Could we just do dataset data once and then keep up-to-date using ontology API?)
• [ ] #187

Sub-tasks

• [ ] 1. Backend (@joeflack4)
• [ ] 2. Frontend (@sigfried)
• [ ] 3. Allow for other users (@amin)

Sub-task details

1. Backend

• [x] enclave_wrangler functions
• [ ] Unit tests (still need completion w/ teardowns)
• [x] app.py routes (Joe: I think they're done)

2. Frontend

Pending backend.

3. Allow for other users

Amin needs to change permissions so that other users, using their TOKENs, I believe, can make calls.