ServiceX_DID_Finder_Girder
Access datasets for ServiceX from yt Hub
Finding datasets
This DID finder is designed to take a collection id (https://girder.hub.yt/#collections) as the did in ServiceX and yields the download url's for any folders found within the collection. If the collection has no folders, the download url for the collection is passed to the ServiceX transformer.
Build Image
Build the docker image as:
% docker build -t servicex-did-finder-girder .
Deploying the DID Finder
You'll need to create a k8 deployment file in order to run this DID finder. Here is a sample, built to be part of the ServiceX distribution:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-did-finder-girder
spec:
replicas: 1
selector:
matchLabels:
app: {{ .Release.Name }}-did-finder-girder
template:
metadata:
labels:
app: {{ .Release.Name }}-did-finder-girder
spec:
containers:
- name: {{ .Release.Name }}-did-finder-girder
image: {{ .Values.didFinderGirder.image }}:{{ .Values.didFinderGirder.tag }}
imagePullPolicy: {{ .Values.didFinderGirder.pullPolicy }}
env:
- name: INSTANCE_NAME
value: {{ .Release.Name }}
args:
- --rabbit-uri
- amqp://user:{{ .Values.rabbitmq.auth.password }}@{{ .Release.Name }}-rabbitmq:5672
The last argument to --rabbit-uri is perhaps the most crucial - it defines the rabbit queue this DID finder listens on.
Testing Locally
Once deployed locally, you can test the DID Finder by runnign the following command within the tests directory:
python3 post.py ${servicex_port} yt.json
You will see the urls being returned from the did finder in logs of the did finder container.
3 Oct 4, 2022
1 Jan 1, 2022
71 Dec 29, 2022
7 Aug 5, 2022
32 Jan 19, 2022
4 May 4, 2022
17 Oct 21, 2022
10 Sep 11, 2021
130 Dec 20, 2022
3 Sep 25, 2022
5 Sep 24, 2021
23 Oct 9, 2022
29 Dec 4, 2022
1 Nov 11, 2021
2 Nov 30, 2021
2 Nov 1, 2021
7 Aug 5, 2022
17 Oct 21, 2022
2 Apr 7, 2022
18 Jul 31, 2022
9 Dec 27, 2022
1 Feb 22, 2022
22 Nov 9, 2022