Fetch data from: https://public.tableau.com/profile/lee.mendoza#!/vizhome/pharmacies_desktop/Pharmacies_desktop

Put your script in a file named: la/tableau/fetch.py (or fetch.sh, etc.)

Store it (without processing) in a new file created in the directory passed as the first argument (sys.argv[1]).

Check the wiki to learn more about the purpose of the fetch stage and how to get set up for development!

Tips

1. While working on your code, run it at any point:

   poetry run vaccine-feed-ingest fetch la/tableau
   

| Key Details | |-| | Resolves #244 | | Resolves #245 | | Resolves #246 | | State: wy | | Site: wyo_appt_portal |

Data sample

{"id": "wyo_appt_portal:129", "name": "Museum of the Mountain Man", "address": {"street1": "Museum of the Mountain Man", "street2": null, "city": "Pinedale", "state": "WY", "zip": "82941"}, "location": null, "contact": [{"contact_type": "booking", "phone": null, "website": "https://www.wyoapptportal.org/client/registration?clinic_id=129", "email": null, "other": null}], "languages": null, "opening_dates": [{"opens": "2021-05-27", "closes": "2021-05-27"}], "opening_hours": [{"day": "Thursday", "open": "14:00", "closes": "15:30"}], "availability": {"drop_in": null, "appointments": true}, "inventory": [{"vaccine": "moderna", "supply_level": null}], "access": null, "parent_organization": null, "links": null, "notes": ["This is a drive-thru clinic at the Museum of the Mountain Man.  Please wear a mask and a short sleeve shirt.", "If you are signing up for a second dose, you must get the same vaccine brand as your first dose."], "active": null, "source": {"source": "wyo_appt_portal", "id": "Museum of the Mountain Man", "fetched_from_uri": "https://www.wyoapptportal.org/clinic/search", "fetched_at": "2021-04-28T05:45:16.759080", "published_at": null, "data": {"name": "Museum of the Mountain Man", "date": "05/27/2021", "address": "Museum of the Mountain Man, Pinedale WY, 82941", "vaccines": "Moderna COVID-19 Vaccine", "ages": "N/A", "info": "This is a drive-thru clinic at the Museum of the Mountain Man.  Please wear a mask and a short sleeve shirt.", "hours": "02:00 pm - 03:30 pm", "available": "60", "special": "If you are signing up for a second dose, you must get the same vaccine brand as your first dose.", "clinic_id": "129"}}}

Verified output of ak/arkgis, wa/prepmod, and wy/wyo_appt_portal

Before Opening a PR

• [*] I tested this using the CLI (e.g., poetry run vaccine-feed-ingest <state>/<site>)
• [*] I ran auto-formatting: poetry run tox -e lint-fix

| Key Details | |-| | Addresses #254 | State: | Site: |

Notes

This PR uses a sample CSV provided on #254 that contains about 10K rows of parsed provider info to improve coverage of providers while extracting a provider id and name from data sources. Specifically, that CSV was sorted alphabetically, and support was added for providers in that sorted list whose names start from A through K.

This is the first in a series of planned steps to improve coverage. Upcoming plans:

1. Add the newly identified providers to the schema repo
2. Support other providers (L through Z) from the provided CSV.
3. Add a simple helper script that shows a measure of coverage by applying this script to a specified input file. Basically, it will spit out how many of the input rows had a matching provider name + id extracted. This helper can be used to track improvements to coverage over time.
4. [Optimization] Rearrange the regexes in provider_id_from_name() in the order of expected national coverage, so that a national chain like COSTCO or WALGREENS with heavy coverage can be matched first, and the function can then exit early.

Update

This PR also updates the provider_id_from_name() method to return a VaccineProvider (previously str).

Data sample

As a sanity check, I ran:

poetry run vaccine-feed-ingest all-stages ok/vaccinate_gov
poetry run vaccine-feed-ingest all-stages ca/sf_gov

and examined the normalized outputs.

Before Opening a PR

• [X] I tested this using the CLI (e.g., poetry run vaccine-feed-ingest <state>/<site>)
• [X] I ran auto-formatting: poetry run tox -e lint-fix

Blocked by #401

Normalize existing .ndjson records into the Vaccinate The States data schema.

Read all files in the directory passed as the second argument (sys.argv[2]), transform each record into the normalized schema (filling in as much detail as possible), then output them to new files in the directory passed as the first argument (sys.argv[1]).

Check the wiki to learn more about the purpose of the normalize stage and how to get set up for development!

Tips

1. Fetch and parse data for this site before you start developing:

   poetry run vaccine-feed-ingest fetch dc/district
   poetry run vaccine-feed-ingest parse dc/district
   

2. While working on your code, run it at any point:

   poetry run vaccine-feed-ingest normalize dc/district
   

Example

Parsed files for ak/arcgis are converted to normalized files

Fixes #93

Two questions:

1. There wasn't a natural ID in the fetched data, so I used the name of the location. My IDs look like this: immunizenevada_org:Albertsons #1059. Is this acceptable?

2. For the fetched_from_uri value I used the URL for the NV vaccine locator: https://www.immunizenevada.org/covid-19-vaccine-locator. That's not strictly accurate, since the data was actually fetched by POSTing to https://www.immunizenevada.org/views/ajax, but if you GET that url, you get a 404. That seems less helpful, despite being technically accurate. Thoughts?

| Key Details | |-| | Resolves #255 | State: | Site: |

Notes

NA

Data sample

Ran all-stages for ak/arcgisvia:

 poetry run vaccine-feed-ingest all-stages ak/arcgis

and examined the first line of normalization output via:

head -1 out/ak/arcgis/normalized/2021-04-28T22\:44\:45/d92cbd6ff2524d7e92bef109f30cb366_0.parsed.normalized.ndjson

{"id": "ak:arcgis:d92cbd6ff2524d7e92bef109f30cb366_0:85b07ac7-c9b6-4d09-83ec-a720998c2fbf", "name": "North Slope Borough Public Health Nursing-Wellness", "address": {"street1": "579 Kongosak St", "street2": null, "city": "Barrow", "state": "AK", "zip": "99723"}, "location": {"latitude": 71.29069998454801, "longitude": -156.7920099649541}, "contact": [{"contact_type": null, "phone": "(907) 852-0270", "website": null, "email": null, "other": null}], "languages": null, "opening_dates": null, "opening_hours": null, "availability": {"drop_in": null, "appointments": true}, "inventory": [{"vaccine": "pfizer_biontech", "supply_level": null}], "access": null, "parent_organization": null, "links": null, "notes": ["NSB PHN Wellness clinic, please call for appointments 907-852-0270"], "active": null, "source": {"source": "ak:arcgis", "id": "85b07ac7-c9b6-4d09-83ec-a720998c2fbf", "fetched_from_uri": "https://services1.arcgis.com/WzFsmainVTuD5KML/ArcGIS/rest/services/COVID19_Vaccine_Site_Survey_API/FeatureServer/0", "fetched_at": "2021-04-29T05:44:55.676901", "published_at": "2020-12-25T13:00:00", "data": {"attributes": {"address": "579 Kongosak St", "city": "Barrow", "cost_other": null, "datesubmited": 1608930000000, "flu_acceptsInsurance": "yes_please_contact_your_insuran", "flu_cost": "no_cost", "flu_vaccinations": "pfizer", "flu_walkins": "no_please_make_an_appointment", "globalid": "85b07ac7-c9b6-4d09-83ec-a720998c2fbf", "needs_review": "approved", "objectid": 9, "phone": "907-852-0270", "publicEmail": null, "publicNotes": "NSB PHN Wellness clinic, please call for appointments 907-852-0270", "publicWebsite": null, "vaccinationSite": "North Slope Borough Public Health Nursing-Wellness", "zipcode": "99723"}, "geometry": {"spatialReference": {"latestWkid": 4326, "wkid": 4326}, "x": -156.7920099649541, "y": 71.29069998454801}}}}

Before Opening a PR

• [x] I tested this using the CLI (e.g., poetry run vaccine-feed-ingest <state>/<site>)
• [x] I ran auto-formatting: poetry run tox -e lint-fix

We have a function (provider_id_from_name()) that tries to parse retail pharmacy store numbers from a string with the location's name (e.g., "Walgreens #7150 -> 7150).

We want to support this on every chain of retail pharmacies that we know about.

normalize sfsites for il

| Key Details | |-| | Resolves #215 | State: il | Site: sfsites |

Notes

Here's an example of the parsed data:

{
  "Id": "a05t00000096y7sAAA",
  "Geolocation__Latitude__s": 42.37019,
  "Geolocation__Longitude__s": -88.83004,
  "Testing_Center__c": "4-29 - Boone County - Fire District 3 - First Dose",
  "Website__c": "https://covidvaccination.dph.illinois.gov/",
  "Postal_Code__c": "61065",
  "Street__c": "305 West Grove St",
  "City__c": "Poplar Grove",
  "Location_Type__c": "Public health provider - public health clinic",
  "County__c": "Boone"
}

Couple questions:

1. ~There's a statewide.parsed.ndjson, which specifies locations that allow vaccinations for anyone in the state of IL, but there wasn't a clear attribute in the Normalize schema to denote this. Anyone know of an alternative? Maybe notes?~ Added something to the notes, per PR convo
2. ~I filter out all locations that point to https://covidvaccination.dph.illinois.gov, because those will likely be dupes from il/juvare. Is that the right choice, or are dupes ok?~ Left in the dupes
3. ~Some of these are Walgreens/Costco locations, but no specific identifiers so I can't use the authority field. Should I still fill out the parent_organization?~ Added the parent_organization

Normalized Data sample

{
  "id": "sfsites:a05t000000972VtAAI",
  "name": "LIVINGSTON COUNTY HEALTH DEPARTMENT (Moderna)",
  "address": {
    "street1": "310 E TORRANCE AVE",
    "street2": null,
    "city": "PONTIAC",
    "state": "IL",
    "zip": "61764"
  },
  "location": {
    "latitude": 40.8663711547852,
    "longitude": -88.6266403198242
  },
  "contact": [
    {
      "contact_type": "booking",
      "phone": null,
      "website": "https://lchd.us/site/covid-19-vaccination/",
      "email": null,
      "other": null
    }
  ],
  "languages": null,
  "opening_dates": null,
  "opening_hours": null,
  "availability": {
    "drop_in": null,
    "appointments": true
  },
  "inventory": null,
  "access": null,
  "parent_organization": null,
  "links": null,
  "notes": [
    "Public health provider - public health clinic"
  ],
  "active": null,
  "source": {
    "source": "sfsites",
    "id": "a05t000000972VtAAI",
    "fetched_from_uri": "https://coronavirus.illinois.gov/s/vaccination-location",
    "fetched_at": "2021-05-04T03:24:10.938829",
    "published_at": null,
    "data": {
      "Id": "a05t000000972VtAAI",
      "Geolocation__Latitude__s": 40.8663711547852,
      "Geolocation__Longitude__s": -88.6266403198242,
      "Testing_Center__c": "LIVINGSTON COUNTY HEALTH DEPARTMENT (Moderna)",
      "Website__c": "Livingston   https://lchd.us/site/covid-19-vaccination/",
      "Postal_Code__c": "61764",
      "Street__c": "310 E TORRANCE AVE",
      "City__c": "PONTIAC",
      "Location_Type__c": "Public health provider - public health clinic",
      "County__c": "LIVINGSTON"
    }
  }
}
{
  "id": "sfsites:a05t000000972YGAAY",
  "name": "VNA Health Care",
  "address": {
    "street1": "801 Villa Street",
    "street2": null,
    "city": "Elgin",
    "state": "IL",
    "zip": "60120"
  },
  "location": {
    "latitude": 42.027997,
    "longitude": -88.260293
  },
  "contact": [
    {
      "contact_type": "booking",
      "phone": null,
      "website": "https://www.vnahealth.com/covid-vaccine/",
      "email": null,
      "other": null
    }
  ],
  "languages": null,
  "opening_dates": null,
  "opening_hours": null,
  "availability": {
    "drop_in": null,
    "appointments": true
  },
  "inventory": null,
  "access": null,
  "parent_organization": null,
  "links": null,
  "notes": [
    "FQHC"
  ],
  "active": null,
  "source": {
    "source": "sfsites",
    "id": "a05t000000972YGAAY",
    "fetched_from_uri": "https://coronavirus.illinois.gov/s/vaccination-location",
    "fetched_at": "2021-05-04T03:24:10.938829",
    "published_at": null,
    "data": {
      "Id": "a05t000000972YGAAY",
      "Geolocation__Latitude__s": 42.027997,
      "Geolocation__Longitude__s": -88.260293,
      "Testing_Center__c": "VNA Health Care",
      "Website__c": "https://www.vnahealth.com/covid-vaccine/",
      "Postal_Code__c": "60120",
      "Street__c": "801 Villa Street",
      "City__c": "Elgin",
      "Location_Type__c": "FQHC",
      "County__c": "KANE"
    }
  }
}
{
  "id": "sfsites:a05t000000972ffAAA",
  "name": "Walgreens",
  "address": {
    "street1": "345 N LAKE ST",
    "street2": null,
    "city": "MUNDELEIN",
    "state": "IL",
    "zip": "60060"
  },
  "location": {
    "latitude": 42.2670565,
    "longitude": -88.0040094
  },
  "contact": [
    {
      "contact_type": "booking",
      "phone": null,
      "website": "https://www.walgreens.com/findcare/vaccination/covid-19",
      "email": null,
      "other": null
    }
  ],
  "languages": null,
  "opening_dates": null,
  "opening_hours": null,
  "availability": {
    "drop_in": null,
    "appointments": true
  },
  "inventory": null,
  "access": null,
  "parent_organization": {
    "id": "walgreens",
    "name": null
  },
  "links": null,
  "notes": [
    "Pharmacy"
  ],
  "active": null,
  "source": {
    "source": "sfsites",
    "id": "a05t000000972ffAAA",
    "fetched_from_uri": "https://coronavirus.illinois.gov/s/vaccination-location",
    "fetched_at": "2021-05-04T03:24:10.938829",
    "published_at": null,
    "data": {
      "Id": "a05t000000972ffAAA",
      "Geolocation__Latitude__s": 42.2670565,
      "Geolocation__Longitude__s": -88.0040094,
      "Testing_Center__c": "Walgreens",
      "Website__c": "https://www.walgreens.com/findcare/vaccination/covid-19",
      "Postal_Code__c": "60060",
      "Street__c": "345 N LAKE ST",
      "City__c": "MUNDELEIN",
      "Location_Type__c": "Pharmacy",
      "County__c": "LAKE"
    }
  }
}

Before Opening a PR

• [x] I tested this using the CLI (e.g., poetry run vaccine-feed-ingest normalize il/sfsites)
• [x] I ran auto-formatting: poetry run tox -e lint-fix

| Key Details | |-| | Resolves #94 State: LA | Site: Louisiana Department of Health Tableau |

Notes

Kudos to @krouskop and @nuchi for figuring out how to extract JSON from the site.

Data sample

3339663;{"sheetName":"Pharmacies_desktop","layoutId":"10601967672443134028","newSessionId":"48AA9CEF3D1C40B288695149E8FDC864-0:0","isDeferredBootstrap":true,"workbookLocale":"en_US","allowSubscriptions":true,"allowSubscribeOnDataPresent":true,"repositoryUrl":"pharmacies_desktop/Pharmacies_desktop","worldUpdate":{"applicationPresModel":{"renderMode": "render-mode-client","workbookPresModel":{"dashboardPresModel":{"sheetPath":{"sheetName": "Pharmacies_desktop","isDashboard": true},

Before Opening a PR

• [X] I tested this using the CLI (e.g., poetry run vaccine-feed-ingest <state>/<site>)
• [X] I ran auto-formatting: poetry run tox -e lint-fix

Normalize existing .ndjson records into the Vaccinate The States data schema.

Read all files in the directory passed as the second argument (sys.argv[2]), transform each record into the normalized schema (filling in as much detail as possible), then output them to new files in the directory passed as the first argument (sys.argv[1]).

Check the wiki to learn more about the purpose of the normalize stage and how to get set up for development!

Tips

1. Fetch and parse data for this site before you start developing:

   poetry run vaccine-feed-ingest fetch md/arcgis
   poetry run vaccine-feed-ingest parse md/arcgis
   

2. While working on your code, run it at any point:

   poetry run vaccine-feed-ingest normalize md/arcgis
   

Example

Parsed files for ak/arcgis are converted to normalized files

per https://discord.com/channels/799147121357881364/813861006718926848/859812472428822608

it appears that one, and possibly more (pending verification) of the prepmod scrapers do not have stable identifiers

Example: https://vial.calltheshots.us/admin/core/location/53976/change/