First step towards addressing #25.

This moves all path operation functions out of RestAccessor and creates instead fastapi.APIRouter instances in a new routers sub-package. Each module in routers contains a APIRouter instance dedicated to a specific part of the API.

Each function operates on the served dataset by overriding the get_dataset dependency for RestAccessor.app.

TODO:

• [x] move zarr-specific path operation functions after #21
• ~~maybe refactor tests (if directly testing APIRouter instances is possible and a good idea)~~

It would be great if we could publish multiple datasets on the same server.

I'm thinking of something like this:

xpublish.serve(
    {'ds1': xarray.Dataset(...), 'ds2': xarray.Dataset(...)},
    host="127.0.0.1",
    port=9000
)

or

# will launch the server
ds1.rest.serve(host="127.0.0.1", port=9000, name="ds1")

# same host/port -> will reuse the server
ds2.rest.serve(host="127.0.0.1", port=9000, name="ds2")

Would there be any technical challenge in supporting this?

This will certainly break the current API end points, unless both cases (single dataset vs collection of datasets) are supported (perhaps not on the same running server).

For the case of multiple datasets, all the current end points could for example have the prefix /datasets/<name>/. Some additional end points may be useful for listing the datasets in the collection.

I guess the failing roundtrip tests are related to https://github.com/pydata/xarray/pull/2844 but I'm not sure what to do here to fix it. Any idea @jhamman @andersy005?

Hello,

xpublish looks very promising and I want to use it for serving a few datasets in an experiment. I've installed xpublish in a conda environment

I do run into the exception

AttributeError: 'Dataset' object has no attribute 'rest' when running the simple script:

#!/opt/anaconda/envs/env_xpublish/bin/python

import click
import sys
import pandas as pd
import numpy as np
import xarray as xr
import xpublish

ds = xr.Dataset({'foo': (('x', 'y'), np.random.rand(4, 5))},
                 coords={'x': [10, 20, 30, 40],
                        'y': pd.date_range('2000-01-01', periods=5),
                        'z': ('x', list('abcd'))})


ds.rest.serve(host='0.0.0.0', port=9000)

Any help/tips is really appreciated

Overview

This PR modifies xpublish to be able to server multiple datasets based on @benbovy prototype.

This is an attempt to address #23 and #25.

Notes

Further analysis needs to be done to see if dask is working correctly and also caching, otherwise, it seems to work to serve multiple datasets.

A couple of comments:

• The rest accessor API is now documented using sphinx-autosummary-accessors.

• I replaced the ipython directives by regular python code blocks. I don't think using ipython directives are worth relying on ipython + all xpublish's runtime dependencies for building the docs, given that we don't really leverage the interactive output here. I'm not against reverting this change in case anyone has objections.

Recently, @lsetiawan and @benbovy have been making contributions to this repository. Would now be a good time to move the repository to a GitHub organization. I think xarray-contrib is a logical place but Pangeo would also be fine by me.

Overview

Adding init_app method to set additional configuration to FastAPI configuration to allow more control to app and expand.

Need this for sub-application to build proxying for multiple datasets: https://fastapi.tiangolo.com/advanced/sub-applications-proxy/

otherwise this may happen on the /versions route:

  File "/home/mah/.local/lib/python3.7/site-packages/xpublish/utils/info.py", line 38, in get_sys_info
    blob.append(('commit', commit))
UnboundLocalError: local variable 'commit' referenced before assignment

This fixes FastAPI's generated API docs in the case where a single dataset is published. The dataset_id parameter is not shown anymore.

The fix is based on FastAPI docs: https://fastapi.tiangolo.com/advanced/extending-openapi/. I had to do some tricks to support a bunch of changes in last FastAPI versions, though. Hopefully it won't change too often in the future.

This fix won't be needed anymore if it is addressed upstream (see https://github.com/tiangolo/fastapi/issues/1594).

I just released sphinx-autosummary-accessors 0.1, so that should be preferred over installing from github. This also pins sphinx to sphinx>=3.1 which fixes the incomplete summary for callable accessors.

Another variation on #140 with a few of the ideas from the discussion there and #139.

Plugin routers are now nested under a parent Plugin class which now acts as a way to combine multiple related pieces of functionality together (say db management routes and a CLI). This allows new plugin functionality to be added in other plugins or Xpublish related libraries without requiring the parent Plugin class to define everything.

Plugins are loaded from the xpublish.plugin entrypoint group. Plugins can be manually configured via the plugins argument to xpublish.Rest. The specifics of plugin loading can be changed by overriding the .setup_plugins() method.

Some other xpublish.Rest functionality has been refactored out into separate methods to allow easier overriding for instance making a SingleDatasetRest class that will allow simplifying xpublish.Rest.

The ds.rest accessor has been move out into it's own file.

Builds on top of @benbovy's work in building router factories in https://github.com/xarray-contrib/xpublish/pull/89 to build a plugin system, to try to implement some of my thoughts in https://github.com/xarray-contrib/xpublish/discussions/139

The plugin system uses entry points, which are most commonly used for console or GUI scripts. The entry_point group is xpublish.plugin. Right now plugins can provide dataset specific and general (app) routes, with default prefixes and tags for both.

Xpublish will by default load plugins via the entry point. Additionally, plugins can also be loaded directly via the init, as well as being disabled, or configured. The existing dataset router pattern also still works, so that folks aren't forced into using plugins as the only way to extend functionality.

It runs against the existing test suite, but I haven't implemented any new tests or docs yet.

Entry point reference:

• https://setuptools.pypa.io/en/latest/userguide/entry_point.html
• https://packaging.python.org/en/latest/specifications/entry-points/
• https://amir.rachum.com/amp/blog/2017/07/28/python-entry-points.html

On 2022-12-09 we met to discuss various Xarray based data server projects. Discussion post announcing meeting

Purpose: Meetup to discuss progress and plans for OpenDAP, WMS and other API layers on top of the Xarray/Dask (aka Pangeo) Python stack, such as:

• Xpublish
  • xreds built on top of Xpublish
• ZarrDAP
  • Implements OPeNDAP and a custom HTML ZarrDAP Catalog, from which it generates an Intake catalog.

Attendees:

• Rich Signell / USGS / @rsignell-usgs
• Alex Kerney / Gulf of Maine Research Institute & NorthEast Regional Association of Coastal and Ocean Observing Systems / @abkfenris
• Anthony Aufdenkampe / LimnoTech / @aufdenkampe
  • Helping USGS NHGF to configure pygeoapi-edr (+ZarrDAP or Xpublish) against the same stac to document XYZT zarr data in S3
• Joe Hamman / Earthmover / @jhamman
  • started Xpublish
• Filipe Fernandes / IOOS / @ocefpaf
• Don Setiawan / UW OOI Regional Cabled Array @lsetiawan
• Jonathan Joyce / RPS Group / @jonmjoyce
• Matthew Iannucci / RPS Group / @mpiannucci
• Dave Blodgett / USGS Water /
• Andrew Buddenberg / NOAA/NCEI
  • thinks he's in charge of ZarrDAP now
• Shane Mill / NOAA/NWS / @ShaneMill1
• Steve Olson / NOAA/NWS / @solson-nws
  • Implementing EDR
• Jon Blower / National Oceanography Centre, UK / @jonblower
• Chad Whitney / NOAA/NCEI
• Paul Tomasula / LimnoTech / @ptomasula
• Sarah Jordan / LimnoTech / @sjordan29
• Xavier Nogueira / LimnoTech / @xaviernogueira
• Dave Stuebe
• Michah Wengren / IOOS / @mwengren
• Patrick Tripp / RPSgroup / @patrick-tripp

Agenda & Notes

• Intros
  • (Go around by order in attendee list, probably 1-3 min each)
  • who are you, where do you work, background in the space.
• Why are you/org intrested in working on Discussion & Python
  • Xpublish (Matt)): need a THREDDS replacement (not cloud-ready) data servers?
    • See issue Create xpublish routers for building APIs around xarray datasets #23
    • xreds built on top of Xpublish
    • Kerchunk is a bridge to putting legacy grib and netcdf data into zarr aggregations
  • ZarrDAP
    • Chad: Andrew just open-sourced ZarrDAP, but introduced a bug that they need to fix
    • Andrew: We're tired of THREDDS
      • Mark Capece connected ERDAP to ZarrDAP and got a fantastic speedup.
      • Alex's experimentation with replacing Xpublish dataset loading with opening any ERDDAP GridDAP dataset: https://xpublish.onrender.com/docs & https://github.com/abkfenris/xpublish-erddap
    • Dave B: THREDDS team is well-aware of thes issues.
      • THREDDS team taking apart to build microservices from allll THREDDS functionality
      • Key issue with THREDDS is cost of S3 egress fees
      • We need ...
  • PyGeoAPI-EDR
    • Shane building AWS scaling capabilities, which he wants to contribute to PyGeoAPI-EDR
      • AWS API Gateway + Lambda & Fargate, reaching out to ECS.
  • Xpublish update from Joe.
    • Very open to others working on it. Such as Benoit Bovey
    • Could still benefit from more active developers
    • We need example arcitectures that use Xpublish
    • Perhaps a router plugin interface would be useful
  • (similar round robin)
• What are folks working on?
  • (we can start round robin, but this can move into more of a discussion, we will want to keep moving so we don't get bogged down in any one avenue of work)
  • Demos?
• How can we work together, rather than duplicate each others efforts?
  • Can XPublish & ZarrDAP efforts or codebase be "merged"?
    • Matt: interesting to see that Xpublish & ZarrDAP seem to have almost identical approaches for accessing the data despite being developed totally independently
  • Alex's vision for Xpublish
    • Make it modular. Maybe a core/plugin/distro interface
      • Xpublish becomes the core, similar to the Linux kernel with a standard set of interfaces for routers and data loaders to interact with.
      • Routers are plugins, so that 'router' interfaces are all separate repos, such as:
        • OpenDAP (via https://github.com/gulfofmaine/xpublish-opendap)
        • EDR (via https://github.com/gulfofmaine/xpublish-edr)
        • WMS (via https://github.com/asascience-open/xpublish-wms)
      • Various deployments will assemble different router and data loading plugins for various use cases.
      • I'll post a more full fledged write up of my idea in the discussions
    • Andrew: Are you suggesting that ZarrDAP be rewritten to be plugin to XPublish? Alex: maybe...
      • Alex: I've made a very alpha OpenDAP Xpublish router ( https://github.com/gulfofmaine/xpublish-opendap ), but you've tested it much more. I'm thinking that you refactor onto Xpublish and adapt your data loading into xpublish.get_dataset. It also means that as we create new Xpublish routers, you can get those for free
  • Caching discussion
    • Dave S: demo of real-time Forecast Model Run Collection (FMRC) for HRRR, with caching using fsspec 'simplecache' command
    • Will post PR for adding the core parts of the HRRR aggregation to https://github.com/asascience-open/nextgen-dmac

Action items

• Move conversation to XPublish repo, which is followed a bunch of additional people not on this call.
• Try to get a regular meeting going. Possibly under the Pangeo umbrella?

Hi, I get this error when executing rest.serve() with

RuntimeError: asyncio.run() cannot be called from a running event loop
sys:1: RuntimeWarning: coroutine 'Server.serve' was never awaited

I have xpublish 0.2.0, xarray 2022.6.0, uvicorn 0.18.3 I don't have asyncio in my conda list, is it expected ?

Actually I need some more explanations on how it works. I already have a code using fastAPI and the uvicorn server installed.

Should I launch uvicorn for rest.serve() to work ? I get this error when server is switched off too (when I run it in spyder in a quite new environment).

Thank you

I recently learned about zarrdap. ZarrDAP is a FastAPI project that provides access to Zarr and NetCDF data in remote object storage using the Open-source Project for a Network Data Access Protocol (OPeNDAP).

This has me wondering if we can plug in the xarray opendap handler here. @markccapece, wondering if you have run across xpublish before and if you have thoughts on how the xarray handler in zarrdap could be used outside of zarrdap?

xref: #50