@blaylockbk , this is probably the wrong place to raise this, but I saw in your HRRR Archive FAQ, you said:

One day, we hope this data will be archived elsewhere that is more accessible to everyone. Perhaps soon it will be hosted by Amazon by their Opendata initiative. I would advocate to keep it in the GRIB2 format (the original format it is output as), but it would also be nice to store the data in a "cloud-friendly" format such as zarr.

To have archived HRRR data in Zarr would be AMAZING. We were trying to figure out how to download 1 year of HRRR surface fields to drive a Delaware Bay hydrodynamics simulation, and thinking how useful it would be to have the data on AWS. We could store as Zarr but create GRIB-on-demand service for those who need it. I've been active on the Pangeo project, and we have some tools now that could make the conversion, chunking and upload to cloud much easier. And I'd be happy to help out.

@zflamig, you guys would be up for a proposalon this, right ?

I was just following the basic setup guide. Installed with conda using the example environment.yaml file, and ran into this issue:

>>> H.xarray('TMP:2 m')
indexpath value  is ignored
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/herbie/archive.py", line 900, in xarray
    Hxr = cfgrib.open_datasets(
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_store.py", line 105, in open_datasets
    datasets = open_variable_datasets(path, backend_kwargs=backend_kwargs, **kwargs)
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_store.py", line 93, in open_variable_datasets
    datasets.extend(raw_open_datasets(path, bk, **kwargs))
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_store.py", line 66, in raw_open_datasets
    datasets.append(open_dataset(path, backend_kwargs=backend_kwargs, **kwargs))
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_store.py", line 39, in open_dataset
    return xr.open_dataset(path, **kwargs)  # type: ignore
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/xarray/backends/api.py", line 495, in open_dataset
    backend_ds = backend.open_dataset(
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_plugin.py", line 99, in open_dataset
    store = CfGribDataStore(
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/xarray_plugin.py", line 39, in __init__
    self.ds = opener(filename, **backend_kwargs)
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/dataset.py", line 728, in open_fieldset
    index = messages.FieldsetIndex.from_fieldset(fieldset, index_keys, computed_keys)
  File "/usr/local/anaconda3/envs/herbie/lib/python3.9/site-packages/cfgrib/messages.py", line 365, in from_fieldset
    iteritems = enumerate(fieldset)
TypeError: 'PosixPath' object is not iterable

It seems like that cfgrib either expects a str filename, or an already opened object: https://github.com/ecmwf/cfgrib/blob/a0d9763/cfgrib/xarray_plugin.py#L35-L38

Converting the path to str here makes the example pass for me.

Hi!

I have tried to download some radiation variables from GFS with no success. Some months ago I was able to download this data with the same version (Herbie 0.0.6), however, now it only gives me the following set of allowed variables:

array(['PRMSL', 'CLWMR', 'ICMR', 'RWMR', 'SNMR', 'GRLE', 'REFD', 'REFC',
       'VIS', 'UGRD', 'VGRD', 'VRATE', 'GUST', 'HGT', 'TMP', 'RH', 'SPFH',
       'VVEL', 'DZDT', 'ABSV', 'O3MR', 'TCDC', 'HINDEX', 'MSLET', 'PRES',
       'TSOIL', 'SOILW', 'SOILL', 'CNWAT', 'WEASD', 'SNOD', 'ICETK',
       'DPT', 'APTMP', 'ICEG', 'CPOFP', 'PRATE', 'CSNOW', 'CICEP',
       'CFRZR', 'CRAIN', 'SFCR', 'FRICV', 'VEG', 'SOTYP', 'WILT', 'FLDCP',
       'SUNSD', 'LFTX', 'CAPE', 'CIN', 'PWAT', 'CWAT', 'TOZNE', 'LCDC',
       'MCDC', 'HCDC', 'HLCY', 'USTM', 'VSTM', 'ICAHT', 'VWSH', '4LFTX',
       'HPBL', 'POT', 'PLPL', 'LAND', 'ICEC', 'ICETMP'], dtype=object)

It is only a small subset of the full set of variables in NOAA GFS. Is not possible to download radiation variables from GFS anymore with Herbie?

I am truly aweful at python I can't seem to work with grib outside of python so please don't laugh too hard when you read my code:

from herbie.archive import Herbie
import numpy as np

H = Herbie('2022-01-26', model='ecmwf', product='oper', fxx=12)

ds = H.xarray(':2t:', remove_grib=False)

dsw = H.xarray(':10(u|v):', remove_grib=False)
ds['spd'] = np.sqrt(dsw['u10'] ** 2 + dsw['v10'] ** 2)

dsp = H.xarray(':tp:', remove_grib=False)
ds['tp'] = dsp['tp']

file = open('test.txt', 'a')
for lon in ds['longitude']:
    for lat in ds['latitude']:
        point = ds.sel(longitude=lon, latitude=lat, method='nearest')
        line = str(point['longitude'].values) + ',' + str(point['latitude'].values) + ',' + str(point['t2m'].values) + ',' + str(point['spd'].values) + ',' + str(point['tp'].values) + '\n'
        file.write(line)
file.close()

After 5 minutes it was like 5% done. I get why it's bad, but I honestly just don't want to spend a month learning python.

I prefer to just make like a (405900, 5) array and store a raw blob file of float32s like so:

lon1,lat1,t2m1,spd1,tp1,.....,lonN,latN,t2mN,spdN,tpN

Any advice would be amazing.

First, this tool is awesome. Thanks for publishing and maintaining it!

I installed your package using conda install -c conda-forge herbie-data, but wasn't able to run this portion of your tutorial https://blaylockbk.github.io/Herbie/_build/html/user_guide/notebooks/data_hrrr.html without first installing Carpenter Workshop, e.g.,

pip install git+https://github.com/blaylockbk/Carpenter_Workshop.git

It might be helpful to add some additional instruction/details in the tutorials or here https://github.com/blaylockbk/Herbie#installation.

I am trying to install Herbie using the conda environment.yml file but conda is finding conflicts that are causing the install to fail:

(base) jmiller@ubuntu:~$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:



Package geos conflicts for:
metpy -> cartopy[version='>=0.15.0'] -> shapely[version='>=1.6.4'] -> geos[version='>=3.4']
cartopy[version='>=0.20.3'] -> geos[version='>=3.10.3,<3.10.4.0a0|>=3.11.0,<3.11.1.0a0']
metpy -> cartopy[version='>=0.15.0'] -> geos[version='3.6.2|>=3.10.0,<3.10.1.0a0|>=3.10.1,<3.10.2.0a0|>=3.10.2,<3.10.3.0a0|>=3.10.3,<3.10.4.0a0|>=3.11.0,<3.11.1.0a0|>=3.6.2,<3.6.3.0a0|>=3.7.0,<3.7.1.0a0|>=3.7.1,<3.7.2.0a0|>=3.7.2,<3.7.3.0a0|>=3.8.0,<3.8.1.0a0|>=3.8.1,<3.8.2.0a0|>=3.9.0,<3.9.1.0a0|>=3.9.1,<3.9.2.0a0']
geopandas -> shapely -> geos[version='3.6.2|>=3.10.0,<3.10.1.0a0|>=3.10.1,<3.10.2.0a0|>=3.10.2,<3.10.3.0a0|>=3.10.3,<3.10.4.0a0|>=3.11.0,<3.11.1.0a0|>=3.4|>=3.6.2,<3.6.3.0a0|>=3.7.0,<3.7.1.0a0|>=3.7.1,<3.7.2.0a0|>=3.7.2,<3.7.3.0a0|>=3.8.0,<3.8.1.0a0|>=3.8.1,<3.8.2.0a0|>=3.9.0,<3.9.1.0a0|>=3.9.1,<3.9.2.0a0']

I was able to manually create a conda environment and then install using pip

UPDATE: Turns out the conda environment I set up using pip actually doesn't want to work either. I'll keep trying and see if I can get it working.

Would Multithreading help download many files (and many parts of files) quickly?

This would be a helper tool used in herbie.tools (in bulk_download)

Check out this article for some inspiration.

• https://superfastpython.com/threadpoolexecutor-download-books/
• https://superfastpython.com/threadpoolexecutor-map-vs-submit/

It seems like simply downloading files are parts of files is an IO-bound task that could see some speedup fro multi threading.

Possiblly could see speedup when iterating on downloading chunks of the file.

I'm trying to add IMERG to the list of models. To download the data one must register with NASA and then use a username/password to access the data.

Typically I've gotten the data using python through a url using this format https://username:password@URL however when I add that style of url to the model's self.SOURCES I get the following Traceback

Traceback (most recent call last):
  File "herbie_methods.py", line 37, in <module>
    H.download(verbose=False)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/site-packages/herbie/archive.py", line 642, in download
    urllib.request.urlretrieve(self.grib, outFile, _reporthook)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/urllib/request.py", line 1323, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/http/client.py", line 1383, in __init__
    super(HTTPSConnection, self).__init__(host, port, timeout,
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/http/client.py", line 834, in __init__
    (self.host, self.port) = self._get_hostport(host, port)
  File "/Users/judson/opt/anaconda3/envs/main/lib/python3.8/http/client.py", line 877, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: '[email protected]'

Is there a way around this, or should I add the username/password differently?

It would be nice if Herbie could be installed with Conda directly, instead of using Pip. Especially since Herbie depends on cfgrib and cartopy, which have dependencies that can't be installed with pip (proj, GEOS, eccodes)

I don't have experience with this, but want to learn. If anyone can help out with this, that would be awesome 😁

• https://blog.gishub.org/how-to-publish-a-python-package-on-conda-forge
• https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs-skeleton.html
• https://github.com/conda-forge/staged-recipes/blob/main/recipes/example/meta.yaml
• https://github.com/blaylockbk/staged-recipes#getting-started

In attempting to run

dates = pd.date_range('2022-01-01 1:00', 
                      '2022-01-01 3:00',
                      freq='1H')
fxx = 1
h_list = fast_Herbie_xarray(DATES=dates, fxx=fxx, model='hrrr', product='subh', searchString=':PRES:surface')

I get the following traceback

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Untitled-1.ipynb Cell 5' in <cell line: 1>()
----> [1](vscode-notebook-cell:Untitled-1.ipynb?jupyter-notebook#ch0000011untitled?line=0) hh = fast_Herbie_xarray(DATES=dates, fxx=fxx, model='hrrr', product='subh', searchString=':PRES:surface')

File ~/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py:228, in fast_Herbie_xarray(DATES, searchString, fxx, max_threads, xarray_kw, **kwargs)
    [225](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=224)     ds_list = [future.result() for future in as_completed(futures)]
    [227](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=226) # Sort the DataSets, first by lead time (step), then by run time (time)
--> [228](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=227) ds_list.sort(key=lambda ds: ds.step.item())
    [229](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=228) ds_list.sort(key=lambda ds: ds.time.item())
    [231](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=230) # Reshape list with dimensions (len(DATES), len(fxx))

File ~/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py:228, in fast_Herbie_xarray.<locals>.<lambda>(ds)
    [225](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=224)     ds_list = [future.result() for future in as_completed(futures)]
    [227](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=226) # Sort the DataSets, first by lead time (step), then by run time (time)
--> [228](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=227) ds_list.sort(key=lambda ds: ds.step.item())
    [229](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=228) ds_list.sort(key=lambda ds: ds.time.item())
    [231](file:///Users/judson/miniforge3/envs/herbie/lib/python3.8/site-packages/herbie/tools.py?line=230) # Reshape list with dimensions (len(DATES), len(fxx))

AttributeError: 'list' object has no attribute 'step'

When I just get one herbie obj and read data into xarray I get the following note

Note: Returning a list of [2] xarray.Datasets because of multiple hypercubes.

and H.xarray() ends up returning a list of two xarray.Datasets (one with the 15min,30min,45min forecast and one with the 1hr fcst). Pretty sure this is what's causing the issue of not being able to use fast_Herbie_xarray() with hrrr subh. Not sure if there's a way around it?

When using the subset function and attempting to subset multiple levels (t2m, surface, 10m wind, etc), xarray does not open all variables, only attained surface variables. Wondering if this has to do with Herbie or xarray. Thought I would make you aware of this.

I would like to make changes to the way the config file is used, like as the ability to change the config values temporarily (adjust the defaults inside a single script)

Here is what ChatGPT says I could do:

To allow a user to temporarily update config attributes without modifying the config file, you can modify the Config class to store the temporary updates in a separate dictionary. Here's an example of how you can do this:

class Config:
    def __init__(self, config_file):
        self.config_file = config_file
        self.config = {}
        self.temp_updates = {}
        with open(self.config_file, 'r') as f:
            for line in f:
                key, value = line.strip().split('=')
                self.config[key] = value

    def get(self, key):
        if key in self.temp_updates:
            return self.temp_updates[key]
        return self.config[key]

    def set(self, key, value, temporary=False):
        if temporary:
            self.temp_updates[key] = value
        else:
            self.config[key] = value
            with open(self.config_file, 'w') as f:
                for k, v in self.config.items():
                    f.write(f'{k}={v}\n')

    def reset(self):
        self.temp_updates = {}

With this modified Config class, you can use the set method to temporarily update a config value by setting the temporary argument to True. The temporary update will be stored in the temp_updates dictionary, and the get method will return the temporary value if it exists.

To reset all temporary updates and restore the original config values, you can use the reset method. This will clear the temp_updates dictionary.

Here's an example of how you might use these new methods:

# Load the config file
config = Config('/path/to/config.txt')

# Get the original value of a config key
value = config.get('key')

# Temporarily update the value of the key
config.set('key', 'new_value', temporary=True)

# Get the temporary updated value of the key
temp_value = config.get('key')

# Reset the temporary updates and restore the original value
config.reset()

# Get the original value of the key again
value = config.get('key')

There, that might give you a starting place.

Ok, so I saw this on LinkedIn and it would be super helpful for Herbie...

This python package would enable Herbie to subset data even when an index file isn't available for the file.

https://github.com/joxtoby/beltzer

Discussed in https://github.com/blaylockbk/Herbie/discussions/116

variable = "tp" #tp for precipitation
tp_all = []
valid_times = []
forecast_hours_qpf = range(3,147,3)
model_search_string = ":"+variable+":sfc:"
#forecast_hours_qpf = range(3,147,3)
#ptotal = fast_Herbie_xarray(DATES=model_run.strftime('%Y-%m-%d %H:00'), model="ecmwf", product="enfo", fxx=forecast_hours_qpf, max_threads=5, search_string=model_search_string)

for t in forecast_hours_qpf:
    H = Herbie(model_run.strftime('%Y-%m-%d %H:00'), model="ecmwf", product="enfo", fxx=t)
    tp = H.xarray(":"+variable+":sfc:")[0]
    #tp = tp.rename({"number":"pertubation"})
    tp_all.append(tp)
    valid_times.append(model_run + timedelta(hours=t))

ptotal = xr.concat([tp_all[i] for i in range(0,len(forecast_hours_qpf))], dim='step')

Codespaces just rolled out. Let's see what I can use it for.

60 free hours seems like plenty of free compute time to check bugs and some development work...without needing to install the python environment on my laptop.

Requested more than once...

it is possible to set a region and some variables names to download.

No, it's not possible, but wgrib2 can subset a grib file for you after it is downloaded.