Pull Request

Description

Fixes #6

How Has This Been Tested?

• [ ] No
• [x] Yes, the conversion script runs

Checklist:

• [x] My code follows OCF's coding style guidelines
• [x] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

Detailed Description

Rename x to x_osgb and y to y_osgb

Context

Good to use the same variable names across different repos linked with https://github.com/openclimatefix/nowcasting_dataset/pull/558

Possible Implementation

just rename zarr file, rather than remake all data

Detailed Description

Check that values are in the expected range, and that datetimes are correct, etc.

Possible Implementation

Validation could happen in two places:

1. As part of the native to Zarr conversion.
2. As a separate zarr_validation.py script.

(related to https://github.com/openclimatefix/Satip/issues/33)

scripts/download_UK_Met_Office_NWPs_from_CEDA.sh should reject:

• NWP init times at 03, 09, 15, and 21 hours past midnight
• T120 files
• Wholesale5 files

It appears it is only rejecting Wholesale5 files. It is still downloading NWP init times at 03, 09, 15, and 21 hours; and T120 files. This isn't terrible. But it is using a lot more disk space than perhaps we need.

For example, here's a selection of what the script just downloaded:

jack@leonardo$ ls /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/UK_Met_Office/UKV/native/2021/11/01
202111010000_u1096_ng_umqv_Wholesale1.grib      202111011200_u1096_ng_umqv_Wholesale1.grib
202111010000_u1096_ng_umqv_Wholesale1T54.grib   202111011200_u1096_ng_umqv_Wholesale1T54.grib
202111010000_u1096_ng_umqv_Wholesale2.grib      202111011200_u1096_ng_umqv_Wholesale2.grib
202111010000_u1096_ng_umqv_Wholesale2T54.grib   202111011200_u1096_ng_umqv_Wholesale2T54.grib
202111010000_u1096_ng_umqv_Wholesale3.grib      202111011200_u1096_ng_umqv_Wholesale3.grib
202111010000_u1096_ng_umqv_Wholesale3T54.grib   202111011200_u1096_ng_umqv_Wholesale3T54.grib
202111010000_u1096_ng_umqv_Wholesale4.grib      202111011200_u1096_ng_umqv_Wholesale4.grib
202111010300_u1096_ng_umqv_Wholesale1.grib      202111011500_u1096_ng_umqv_Wholesale1.grib
202111010300_u1096_ng_umqv_Wholesale1T120.grib  202111011500_u1096_ng_umqv_Wholesale1T120.grib
202111010300_u1096_ng_umqv_Wholesale1T54.grib   202111011500_u1096_ng_umqv_Wholesale1T54.grib
202111010300_u1096_ng_umqv_Wholesale2.grib      202111011500_u1096_ng_umqv_Wholesale2.grib
202111010300_u1096_ng_umqv_Wholesale2T120.grib  202111011500_u1096_ng_umqv_Wholesale2T120.grib
202111010300_u1096_ng_umqv_Wholesale2T54.grib   202111011500_u1096_ng_umqv_Wholesale2T54.grib
202111010300_u1096_ng_umqv_Wholesale3.grib      202111011500_u1096_ng_umqv_Wholesale3.grib
202111010300_u1096_ng_umqv_Wholesale3T120.grib  202111011500_u1096_ng_umqv_Wholesale3T120.grib
202111010300_u1096_ng_umqv_Wholesale3T54.grib   202111011500_u1096_ng_umqv_Wholesale3T54.grib
202111010300_u1096_ng_umqv_Wholesale4.grib      202111011500_u1096_ng_umqv_Wholesale4.grib
202111010600_u1096_ng_umqv_Wholesale1.grib      202111011800_u1096_ng_umqv_Wholesale1.grib
202111010600_u1096_ng_umqv_Wholesale1T54.grib   202111011800_u1096_ng_umqv_Wholesale1T54.grib
202111010600_u1096_ng_umqv_Wholesale2.grib      202111011800_u1096_ng_umqv_Wholesale2.grib
202111010600_u1096_ng_umqv_Wholesale2T54.grib   202111011800_u1096_ng_umqv_Wholesale2T54.grib
202111010600_u1096_ng_umqv_Wholesale3.grib      202111011800_u1096_ng_umqv_Wholesale3.grib
202111010600_u1096_ng_umqv_Wholesale3T54.grib   202111011800_u1096_ng_umqv_Wholesale3T54.grib
202111010600_u1096_ng_umqv_Wholesale4.grib      202111011800_u1096_ng_umqv_Wholesale4.grib
202111010900_u1096_ng_umqv_Wholesale1.grib      202111012100_u1096_ng_umqv_Wholesale1.grib
202111010900_u1096_ng_umqv_Wholesale1T54.grib   202111012100_u1096_ng_umqv_Wholesale1T54.grib
202111010900_u1096_ng_umqv_Wholesale2.grib      202111012100_u1096_ng_umqv_Wholesale2.grib
202111010900_u1096_ng_umqv_Wholesale2T54.grib   202111012100_u1096_ng_umqv_Wholesale2T54.grib
202111010900_u1096_ng_umqv_Wholesale3.grib      202111012100_u1096_ng_umqv_Wholesale3.grib
202111010900_u1096_ng_umqv_Wholesale3T54.grib   202111012100_u1096_ng_umqv_Wholesale3T54.grib
202111010900_u1096_ng_umqv_Wholesale4.grib      202111012100_u1096_ng_umqv_Wholesale4.grib

Detailed Description

The NWP conversion script I'm working on in openclimatefix/nowcasting_dataset#344 will, for now, only convert the same NWPs we've used in the past (NWPs at the Earth's surface)

At some point, we should also experiment with NWPs at altitude (e.g. temperature and humidity at cloud altitude).

Context

NWPs at cloud altitude may help our models figure out when clouds are likely to appear / disappear.

But it's a lot more data.

Possible Implementation

The UKV NWPs have something like 70 vertical levels. We probably don't need all those. Maybe we could compute the mean for three vertical levels: 0 to 3.3 km; 3.3 km to 6.6 km; 6.6 km to 10 km.

Priority

I'll put this into WP1 for now as a "nice to have"... but this may have to wait until WP2.