A ninja python package that unifies the Google Earth Engine ecosystem.

Hi, @aazuspan!

@csaybar and I are giving ee_extra the final touch and we want to know if you would accept to move the amazing matchHist and panSharpen resources from eemont to this repo and make them available for the whole ecosystem.

Please let me know if you accept! :)

And if you want, I can move them myself (and also re-arrange the resources in eemont by using ee_extra), just let me know!

Cheers!

Dave

Hi @davemlz, please :pray: can you add a drop=False argument to ee_extra.Spectral.core.spectralIndices?

library(rgee)
library(rgeeExtra)

ee_Initialize()

S2 <- ee$ImageCollection('COPERNICUS/S2_SR')$first()
names(ee_ImageCollection_spectralIndex(S2, 'kernel'))

I got:

[1] "B1" "B2" "B3" "B4" "B5" "B6" "B7" "B8" "B8A" "B9"
[11] "B11" "B12" "AOT" "WVP" "SCL" "TCI_R" "TCI_G" "TCI_B" "QA10" "QA20" [21] "QA60" "kEVI" "kNDVI" "kRVI" "kVARI"

I would like: [1] "kEVI" "kNDVI" "kRVI" "kVARI"

Tasseled cap coefficients for Landsat 8/9 TOA and SR data were recently published by Zhai et al. We already have Landsat 8 coefficients from Baig et al, so I'll need to do some reading to see which to include...

import ee, eemont, ee_extra
ee.Initialize()
S2 = ee.ImageCollection('MODIS/006/MCD12Q1').closest('2020-10-15')
S2.getInfo()

ee_extra.ImageCollection.core.closest(
    ee.ImageCollection('MODIS/006/MCD12Q1'), '2020-10-15'
).getInfo()

Traceback (most recent call last): File "", line 1, in File "/home/csaybar/.local/lib/python3.8/site-packages/ee/collection.py", line 127, in getInfo return super(Collection, self).getInfo() File "/home/csaybar/.local/lib/python3.8/site-packages/ee/computedobject.py", line 98, in getInfo return data.computeValue(self) File "/home/csaybar/.local/lib/python3.8/site-packages/ee/data.py", line 674, in computeValue return _execute_cloud_call( File "/home/csaybar/.local/lib/python3.8/site-packages/ee/data.py", line 336, in _execute_cloud_call raise _translate_cloud_exception(e) ee.ee_exception.EEException: Element.get: Parameter 'object' is required.

• Fix #5 by adding tasseledCap function to Spectral module for calculating brightness, greenness, and wetness from Sentinel-2, MODIS, and Landsat 4-8 images and image collections
• Add utility function for matching platforms to tasseled cap coefficients
• Update tests and docs to include tasseledCap

Let me know if any changes are needed :)

See the original feature request on eemont.

This would add an ee_extra.Transformations module with a tasseledCap function to calculate tasseled cap brightness, wetness, and greenness bands from an input ee.Image from the MODIS, Landsat 5/7/8, or Sentinel-2 platforms.

A couple design questions:

• Should the tasseled cap bands be added to the original image (like spectralIndices) or returned as a new ee.Image?
• How strict should the platform matching be? For example, published Landsat 8 coefficients were generated for at-sensor reflectance and technically aren't accurate for TOA or Surface Reflectance (see the warning on Hexagon Geospatial). Should that be strictly enforced or just mentioned in the docs?

Feel free to assign me, thanks!

@davemlz

x = """
var landsatALL = (landsatSR.combine(landsatTOA.select(tir), true));
"""

x = fix_identation(x)
x = normalize_fn_style(x)
x = variable_definition(x)
x = logical_operators_boolean_null_comments(x)
x = multiline_comments(x)
x = multiline_method_chain(x)
x = function_definition(x)
x = dictionary_keys(x) 
x = dictionary_object_access(x)
x = keyword_arguments_object(x) # ERROR HERE!
x = if_statement(x)
x = array_isArray(x)
x = for_loop(x) # ERROR HERE!
x = extra_work(x)

with open("/home/csaybar/Desktop/test3.py", 'w') as f:
    f.write(x)

Got:

landsatALL = (landsatSR.combine(landsatTOA.select(tir),True)

Expected:

landsatALL = (landsatSR.combine(landsatTOA.select(tir), True))

Hey folks, it looks like all of the Sphinx autosummaries are failing to build, so the latest API reference is empty. Trying to build locally gives a lot of No module named 'ee' warnings.

I've fixed this in a few of my projects by adding autodoc_mock_imports = ["ee"] to docs/conf.py, and that seems to work here, but there's probably a better solution since I think that's just hiding the problem. Any Sphinx pros know how to tackle this the right way?

JS file :

  var levelsDic = ee.Dictionary({
        'level0': {'zoom': 0, 'scale': 157000},
        'level1': {'zoom': 1, 'scale': 78000},
        'level2': {'zoom': 2, 'scale': 39000},
        'level3': {'zoom': 3, 'scale': 20000},
        'level4': {'zoom': 4, 'scale': 10000},
        'level5': {'zoom': 5, 'scale': 5000},
        'level6': {'zoom': 6, 'scale': 2000},
        'level7': {'zoom': 7, 'scale': 1000},
        'level8': {'zoom': 8, 'scale': 611},
        'level9': {'zoom': 9, 'scale': 306},
        'level10': {'zoom': 10, 'scale': 153},
        'level11': {'zoom': 11, 'scale': 76},
        'level12': {'zoom': 12, 'scale': 38},
        'level13': {'zoom': 13, 'scale': 19},
        'level14': {'zoom': 14, 'scale': 10},
        'level15': {'zoom': 15, 'scale': 5},
  });

GOT:

import ee
import math
from collections import UserDict

exports = UserDict()

levelsDic = ee.Dictionary({
'level0:{zoom': 0,'scale': 157000},
      'level1': {'zoom': 1,'scale': 78000},
      'level2': {'zoom': 2,'scale': 39000},
      'level3': {'zoom': 3,'scale': 20000},
      'level4': {'zoom': 4,'scale': 10000},
      'level5': {'zoom': 5,'scale': 5000},
      'level6': {'zoom': 6,'scale': 2000},
      'level7': {'zoom': 7,'scale': 1000},
      'level8': {'zoom': 8,'scale': 611},
      'level9': {'zoom': 9,'scale': 306},
      'level10': {'zoom': 10,'scale': 153},
      'level11': {'zoom': 11,'scale': 76},
      'level12': {'zoom': 12,'scale': 38},
      'level13': {'zoom': 13,'scale': 19},
      'level14': {'zoom': 14,'scale': 10},
      'level15': {'zoom': 15,'scale': 5},
})

EXPECTED:

import ee
import math
from collections import UserDict

exports = UserDict()

levelsDic = ee.Dictionary({
      'level0': {'zoom': 0,'scale': 157000},
      'level1': {'zoom': 1,'scale': 78000},
      'level2': {'zoom': 2,'scale': 39000},
      'level3': {'zoom': 3,'scale': 20000},
      'level4': {'zoom': 4,'scale': 10000},
      'level5': {'zoom': 5,'scale': 5000},
      'level6': {'zoom': 6,'scale': 2000},
      'level7': {'zoom': 7,'scale': 1000},
      'level8': {'zoom': 8,'scale': 611},
      'level9': {'zoom': 9,'scale': 306},
      'level10': {'zoom': 10,'scale': 153},
      'level11': {'zoom': 11,'scale': 76},
      'level12': {'zoom': 12,'scale': 38},
      'level13': {'zoom': 13,'scale': 19},
      'level14': {'zoom': 14,'scale': 10},
      'level15': {'zoom': 15,'scale': 5},
})

This PR would close #42 by adding tasseled cap coefficients for Landsat 8 and 9, TOA and SR, from Zhai et al. 2022. To keep things consistent across the platforms, the L8 TOA coefficients replace the existing coefficients from Baig et al.

Zhai et al. provided 5- and 6-band coefficients (with and without the blue band) and I chose to include the 5-band to minimize atmospheric effects, as recommended in their conclusions.

Aside from adding the coefficients, I just did a little bit of clean up, shortening the redundant docstring and passing the platform name rather than entire platform dict to _get_tc_coefficients.

Hi @csaybar and @davemlz, here's the first eemont feature migrated over!

• Implement part of #28 by adding the ee_extra.Spectral.matchHistogram function
• Fix a bug in the original matchHistogram implementation that caused band order and image bounds to change on matched images
• Make bands arg optional to simplify matching bands within collections

Hi, Earth Engine community! Welcome to the list of metrics of ee_extra. Here you can find the metrics that are included, planned or in progress for ee_extra. Please feel free to pin one of the admins to add, edit or report a new metric in this list: @davemlz, @csaybar, @aazuspan @KMarkert. Note: This Issue will remain open.

| Metric | Status | Assigned | PR |:----------:|:-------------:|:------:|:-----:| | MSE | In Progress | @aazuspan | #33 | | RMSE | In Progress | @aazuspan | #33 | | RASE | In Progress | @aazuspan | #33 | | ERGAS | In Progress | @aazuspan | #33 | | DIV | In Progress | @aazuspan | #33 | | bias | In Progress | @aazuspan | #33 | | CC | In Progress | @aazuspan | #33 | | CML | In Progress | @aazuspan | #33 | | CMC | In Progress | @aazuspan | #33 | | UIQI | In Progress | @aazuspan | #33 | | IoU | In Progress | @csaybar | | | Dice | In Progress | @csaybar | |

Hi, all. I encountered some issues when installing javascript module using ee.install. It raised HTTP Error 404: Not Found.

Downloading 'users/marcyinfeng/Algorithms:/mcd19_prior'...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\site-packages\eemont\extra.py", line 80, in install
    return ee_install(module, update, quiet)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\site-packages\ee_extra\JavaScript\install.py", line 200, in install
    return _install_dependencies(x=deps, update=update, installed=[])
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\site-packages\ee_extra\JavaScript\install.py", line 188, in _install_dependencies
    _install(dep, update, quiet)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\site-packages\ee_extra\JavaScript\install.py", line 158, in _install
    with urllib.request.urlopen(_convert_path_to_ee_sources(x)) as url:
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 640, in http_response
    response = self.parent.error(
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 502, in _call_chain
    result = func(*args)
  File "C:\Users\WSBARATA01\miniconda3\envs\gee\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

It seems caused by the colon replacement which is insufficient for other cases. I found some javascript modules which have unusual path such as: users/marcyinfeng/Algorithms:/mcd19_prior and users/marcyinfeng/Algorithms/:/S2_view_angle

def _convert_path_to_ee_sources(path: str) -> str:
    """Get the remote module path from the 'ee-sources' GCS bucket.

    Args:
        path: str

    Returns:
        An ee-sources module url.
    """
    if path.startswith("http"):
        eempath = path
    else:
        bpath = path.replace(":", "/") # <- insufficient replacement
        eempath = f"https://storage.googleapis.com/ee-sources/{bpath}"
    return eempath

I try to normalize the path using pathlib as follows:

def _convert_path_to_ee_sources(path: str) -> str:
    """Get the remote module path from the 'ee-sources' GCS bucket.

    Args:
        path: str

    Returns:
        An ee-sources module url.
    """
    if path.startswith("http"):
        eempath = path
    else:
        bpath = str(pathlib.Path(path.replace(":", "/"))).replace("\\", "/") # <- fix my issue, but only for Windows case
        eempath = f"https://storage.googleapis.com/ee-sources/{bpath}"
    return eempath

All dependencies were installed successfully:

The module 'users/marcyinfeng/utils:SIAC' is already installed!
Checking dependencies for users/marcyinfeng/utils:SIAC...
The module 'users/marcyinfeng/Algorithms:S2_MT_Cloud' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms:S2_MT_Cloud...
The module 'users/marcyinfeng/Algorithms:NN_cloud' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms:NN_cloud...
The module 'users/marcyinfeng/Algorithms:/mcd19_prior' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms:/mcd19_prior...
The module 'users/marcyinfeng/Algorithms:/get_aot' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms:/get_aot...
The module 'users/marcyinfeng/Algorithms/:/S2_view_angle' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms/:/S2_view_angle...
The module 'users/marcyinfeng/Algorithms/:/landsat_view_angles' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms/:/landsat_view_angles...
The module 'users/marcyinfeng/Algorithms/:/NN_prosail' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms/:/NN_prosail...
The module 'users/marcyinfeng/Algorithms/:/get_xps' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms/:/get_xps...
The module 'users/marcyinfeng/utils/:/load_NN' is already installed!
Checking dependencies for users/marcyinfeng/utils/:/load_NN...
The module 'users/gena/packages:palettes' is already installed!
Checking dependencies for users/gena/packages:palettes...
The module 'users/marcyinfeng/Algorithms/:/simu_ref' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms/:/simu_ref...
The module 'users/marcyinfeng/Algorithms:mcd19_prior' is already installed!
Checking dependencies for users/marcyinfeng/Algorithms:mcd19_prior...
The module 'users/gena/packages:text' is already installed!
Checking dependencies for users/gena/packages:text...
All dependencies were successfully installed!

But it takes a long time when loading the module using ee.require. Is it depend on how many other dependent modules?

Please, fixed this issue if possible. Thank you in advance.

Best regards, Suhendra

Hi all! :)

These are the main javascript modules we are considering adding to ee_extra algorithms subpackage. If you know about other JS modules we must considerate please write it here :)

• gena
  • users/gena/packages/utils (integrate/test)
• gee_s1_srd (integrate/test)
• TAGEE (integrate/test)
• Landsat_SMW_LTS (integrate/test)
• radiometric-slope-correction (integrate/test)
• BFAST (integrate/test)
• LandTrendr (test)
• Open Earth Engine Library (integrate/test)

Ping: @KMarkert @giswqs @aazuspan @davemlz @jdbcode