Digital Earth Australia notebooks and tools repository

Overview

Digital Earth Australia logo

Digital Earth Australia notebooks and tools repository

DOI Apache license PyPI Notebook testing

License: The code in this repository is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance with any of the Jupyter Notebooks or Python code in this repository, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Citing DEA Notebooks: If you use any of the notebooks, code or tools in this repository in your work, please reference them using the following citation:

Krause, C., Dunn, B., Bishop-Taylor, R., Adams, C., Burton, C., Alger, M., Chua, S., Phillips, C., Newey, V., Kouzoubov, K., Leith, A., Ayers, D., Hicks, A., DEA Notebooks contributors 2021. Digital Earth Australia notebooks and tools repository. Geoscience Australia, Canberra. https://doi.org/10.26186/145234

The Digital Earth Australia notebooks and tools repository (dea-notebooks) hosts Jupyter Notebooks, Python scripts and workflows for analysing Digital Earth Australia (DEA) satellite data and derived products. This documentation is designed to provide a guide to getting started with DEA, and to showcase the wide range of geospatial analyses that can be achieved using DEA data and open-source software including Open Data Cube and xarray.

The repository is based around the following directory structure (from simple to increasingly complex applications):

  1. Beginners_guide: Introductory notebooks aimed at introducing Jupyter Notebooks and how to load, plot and interact with DEA data
  2. DEA_datasets: Notebooks introducing DEA's satellite datasets and derived products, including how to load each dataset and any special features of the data
  3. Frequently_used_code: A recipe book of simple code examples demonstrating how to perform common geospatial analysis tasks using DEA and open-source software
  4. Real_world_examples: More complex workflows demonstrating how DEA can be used to address real-world problems
  5. Scientific_workflows: Production code and notebooks for generating published DEA products or analysis tools

Supporting functions and data for the notebooks are kept in the following directories:

  • Tools: Python module dea-tools, containing functions and algorithms developed to assist in analysing DEA data (e.g. loading data, plotting, spatial analysis, machine learning)
  • Supplementary_data: Supplementary files required for the analyses above (e.g. images, rasters, shapefiles, training data)

The Jupyter notebooks on the stable branch of this repository are used to generate the Digital Earth Australia User Guide located at: https://docs.dea.ga.gov.au/

All notebooks in the dea-notebooks repository contain tags describing their functionality. If you are searching for a specific functionality, use the Tags Index to search for a suitable example. If there is a functionality that has not been documented that you think should be, please create an 'Issue' in the dea-notebooks repository.


Getting started with DEA Notebooks

To get started with using dea-notebooks, visit the DEA Notebooks Wiki page. This page includes guides for getting started on both the DEA Sandbox and NCI environments.

Once you're set up, there are two main options for interacting with dea-notebooks and contributing back to the repository:

  • DEA notebooks using git: Git is a version-control software designed to help track changes to files and collaborate with multiple users on a project. Using git is the recommended workflow for working with dea-notebooks as it makes it easy to stay up to date with the latest versions of functions and code and makes it impossible to lose your work.
  • DEA notebooks using Github: Alternatively, the Github website can be used to upload and modify the dea-notebooks repository directly. This can be a good way to get started with dea-notebooks.

Contributing to DEA Notebooks

Develop, stable and working branches

The dea-notebooks repository uses 'branches' to manage individuals' notebooks, and to allow easy publishing of notebooks ready to be shared. There are two main types of branches:

  • develop branch: The develop branch is the default branch where notebooks are put as they are being prepared to be shared publicly. Notebooks added to this branch will be periodically merged into the stable branch after testing and evaluation. The develop branch is protected and requires changes to be approved via a 'pull request' and review checklist before they appear on the branch.
  • stable branch: The stable branch contains DEA's collection of publicly available notebooks. Notebooks added to this branch will become part of the official DEA documentation and are published on the DEA User Guide. The stable branch is protected, and is periodically updated with new content from the develop branch via a 'pull request' (for develop > stable pull requests, merge using the 'Create a merge commit' option).
  • Working branches: All other branches in the repository are working spaces for users of dea-notebooks. They have a unique name (typically named after the user, e.g. ClaireK, BexDunn). The notebooks on these branches can be works-in-progress and do not need to be pretty or complete. By using a working branch, it is easy to use scripts and algorithms from dea-notebooks in your own work or share and collaborate on a working version of a notebook or code.

Publishing notebooks to the stable branch

Once you have a notebook that is ready to be published on the develop branch, you can submit a 'pull request' in the Pull requests tab at the top of the repository. The default pull request template contains a check-list to ensure that all stable branch Jupyter notebooks are consistent and well-documented so they can be understood by future users, and rendered correctly in the DEA User Guide. Please ensure that as many of these checklist items are complete as possible or leave a comment in the pull request asking for help with any remaining checklist items.

Draft pull requests

For pull requests you would like help with or that are a work in progress, consider using Github's draft pull request feature. This indicates that your work is still a draft, allowing you to get feedback from other DEA users before it is published on the ``develop` branch.

DEA Notebooks template notebook

A template notebook has been developed to make it easier to create new notebooks that meet all the pull request checklist requirements. The template notebook contains a simple structure and useful general advice on writing and formatting Jupyter notebooks. The template can be found here: DEA_notebooks_template.ipynb

Using the template is not required for working branch notebooks but is highly recommended as it will make it much easier to publish any notebooks on develop in the future.

Approving pull requests

Anyone with admin access to the dea-notebooks repository can approve 'pull requests'. You can see a list of the 'pull requests' ready for review in the Pull requests tab at the top of the repository. Click this tab, then click on the open pull request. You will need to review the code before you can approve the request. Ensure that all items in the pull request checklist have been ticked off and incorporated into the notebook. To make changes to someone else's pull request directly, first check out the branch you want to edit (e.g. pull_request_branch):

git pull
git checkout --track origin/pull_request_branch

Commit and push any changes you make, which will become part of the open pull request.

If the notebook meets all the checklist requirements, click the green 'Review' button and click 'Approve' (with an optional comment). You can also 'Request changes' here if any of the checklist items are not complete.

Once the pull request has been approved, you can merge it into the develop branch. Select the 'Squash and merge' option from the drop-down menu to the right of the green 'merge' button. Once you have merged the new branch in, you need to delete the branch. There is a button on the page that asks you if you would like to delete the now merged branch. Select 'Yes' to delete it.

Comments
  • Implement automated notebook testing

    Implement automated notebook testing

    As this repository continues to grow, a larger and larger amount of time is going to be spent re-running notebooks and verifying that all our functionality still works after every change to the DEA environments and our scripts/functions.

    This seems like a natural fit for some kind of automated testing, e.g. to automatically re-run all notebooks after changes are pushed to develop to verify that updates haven't broken any notebooks before we ultimately update master.

    ~~Similarly, being able to automatically re-build a test version of the DEA User Guide from develop would also greatly assist in reducing formatting errors and re-writes required to generate the documentation.~~ (added as of #719 )

    This issue is to capture discussion around the practicality of implementing automated testing on this repository. Any thoughts would be greatly appreciated!

    new feature/enhancement discussion 
    opened by robbibt 33
  • Convert Scripts folder to `dea_tools` Python module

    Convert Scripts folder to `dea_tools` Python module

    This PR adds a dea_tools module to Tools/, which is installable with pip. It also updates every notebook to use that module instead of inserting Scripts/ into the path.

    This is not meant to be a breaking change, so every notebook should still work. I ran every notebook that I could (a handful were too slow or too memory-hungry), so I have confirmed that they still work where possible.

    The only breaking change that could happen is on the Sandbox, where user modules don't persist. The Sandbox will need to be updated to pip install this module from the most recent version (or cached version?) on load.

    I don't want to merge this PR until we have resolved the Sandbox question (@Kirill888 may be the person to ask?), but I do want to kick-start the process of moving things over.

    As discussed on Slack with @cbur24, once we have a functioning module for DEA, we can pretty easily build a DEAfrica component with a dea_tools dependency.


    Here's what the old method of loading DEA scripts looked like:

    import sys
    sys.path.insert(1, '../Scripts')  # or ../../Scripts etc
    
    from dea_plotting import rgb
    from dea_spatialtools import xr_rasterize
    

    New method that I introduce here:

    from dea_tools.plotting import rgb
    from dea_tools.spatial import xr_rasterize
    
    new feature/enhancement 
    opened by MatthewJA 30
  • Add new Collection 3 Fractional Cover notebook to DEA Datasets

    Add new Collection 3 Fractional Cover notebook to DEA Datasets

    Proposed changes

    This PR adds a brand new Collection 3 Fractional Cover notebook, explaining how to inspect the available FC products and measurements, how to load data, and a simple best-practice example of analysing the data (including cloud masking from WO):

    https://github.com/GeoscienceAustralia/dea-notebooks/blob/c3_dea_datasets/DEA_datasets/DEA_Fractional_Cover.ipynb

    The format of the notebook is intended to match the other "DEA *" notebooks in DEA datasets. All content in the intro section is taken directly from CMI.

    I'd love a quick review for typos/formatting/usability, but particularly that I've characterised the product and best approaches for loading/analysing it - if you have a chance before Friday that would be amazing!

    Closes issues (optional)

    • Closes Issue #667

    Checklist (replace [ ] with [x] to check off)

    • [X] Notebook created using the DEA-notebooks template
    • [X] Remove any unused Python packages from Load packages
    • [X] Remove any unused/empty code cells
    • [X] Remove any guidance cells (e.g. General advice)
    • [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [X] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    opened by robbibt 25
  • Add notebook. Notebook updated from MDBA tag in this repo

    Add notebook. Notebook updated from MDBA tag in this repo

    Proposed changes

    This notebook was originally written by @katedugdale. I have updated it for inclusion in the stable branch of dea-notebooks.

    Checklist (replace [ ] with [x] to check off)

    • [x] Notebook created using the DEA-notebooks template
    • [x] Remove any unused Python packages from Load packages
    • [x] Remove any unused/empty code cells
    • [x] Remove any guidance cells (e.g. General advice)
    • [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    hacktoberfest-accepted 
    opened by CEKrause 24
  • Sentinel 2 waterbodies notebook

    Sentinel 2 waterbodies notebook

    Here's a notebook that generates polygons with Sentinel 2 MNDWI, Landsat 8 MNDWI, and WOfS, and compares their time series.

    I haven't a) run it in sequential order, or b) run it on the NCI, because it takes ages to run and I want to make sure it's feature complete first.

    Checklist (replace [ ] with [x] to check off)

    • [x] Notebook created using the DEA-notebooks template
    • [x] Remove any unused Python packages from Load packages
    • [x] Remove any unused/empty code cells
    • [x] Remove any guidance cells (e.g. General advice)
    • [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    hacktoberfest-accepted 
    opened by MatthewJA 22
  • Refactor `load_ard` to be more

    Refactor `load_ard` to be more "dask friendly"

    Background

    #526 discusses issues with the current implementation of load_ard that can cause performance issues in combination with dask due to loading individual products then stacking them with xr.concatenate (potentially running into problems with mis-aligned chunks and peak memory issues).

    Proposed changes

    This PR aims to re-factor load_ard using the following workflow:

    1. Uses dc.find_datasets() across all products first, building up a simple list of datasets named dataset_list
    2. Uses dc.load(datasets=dataset_list, ...) to load datasets across all products in one go
    3. Masks data without converting to floats until the final step using tools from odc.algo, avoiding issues with high peak memory usage

    This change has greatly simplified the function code, and has also enabled some potentially exciting new features, including the new filter_func parameter. This lets you pass in a custom function which filters out individual datasets prior to them even being loaded using dc.load. For example, to load only summer months from multiple years:

    query = {
        'x': (153.38, 153.47),
        'y': (-28.83, -28.92),
        'time': ('2015', '2018'),
        'measurements': ['nbart_red', 'nbart_green', 'nbart_blue'],
        'output_crs': 'EPSG:3577',
        'resolution': (-30, 30),
        'group_by': 'solar_day',
        'dask_chunks': {}
    }
    
    def month_filter(dataset, months=[12, 1, 2]):    
        # Return True if dataset month is in list of months
        return dataset.time.begin.month in months
    
    ds = load_ard(dc=dc,
                  products=['ga_ls5t_ard_3', 'ga_ls7e_ard_3', 'ga_ls8c_ard_3'],
                  filter_func=month_filter,
                  **query)
    

    Closes issues (optional)

    • Closes Issue #526
    new feature/enhancement load_ard 
    opened by robbibt 21
  • Add DEA Landsat Collection 3 notebook

    Add DEA Landsat Collection 3 notebook

    Proposed changes

    This PR adds a much over-due DEA Datasets notebook that demonstrates how to load data from DEA Landsat Collection 3 Surface Reflectance products using the Digital Earth Australia datacube. Topics covered include:

    • Inspecting the Landsat products and measurements available in the datacube
    • Loading Landsat data for specific location and time
    • Plotting Landsat data in true and false colour
    • Applying a cloud mask using the oa_fmask band
    • Advanced: Loading data in its native projection and resolution

    Most of the markdown is based directly on CMI, and I've tried to direct users back towards CMI for more info at every opportunity.

    Closes issues (optional)

    • Closes Issue #665

    Checklist (replace [ ] with [x] to check off)

    • [X] Notebook created using the DEA-notebooks template
    • [X] Remove any unused Python packages from Load packages
    • [X] Remove any unused/empty code cells
    • [X] Remove any guidance cells (e.g. General advice)
    • [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    opened by robbibt 18
  • Adding Sandbox notebooks

    Adding Sandbox notebooks

    This pull request will add notebooks developed for the DEA Sandbox to the general notebook repository for DEA.

    Still to do:

    • test notebooks on GA-hosted sandbox to validate indexing
    • modify notebook content/formating to fit with needs of GA-hosted sandbox (e.g. dask_average_pixel_example.ipynb needs explanatory text).

    Happy to take any other feedback and update as necessary! I'll leave this as a draft PR until we're ready.

    opened by caitlinadams 16
  • Add ARD overpass notebook + supplementary data

    Add ARD overpass notebook + supplementary data

    Supplementary data for ARD_overpass_predictor notebook in dea-notebooks/Frequently_used_code/

    Proposed changes

    Supplementary data for ARD_overpass_predictor notebook in dea-notebooks/Frequently_used_code/

    Checklist (replace [ ] with [x] to check off)

    • [x] Notebook created using the DEA-notebooks template
    • [x] Remove any unused Python packages from Load packages
    • [x] Remove any unused/empty code cells
    • [x] Remove any guidance cells (e.g. General advice)
    • [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    opened by Eric-git-999 15
  • Added WOfS errors notebook

    Added WOfS errors notebook

    Proposed changes

    Added a notebook that estimates WOfS errors and probabilities as part of the waterbodies toolkit - thinking of using this for waterbody edges later on.

    Checklist (replace [ ] with [x] to check off)

    • [x] Notebook created using the DEA-notebooks template
    • [x] Remove any unused Python packages from Load packages
    • [x] Remove any unused/empty code cells
    • [x] Remove any guidance cells (e.g. General advice)
    • [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    hacktoberfest-accepted 
    opened by MatthewJA 15
  • Compositing notebooks and functions

    Compositing notebooks and functions

    Proposed changes

    Started redoing the xr_geomedian notebook and then decided to port over from DE Africa all things image compositing. Two new notebooks: Generating geomedian composites and Generating composites; five new functions for dea_datahandling: calc_geomedian, first, last, nearest, _select_along_axis

    I haven't run these on the NCI.

    Checklist (replace [ ] with [x] to check off)

    • [X] Notebook created using the DEA-notebooks template
    • [X] Remove any unused Python packages from Load packages
    • [X] Remove any unused/empty code cells
    • [X] Remove any guidance cells (e.g. General advice)
    • [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    opened by cbur24 15
  • pip doesnt support in-tree-build as a choice for --use-feature

    pip doesnt support in-tree-build as a choice for --use-feature

    https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/.github/workflows/test_notebooks.yml#L31

    (base) ubuntu@:~/dea-sandbox$ pip --use-feature=in-tree-build
    
    Usage:   
      pip <command> [options]
    
    option --use-feature: invalid choice: 'in-tree-build' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')
    
    bug 
    opened by pindge 1
  • Update load_ard to support S2 C3 data, including s2cloudless

    Update load_ard to support S2 C3 data, including s2cloudless

    Proposed changes

    Work in progress...

    Closes issues (optional)

    • Closes Issue #000
    • Closes Issue #000

    Checklist (replace [ ] with [x] to check off)

    • [x] Notebook created using the DEA-notebooks template
    • [x] Remove any unused Python packages from Load packages
    • [x] Remove any unused/empty code cells
    • [x] Remove any guidance cells (e.g. General advice)
    • [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
    • [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
    • [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
    • [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
    • [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with
    opened by robbibt 1
  • Add S2 C3 notebook

    Add S2 C3 notebook

    Add a new Sentinel-2 notebook which contains new up-to-date information, including the background from the Sentinel-2 Collection 3 Content Management Interface (CMI) pages.

    To flag:

    • Are we removing the old Sentinel-2 notebook (DEA_datasets/Sentinel_2.ipynb)? I have not included the removal of the old notebook as part of this PR.
    • Sentinel-2B CMI links are not publicly accessible yet. Aaron followed up this morning as they are currently pending approval.
    • @robbibt will add s2cloudless as an option for Sentinel-2 in the load_ard() function.
    Sentinel-2 Collection 3 
    opened by geoscience-aman 2
  • Document the new default CRS/resolution

    Document the new default CRS/resolution "load hints" functionality

    This PR adds load hints to our products that will allow users to load them into a default CRS and resolution without having to supply them manually using output_crs and resolution: https://github.com/GeoscienceAustralia/dea-config/pull/1117

    We should document this in DEA Notebooks/user guide so that this change does not catch users unawares.

    documentation 
    opened by robbibt 0
  • Add new

    Add new "DEA datasets" notebook for Sentinel-2 Collection 3 data

    The DEA datasets folder of DEA Notebooks includes standardised notebooks that introduce each of our products and how to get started with them using the Datacube: https://docs.dea.ga.gov.au/notebooks/DEA_datasets/README.html

    These notebooks include a set of standard sections to make it easy for users to follow along and compare between products:

    1. Background: Taken directly from CMI 2. Description: A brief summary of what is included in the notebook 3. Getting started: Some setup steps like loading Python packages and connecting to the datacube 4. Available products and measurements: Inspecting the data using dc.list_products() and dc.list_mesurements() 5. Loading data: Simplest/minimal example of loading the data and plotting it 6. Example application: A small "demo" using the product, e.g. designed to show how it could be loaded and manipulated in a realistic workflow (e.g. loading the data, applying a cloud mask etc), and highlighting any common "gotchas" or things users should be aware of when using it

    We should create a similar notebook for Sentinel-2 Collection 3, showing users what products and bands are available, how to load and plot the data, and perhaps how to use some of its special features (e.g. s2cloudless, filtering by GQA metadata, maturity etc).

    You can see some nice examples of DEA dataset notebooks here; these could be a nice template for this work:

    • https://docs.dea.ga.gov.au/notebooks/DEA_datasets/DEA_Water_Observations.html
    • https://docs.dea.ga.gov.au/notebooks/DEA_datasets/DEA_Fractional_Cover.html

    There is an ancient Sentinel-2 notebook here that may be useful for this, but it doesn't match the formatting of the other DEA dataset notebooks... we might be better off starting from scratch rather than adapting this

    Sentinel-2 Collection 3 
    opened by robbibt 0
  • Update at least some Sentinel-2 Collection 3 notebooks to use cloud masking from `s2cloudless`

    Update at least some Sentinel-2 Collection 3 notebooks to use cloud masking from `s2cloudless`

    We should update this repository to include at least some nice examples of applying cloud masking using the new s2cloudless_mask band. This will provide users with code to get started with this new cloud mask.

    An optional extra could be a simple notebook in Frequently_used_code that shows how to load and manipulate the quantitative s2cloudless_prob cloud probability layer (and any cool application of using it that we can think up)

    Sentinel-2 Collection 3 
    opened by robbibt 0
Releases(0.2.7)
  • 0.2.7(Aug 8, 2022)

    What's Changed

    • Fix missing function import in pixel_tides by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/940

    PyPI

    https://pypi.org/project/dea-tools/0.2.7/

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.6...0.2.7

    Source code(tar.gz)
    Source code(zip)
  • 0.2.6(Aug 5, 2022)

    New features

    Real_world_examples/Generating_satellite_animations.ipynb

    • An interactive "app" notebook for exporting time series satellite imagery animations by drawing a box on a map and selecting a custom time period.

    https://user-images.githubusercontent.com/17680388/183581425-bd015371-8ec6-49d0-a035-1f64f98c7da2.mp4

    dea_tools.coastal

    • dea_tools.coastal.model_tides: A replacement for the existing tidal_tag func that uses the open-source FES2014 tidal model.
    • dea_tools.coastal.pixel_tides: A more advanced tidal model function that models tides across the 2D extent of a satellite imagery dataset, then interpolates that data into each pixel of the dataset. Used in DEA Coastlines to create seamless tidal composites and shorelines (with no discontinuities along study area boundaries).

    dea_tools.datahandling

    • dea_tools.datahandling.parallel_apply: A function for applying a custom function in parallel along a specific dimension in an xarray (e.g. for each timestep), using multiprocessing from the concurrent.futures package

    dea_tools.spatial

    • dea_tools.spatial.hillshade: A function for generating hillshades on DEM data, using custom sun angles and azimuths.
    • dea_tools.spatial.sun_angles: A function for extracting average sun angles and azimuths from DEA Collection 3 metadata for each timestep in a satellite dataset. Can be used to identify pixels that are likely to be affected by terrain shadow in surface reflectance data.

    What's Changed (full changelog)

    • Fix failed Dask worker pod launch due to invalid username characters by @jmettes in https://github.com/GeoscienceAustralia/dea-notebooks/pull/928
    • New generating animations interactive notebook by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/927
    • Waterbodies fix by @vnewey in https://github.com/GeoscienceAustralia/dea-notebooks/pull/932
    • Move datacube import lower in STAC notebook to reduce confusion by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/933
    • remove unused import of write_geotiff by @ErinKenna in https://github.com/GeoscienceAustralia/dea-notebooks/pull/930
    • Add missing rasterstats and geopy dependency by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/934
    • Add funcs from DEA Coastlines and DEA Intertidal to DEA Tools by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/936
    • fix to Surface Area Duration NB by @vnewey in https://github.com/GeoscienceAustralia/dea-notebooks/pull/937

    New Contributors

    • @jmettes made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/928
    • @vnewey made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/932
    • @ErinKenna made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/930

    PyPI

    https://pypi.org/project/dea-tools/

    Source code(tar.gz)
    Source code(zip)
  • 0.2.5(Jun 8, 2022)

    What's Changed

    • Add ability to plot a colour bar on level 4 animation by @Kooie-cate in https://github.com/GeoscienceAustralia/dea-notebooks/pull/915
    • Update notebooks to use C3 derivatives by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/917
    • Optimise test github action by @pindge in https://github.com/GeoscienceAustralia/dea-notebooks/pull/920
    • Unpin index image for docker-compose by @pindge in https://github.com/GeoscienceAustralia/dea-notebooks/pull/921
    • Remove ignore from basic analysis notebook tests by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/922
    • Support for FES2014 tide model via CNES aviso-fes. by @rtaib in https://github.com/GeoscienceAustralia/dea-notebooks/pull/919
    • Pin pip to allow build Action to work by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/924

    New Contributors

    • @pindge made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/920
    • @rtaib made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/919

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.4...0.2.5

    Source code(tar.gz)
    Source code(zip)
  • 0.2.4(Apr 13, 2022)

    What's Changed

    Updates to ML notebooks/scripts, load_ard, and phenology

    • Update Polygon_drill.ipynb by @CEKrause in https://github.com/GeoscienceAustralia/dea-notebooks/pull/897
    • DEA Land Cover notebook suite by @bellemae in https://github.com/GeoscienceAustralia/dea-notebooks/pull/898
    • Update animation notebook to use MP4s by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/903
    • Add Fractional Cover reprocessing notebook and results by @BexDunn in https://github.com/GeoscienceAustralia/dea-notebooks/pull/905
    • Add DEA training to repo by @BexDunn in https://github.com/GeoscienceAustralia/dea-notebooks/pull/906
    • Update README.rst by @BexDunn in https://github.com/GeoscienceAustralia/dea-notebooks/pull/907
    • Ah, add my conf paper by @BexDunn in https://github.com/GeoscienceAustralia/dea-notebooks/pull/908
    • Remove spatial cross validation from Scalable ML by @cbur24 in https://github.com/GeoscienceAustralia/dea-notebooks/pull/909
    • Update phenology notebook and script by @cbur24 in https://github.com/GeoscienceAustralia/dea-notebooks/pull/910
    • Add morphological filtering to load_ard by @cbur24 in https://github.com/GeoscienceAustralia/dea-notebooks/pull/912
    • Conference contribution GeoPython by @sebfoe in https://github.com/GeoscienceAustralia/dea-notebooks/pull/914

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.3...0.2.4

    Source code(tar.gz)
    Source code(zip)
  • 0.2.3(Feb 17, 2022)

    What's Changed

    • Updated image export notebook to an interactive app that requires no coding by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/894

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.2...0.2.3

    Source code(tar.gz)
    Source code(zip)
  • 0.2.2(Jan 28, 2022)

    What's Changed

    • Add new func for plotting fmask data with automatic colours by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/886
    • Update to document new Coastlines paper and use recently added Dask and Geopandas functionality by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/887
    • Updated courses and training by @Bethany-M in https://github.com/GeoscienceAustralia/dea-notebooks/pull/889
    • Add coastlines paper to repo usage by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/890
    • Add new interactive DEA Coastlines app and move all widgets to dea_tools.app submodule by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/892

    New Contributors

    • @Bethany-M made their first contribution in https://github.com/GeoscienceAustralia/dea-notebooks/pull/889

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.1...0.2.2

    Source code(tar.gz)
    Source code(zip)
  • 0.2.1(Oct 15, 2021)

    What's Changed

    • Notebook for exporting high quality satellite images from an interactive map by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/870
    • Add new functionality to Image export notebook and functions by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/873
    • Fix image export for Sentinel-2 NRT by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/876
    • Add first set of basic Testbook tests to DEA Notebooks by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/872
    • Add option for returning dask client by @cbur24 in https://github.com/GeoscienceAustralia/dea-notebooks/pull/882
    • Remove in-text references to Scripts by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/883
    • Fix broken tidal_tag function in dea_tools.coastal by @robbibt in https://github.com/GeoscienceAustralia/dea-notebooks/pull/884

    Full Changelog: https://github.com/GeoscienceAustralia/dea-notebooks/compare/0.2.0...0.2.1

    Source code(tar.gz)
    Source code(zip)
  • 0.2.0(Sep 10, 2021)

    Changes

    • Updated all notebooks for compatibility with the recently updated Python 3.8 DEA Sandbox environment.
    • Updated the DEA Notebooks repository to remove the Scripts directory, and load functions from the dea-tools Python module instead (located in the Tools directory of the repository). If you previously imported functions from Scripts using the following syntax:
    import sys
    sys.path.insert(1, '../Scripts')
    from dea_datahandling import load_ard
    

    Please replace this with:

    import sys
    sys.path.insert(1, '../Tools')
    from dea_tools.datahandling import load_ard
    

    Alternatively, dea-tools can now be installed directly from PyPI using:

    pip install dea-tools
    from dea_tools.datahandling import load_ard
    

    Note: If you previously accessed the DEA Sandbox environment, the Scripts folder will not be removed automatically. We recommend deleting it yourself by opening a terminal, then typing rm -r Scripts:

    image

    Source code(tar.gz)
    Source code(zip)
  • 0.1.1(Sep 3, 2021)

    Release of dea-tools onto PyPI: https://pypi.org/project/dea-tools/

    This allows functions and algorithms from dea-tools to be installed easily using:

    pip install dea-tools
    
    Source code(tar.gz)
    Source code(zip)
  • 0.1.0a1(Sep 2, 2021)

  • 0.1.0(Sep 2, 2021)

  • pre-dea-tools(Dec 1, 2020)

    1 December 2020 update: To make it easier to import DEA functions on the NCI, Sandbox as well as external repositories or platforms like DEAfrica, we are planning to build the modules and functions currently located in /Scripts into a pip-installable Python package (https://github.com/GeoscienceAustralia/dea-notebooks/issues/538, https://github.com/GeoscienceAustralia/dea-notebooks/pull/705). This release tags the last version of the develop branch prior to merging in these changes.

    Source code(tar.gz)
    Source code(zip)
  • notebooks_refresh(Jan 20, 2020)

    9 January 2020 update: This repository was recently upgraded to a new directory structure and set of improved and easier to use Jupyter notebooks and scripts. To access an archive of the previous version of the repository, visit the master_archive branch. Please note that notebooks and code in the master_archive branch are no longer maintained or supported.

    Source code(tar.gz)
    Source code(zip)
  • master_archive(Jan 7, 2020)

    On January 9th, dea-notebooks was upgraded to a new directory structure and a new and improved set of Jupyter notebooks and scripts. This release preserves the history of the repository prior to this change. Please note that notebooks and code in this release (and earlier) are no longer maintained or supported.

    Also archived in the master_archive branch: https://github.com/GeoscienceAustralia/dea-notebooks/tree/master_archive

    Source code(tar.gz)
    Source code(zip)
Owner
Geoscience Australia
Geoscience Australia
A Python interface between Earth Engine and xarray

eexarray A Python interface between Earth Engine and xarray Description eexarray was built to make processing gridded, mesoscale time series data quic

Aaron Zuspan 159 Dec 23, 2022
A simple python script that, given a location and a date, uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed on the command-line.

What does it do? Given a location and a date, it uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed

Caio 42 Nov 26, 2022
Calculate & view the trajectory and live position of any earth-orbiting satellite

satellite-visualization A cross-platform application to calculate & view the trajectory and live position of any earth-orbiting satellite in 3D. This

Space Technology and Astronomy Cell - Open Source Society 3 Jan 8, 2022
Python package for earth-observing satellite data processing

Satpy The Satpy package is a python library for reading and manipulating meteorological remote sensing data and writing it to various image and data f

PyTroll 882 Dec 27, 2022
Simple CLI for Google Earth Engine Uploads

geeup: Simple CLI for Earth Engine Uploads with Selenium Support This tool came of the simple need to handle batch uploads of both image assets to col

Samapriya Roy 79 Nov 26, 2022
A toolbox for processing earth observation data with Python.

eo-box eobox is a Python package with a small collection of tools for working with Remote Sensing / Earth Observation data. Package Overview So far, t

null 13 Jan 6, 2022
Get Landsat surface reflectance time-series from google earth engine

geextract Google Earth Engine data extraction tool. Quickly obtain Landsat multispectral time-series for exploratory analysis and algorithm testing On

Loïc Dutrieux 50 Dec 15, 2022
Open Data Cube analyses continental scale Earth Observation data through time

Open Data Cube Core Overview The Open Data Cube Core provides an integrated gridded data analysis environment for decades of analysis ready earth obse

Open Data Cube 410 Dec 13, 2022
A python package that extends Google Earth Engine.

A python package that extends Google Earth Engine GitHub: https://github.com/davemlz/eemont Documentation: https://eemont.readthedocs.io/ PyPI: https:

David Montero Loaiza 307 Jan 1, 2023
ESMAC diags - Earth System Model Aerosol-Cloud Diagnostics Package

Earth System Model Aerosol-Cloud Diagnostics Package This Earth System Model (ES

Pacific Northwest National Laboratory 1 Jan 4, 2022
A ninja python package that unifies the Google Earth Engine ecosystem.

A Python package that unifies the Google Earth Engine ecosystem. EarthEngine.jl | rgee | rgee+ | eemont GitHub: https://github.com/r-earthengine/ee_ex

null 47 Dec 27, 2022
Enable geospatial data mining through Google Earth Engine in Grasshopper 3D, via its most recent Hops component.

AALU_Geo Mining This repository is produced for a masterclass at the Architectural Association Landscape Urbanism programme. Requirements Rhinoceros (

null 4 Nov 16, 2022
A Python package for delineating nested surface depressions from digital elevation data.

Welcome to the lidar package lidar is Python package for delineating the nested hierarchy of surface depressions in digital elevation models (DEMs). I

Qiusheng Wu 166 Jan 3, 2023
Google maps for Jupyter notebooks

gmaps gmaps is a plugin for including interactive Google maps in the IPython Notebook. Let's plot a heatmap of taxi pickups in San Francisco: import g

Pascal Bugnion 747 Dec 19, 2022
Python tools for geographic data

GeoPandas Python tools for geographic data Introduction GeoPandas is a project to add support for geographic data to pandas objects. It currently impl

GeoPandas 3.5k Jan 3, 2023
Tools for the extraction of OpenStreetMap street network data

OSMnet Tools for the extraction of OpenStreetMap (OSM) street network data. Intended to be used in tandem with Pandana and UrbanAccess libraries to ex

Urban Data Science Toolkit 47 Sep 21, 2022
Record railway train route profile with GNSS tools

Train route profile recording with GNSS technology based on ARDUINO platform Project target Develop GNSS recording tools based on the ARDUINO platform

tomcom 1 Jan 1, 2022
This repository contains the scripts to derivate the ENU and ECEF coordinates from the longitude, latitude, and altitude values encoded in the NAD83 coordinates.

This repository contains the scripts to derivate the ENU and ECEF coordinates from the longitude, latitude, and altitude values encoded in the NAD83 coordinates.

Luigi Cruz 1 Feb 7, 2022
A public data repository for datasets created from TransLink GTFS data.

TransLink Spatial Data What: TransLink is the statutory public transit authority for the Metro Vancouver region. This GitHub repository is a collectio

Henry Tang 3 Jan 14, 2022