Fast methods to work with hydro- and topography data in pure Python.

Overview
https://codecov.io/gh/Deltares/pyflwdir/branch/main/graph/badge.svg?token=N4VMHJJAV3 Latest developers docs Latest PyPI version

PyFlwDir

Intro

PyFlwDir contains a series of methods to work with gridded DEM and flow direction datasets, which are key to many workflows in many earth siences. Pyflwdir supports several flow direction data conventions and can easily be extended to include more. The package contains some unique methods such as Iterative Hydrography Upscaling (IHU) method to upscale flow directions from high resolution data to coarser model resolution.

Pyflwdir is in pure python and powered by numba to keep it fast.

Featured methods

https://raw.githubusercontent.com/Deltares/pyflwdir/master/docs/_static/pyflwdir.png

  • flow directions from elevation data using a steepest gradient algorithm
  • strahler stream order
  • flow direction upscaling
  • (sub)basin delineation
  • pfafstetter subbasins delineation
  • classic stream order
  • height above nearest drainage (HAND)
  • geomorphic floodplain delineation
  • up- and downstream tracing and arithmetics
  • hydrologically adjusting elevation
  • upstream accumulation
  • vectorizing streams
  • many more!

Installation

We recommend installing PyFlwdir using conda or pip.

Install the package from conda using:

$ conda install pyflwdir -c conda-forge

Install the package from pip using:

$ pip install pyflwdir

In order to run the examples in the notebook folder some aditional packages to read and write raster and vector data, as well as to plot these data are required. A complete environment can be installed from the environment.yml file using:

$ conda env create -f environment.yml
$ pip install pyflwdir

Quickstart

The most common workflow to derive flow direction from digital elevation data and subsequent delineate basins or vectorize a stream network can be done in just a few lines of code.

To read elevation data from a geotiff raster file elevation.tif do:

import rasterio
with rasterio.open("elevation.tif", "r") as src:
    elevtn = src.read(1)
    nodata = src.nodata
    transform = src.transform
    crs = src.crs

Derive a FlwdirRaster object from this data:

import pyflwdir
flw = pyflwdir.from_dem(
    data=elevtn,
    nodata=src.nodata,
    transform=transform,
    latlon=crs.is_geographic,
)

Delineate basins and retrieve a raster with unique IDs per basin: Tip: This raster can directly be written to geotiff and/or vectorized to save as vector file with rasterio

basins = flw.basins()

Vectorize the stream network and save to a geojson file:

import geopandas as gpd
feat = flw.streams()
gdf = gpd.GeoDataFrame.from_features(feats, crs=crs)
gdf.to_file('streams.geojson', driver='GeoJSON')

Documentation

See docs for a many examples and a full reference API.

Development and Testing

Welcome to the pyflwdir project. All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. See CONTRIBUTING.rst for how we work.

Changes

See CHANGELOG.rst

Authors

See AUTHORS.txt

License

This is free software: you can redistribute it and/or modify it under the terms of the MIT License. A copy of this license is provided in LICENSE

Comments
  • use pyflwdir package to parse nextxy data of CaMa-Flood!

    use pyflwdir package to parse nextxy data of CaMa-Flood!

    Dear @DirkEilander ,

    This is Jiangchao Qiu, a Ph.D. student from Sun Yat-sen University, China. I am major in simulating storm surge using physical model.

    I know you by reading your paper : The effect of surge on riverine flood hazard and impact in deltas globally. Great works!

    Recently, my collaborator provides me some global modeling result using CaMa-Flood in 15 arcmin resolution. I am trying to analyze these result and make some visualization, especially the river network and river mouths along the coastlines.

    Firstly, I used the function of cmf_maps_io.py (in your compound_hotpots repositories) to transform the nextxy.bin file to nextxy.tif file glb_15min.zip

    Secondly, I want to use the pydlwdir package to visualize the river network, but some errors occurred when I parse the nextxy.tif file to
    the actionable common format.

    the following is the screensnaps image

    image

    I try to use the function of pyflwdir.FlwdirRaster and pyflwdir.from_array, but both failed, I don't know how to fix it, could you please help me check and give some suggestion about how to solve this problem.

    By the way, the environment of my package seems no problem, since all of the example in the notebook folder can be carried out smothly.

    Many thanks to you and looking forward to your reply.

    Best regards, Jiangchao

    opened by Jiangchao3 8
  • error in plotting the river mouths using the nextxy.bin dataset

    error in plotting the river mouths using the nextxy.bin dataset

    Hi @DirkEilander ,

    I want to identify all of the river mouth and make a plot using the 3 arcmin river network dataset nextxy.bin.

    But it is odd that the figure I plotted here seems reversed in the latitude. Could you please have a look? Many thanks.

    Best regards, Jiangchao

    my scripts:

    import pyflwdir import numpy as np

    bbox=[-180, -90, 180, 90] data, transform = pyflwdir.read_nextxy( fn='glb_03min/nextxy.bin', nrow=3600, ncol=7200, bbox=bbox)

    import xarray as xr xr.DataArray(data[0]).plot() image

    x = data[0] lon = np.linspace(-180,180,7200) lat = np.linspace(-90,90,3600) lon[np.where(x==-9)[1]],lat[np.where(x==-9)[0]]

    import geopandas as gpd from shapely.geometry import Point gdf = gpd.GeoDataFrame([Point(x, y) for x,y in zip(lon[np.where(x==-9)[1]], lat[np.where(x==-9)[0]])]) gdf.columns = ['geometry']

    world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres')) fig, ax = plt.subplots() gdf.plot(ax=ax,markersize=0.1, color='red') world.plot(ax=ax); image

    opened by Jiangchao3 7
  • Distance to river network

    Distance to river network

    My apologies for opening this issue as I don't know how else to do this, but I was wondering if the pyfldir toolkit can be used to calculate the distance to a river network? I've been desperately looking for an open-source tool to do this. So a method that uses a flow direction raster to calculate the flow distance to the nearest stream.

    opened by MauKruisheer 3
  • Helper function to get coordinates from linear indices

    Helper function to get coordinates from linear indices

    Feature request: This may be in the code but I haven't see it in the docs anywhere. The flow path trace function returns a list of pixel indices in the linear index format. A helper function to convert to the coordinate system of the raster would be helpful.

    opened by JasonDalton 3
  • User guide instructions for plotting import error

    User guide instructions for plotting import error

    Dear all,

    thanks for this package. I am running conda 4.11 on Win10 and have installed pyflwdir as instructed. When following the user guide instructions for plotting I stumble upon below error.

    Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:22:46) [MSC v.1916 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from utils import quickplot, colors, cm
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'utils'
    

    Any hints on how to alleviate the situation are highly welcome. Same counts for install under Ubuntu WSL.

    Kind regards

    Sebastian

    opened by schoeller 2
  • Numba error creating flow direction raster

    Numba error creating flow direction raster

    I think I have the error narrowed down to how the JIT compiler views either the rasterio object or the parameters sent to pyflwdir.from_dem() function. I have a simple example based on the Quickstart from the documentation. The error occurs when i try to create a flow direction raster from elevation read through rasterio.
    https://colab.research.google.com/drive/1engswQ4uaKB0UMMM4zt8YzC8wEAnK1o1?usp=sharing

    opened by JasonDalton 2
  • Streams

    Streams

    • FlwdirRaster.streams method includes a (zero-length) line at pits and adds option for trace direction in combination with segment end cells.
    • Moved accuflux method from FlwdirRaster to parent Flwdir class.
    opened by DirkEilander 1
  • increase short river lengths

    increase short river lengths

    Subgrid river lengths can locally be very short which slows down the model. Potentially we can redistribute the subgrid river lengths within a single river branch to speed up calculations without making a large computational error.

    method in flwdir.py

    • inputs: river length, min threshold

    numba jitted method in streams.py

    • input: idxs_ds; idxs_seq (sorted from down- to upstream); confluences (based on number of upstream cells), river lengths, min threshold

    suggested algorithm

    from up- to downstream:
       while not at next confluence or less than n cells:
          get lengths & indices
       average lengths
    
    assert that the total river length is the same
    
    opened by DirkEilander 0
  • unexpected results adjust_dem

    unexpected results adjust_dem

    np.random.seed(11)
    dem = np.random.rand(15, 10)
    flwdir = pyflwdir.from_dem(dem, outlets='min')
    dem1 = flwdir.dem_adjust(dem)
    np.all((dem1 - flwdir.downstream(dem1))>=0)
    >> False
    
    invalid 
    opened by DirkEilander 0
  • subgrid floodplain schematization

    subgrid floodplain schematization

    a method that maps subgrid cells to river cell (at model resolution) outflow points based on D8, sorts the HAND values (in future also manning?) and returns the contributing area for fixed HAND intervals

    opened by DirkEilander 0
  • Question: How can I create shapefiles of the upstream area of >10,000 points?

    Question: How can I create shapefiles of the upstream area of >10,000 points?

    Hi,

    I would like to create polygons of the contributing area for > 10,000 points. I have all information that I need for this, including a 30m DEM, a river network (LineString geodataframe file) and derived flow direction raster. All compiled using pyflwdir.

    I'm currently using the pyflwdir tool, but this takes around 5 minutes per point.. Has anyone any clues on how to approach this easier and quicker? Also, to insert the streams from a LineString geodataframe (instead of this flow.stream_order() > 4 command). This is my current script:

    with rasterio.open(flwdir_fn, "r") as src: flwdir = src.read(1) crs = src.crs flw = pyflwdir.from_array( flwdir, ftype="d8", transform=src.transform, latlon=crs.is_geographic, cache=True, )

    for point in geo_df["geometry"].head(): x = point.x y = point.y print(x, y) subbasins = flw.basins(xy=(x, y), streams=flw.stream_order() >= 4)

    Many many thanks!

    opened by MauKruisheer 2
  • Subcatchment routing hierarchy by landuse

    Subcatchment routing hierarchy by landuse

    Dear all,

    I am trying to resemble a functionality similar to GisToSWMM using pyflwdir. In my understanding of this algorithm a subcatchment hierarchy is determined considering landuse. Every (sub-)subcatchment contains only one landuse and routes either to another (sub-)subcatchment or an outlet. Outlets are given in the form of enlarged nodes, thus rasterized sink information derived from outlet nodes and DEM sinks.

    Reading through pyflwdir documentation I figure that the Pfafstetter method allows for deriving hierarchical subcatchment information. If understood correctly it permits to figure out a form of subcatchment routing, thus which subcatchment drains to another subcatchment or outlet.

    I am seeking ways to mask this routing method using landuse, so that every subcatchment contains only one landuse. Any ideas if and/or how to achieve this goal are highly appreciated.

    Kind regards

    Sebastian

    opened by schoeller 8
  • prettify IHU upscaling code

    prettify IHU upscaling code

    • streams array can be simplified, no need to save cell indices
    • lots of duplicate code between upscale_check and upscale_error methods
    • remove unused code
    • improve comments. consistent use of pixel vs cell in terminology.
    opened by DirkEilander 0
Releases(v0.5.6)
  • v0.5.6(Nov 15, 2022)

    New

    • FlwdirRaster.smooth_rivlen method to smooth river length with a moving window operation over a river network.

    Changed

    • Move to flit and pyproject.toml for installation and publication
    • drop support for python 3.7
    • update docs to Sphinx pydata style
    Source code(tar.gz)
    Source code(zip)
  • v0.5.5(Feb 16, 2022)

    New

    • read_nextxy method to read binary nextxy data

    Bugfix

    • Support -9 (river outlet at ocean) and -10 (inland river pit) pit values for nextxy data
    • Fix 'argmin of an empty sequence' error in dem_dig_d4

    Improved

    • improve gvf and manning estimates in river_depth method
    Source code(tar.gz)
    Source code(zip)
  • v0.5.4(Jan 18, 2022)

    Improved

    • prioritize non-boundary cells with same elevation over boundary cells in dem.fill_depressions #17

    Bugfix

    • fix dem_adjust method #16
    Source code(tar.gz)
    Source code(zip)
  • v0.5.3(Nov 18, 2021)

    Improved

    • add new idxs_pit argument to dem.fill_despressions

    Bugfix

    • min_rivdph argument was not always applied in FlwdirRaster.river_depth
    Source code(tar.gz)
    Source code(zip)
  • v0.5.2(Nov 17, 2021)

    New

    • Flwdir.river_depth for gradually varying flow (gvf) and manning river depth estimation
    • Flwdir.path method to get the indices of flow paths for vector flow directions

    Improved

    • FlwdirRaster.streams method includes a (zero-length) line at pits and adds option for trace direction in combination with segment end cells.
    • Moved accuflux method from FlwdirRaster to parent Flwdir class.
    • Additional how argument in fillnodata to indicate how to combine values at confluences. Min, max, sum and mean are supported.
    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(Oct 3, 2021)

  • v0.5(Sep 28, 2021)

    New

    • General Flwdir object for 1D vector based (instead of raster based) flow directions
    • flwdir.from_dataframe methods to derive a Flwdir object from a (Geo)DataFrame based on the row index and a column with downstream row indices.
    • dem.fill_depressions and pyflwdir.from_dem methods to derive flow directions from DEMs based on Wang & Lui (2015)
    • gis_utils.get_edge method to get a boolean mask of valid cells at the interface with nodata cells or the array edge.
    • gis_utils.spread2d method to spread valid values on a 2D raster with optional friction and mask rasters
    • FlwdirRaster.dem_dig_d4 method to adjust a DEM such that each cell has a 4D neighbor with equal or lower elevation.
    • FlwdirRaster.fillnodata method fill nodata gaps by propagating valid values up or downstream.
    • region.region_outlets method; which is also wrapped in the new FlwdirRaster.basin_outlets method
    • region.region_dissolve method to dissovle regions into their nearest neighboring region
    • FlwdirRaster.subbasins_areas method to derive subbasins based on a minimal area threshold

    Improved

    • added type="classis" for bottum-up stream order to FlwdirRaster.stream_order, default is type="strahler"
    • return subbasin outlet indices for all FlwdirRaster.subbasin* methods
    • improved subgrid slope method with optional lstsq regression based slope
    • FlwdirRaster.streams takes an optional idxs_out argument to derive stream vectors for unit catchments
    • FlwdirRaster.streams takes an optional max_len argument to split large segments into multiple smaller ones.
    • Using the new Flwdir object as common base of FlwdirRaster to share methods and properties
    • gis_utils.IDENTITY transform has North -> South orientation (yres < 0) instead of S->N orientation which is in line with flow direction rasters.
    • new restrict_strord argument in FlwdirRaster.moving_average and FlwdirRaster.moving_median methods to restrict the moving window to cells with same or larger stream order.

    Bugfix

    • strahler stream_order method gave incorrect results
    • basins.subbasins_pfafstetter reimplementation to fix mall functioning when jitted
    • FlwdirRaster.streams fix when called with optional min_sto argument

    Deprecated

    • FlwdirRaster.main_tributaries method is deprecated due to mallfunctioning when jitted
    • FlwdirRaster.inflow_idxs and FlwdirRaster.outflow_idxs
    Source code(tar.gz)
    Source code(zip)
Owner
Deltares
Deltares is an independent institute for applied research in the field of water and subsurface.
Deltares
The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"

Swin-Unet The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"(https://arxiv.org/abs/2105.05537). A validatio

null 869 Jan 7, 2023
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Code for "Intra-hour Photovoltaic Generation Forecasting based on Multi-source Data and Deep Learning Methods."

pv_predict_unet-lstm Code for "Intra-hour Photovoltaic Generation Forecasting based on Multi-source Data and Deep Learning Methods." IEEE Transactions

FolkScientistInDL 8 Oct 8, 2022
Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan

null 68 Dec 14, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation"

SharinGAN Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation" The official project we

Koutilya PNVR 23 Oct 19, 2022
Does MAML Only Work via Feature Re-use? A Data Set Centric Perspective

Does-MAML-Only-Work-via-Feature-Re-use-A-Data-Set-Centric-Perspective Does MAML Only Work via Feature Re-use? A Data Set Centric Perspective Installin

null 2 Nov 7, 2022
Implementing Graph Convolutional Networks and Information Retrieval Mechanisms using pure Python and NumPy

Implementing Graph Convolutional Networks and Information Retrieval Mechanisms using pure Python and NumPy

Noah Getz 3 Jun 22, 2022
A simplistic and efficient pure-python neural network library from Phys Whiz with CPU and GPU support.

A simplistic and efficient pure-python neural network library from Phys Whiz with CPU and GPU support.

Manas Sharma 19 Feb 28, 2022
A general and strong 3D object detection codebase that supports more methods, datasets and tools (debugging, recording and analysis).

ALLINONE-Det ALLINONE-Det is a general and strong 3D object detection codebase built on OpenPCDet, which supports more methods, datasets and tools (de

Michael.CV 5 Nov 3, 2022
Implementation of fast algorithms for Maximum Spanning Tree (MST) parsing that includes fast ArcMax+Reweighting+Tarjan algorithm for single-root dependency parsing.

Fast MST Algorithm Implementation of fast algorithms for (Maximum Spanning Tree) MST parsing that includes fast ArcMax+Reweighting+Tarjan algorithm fo

Miloš Stanojević 11 Oct 14, 2022
FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

This is the official implementation of our neural-network-based fast diffuse room impulse response generator (FAST-RIR) for generating room impulse responses (RIRs) for a given acoustic environment.

Anton Jeran Ratnarajah 89 Dec 22, 2022
Super-Fast-Adversarial-Training - A PyTorch Implementation code for developing super fast adversarial training

Super-Fast-Adversarial-Training This is a PyTorch Implementation code for develo

LBK 26 Dec 2, 2022
Fit Fast, Explain Fast

FastExplain Fit Fast, Explain Fast Installing pip install fast-explain About FastExplain FastExplain provides an out-of-the-box tool for analysts to

null 8 Dec 15, 2022
Scientific Computation Methods in C and Python (Open for Hacktoberfest 2021)

Sci - cpy README is a stub. Do expand it. Objective This repository is meant to be a ready reference for scientific computation methods. Do ⭐ it if yo

Sandip Dutta 7 Oct 12, 2022
Numerical Methods with Python, Numpy and Matplotlib

Numerical Bric-a-Brac Collections of numerical techniques with Python and standard computational packages (Numpy, SciPy, Numba, Matplotlib ...). Diffe

Vincent Bonnet 10 Dec 20, 2021
Source-to-Source Debuggable Derivatives in Pure Python

Tangent Tangent is a new, free, and open-source Python library for automatic differentiation. Existing libraries implement automatic differentiation b

Google 2.2k Jan 1, 2023