A Python package for delineating nested surface depressions from digital elevation data.

Overview

Welcome to the lidar package

Open In Colab image image image image image image image image DOI

lidar is Python package for delineating the nested hierarchy of surface depressions in digital elevation models (DEMs). It is particularly useful for analyzing high-resolution topographic data, such as DEMs derived from Light Detection and Ranging (LiDAR) data.

Citations

  • Wu, Q., (2021). lidar: A Python package for delineating nested surface depressions from digital elevation data. Journal of Open Source Software, 6(59), 2965, https://doi.org/10.21105/joss.02965
  • Wu, Q., Lane, C.R., Wang, L., Vanderhoof, M.K., Christensen, J.R., & Liu, H. (2019). Efficient Delineation of Nested Depression Hierarchy in Digital Elevation Models for Hydrological Analysis Using Level-Set Method. Journal of the American Water Resources Association. https://doi.org/10.1111/1752-1688.12689 (PDF)

Contents

Introduction

lidar is a Python package for delineating the nested hierarchy of surface depressions in digital elevation models (DEMs). In traditional hydrological modeling, surface depressions in a DEM are commonly treated as artifacts and thus filled and removed to create a depressionless DEM, which can then be used to generate continuous stream networks. In reality, however, surface depressions in DEMs are commonly a combination of spurious and actual terrain features. Fine-resolution DEMs derived from Light Detection and Ranging (LiDAR) data can capture and represent actual surface depressions, especially in glaciated and karst landscapes. During the past decades, various algorithms have been developed to identify and delineate surface depressions, such as depression filling, depression breaching, hybrid breaching-filling, and contour tree method. More recently, a level-set method based on graph theory was proposed to delineate the nested hierarchy of surface depressions. The lidar Python package implements the level-set method and makes it possible for delineating the nested hierarchy of surface depressions as well as elevated terrain features. It also provides an interactive Graphical User Interface (GUI) that allows users to run the program with minimal coding.

Statement of Need

The lidar package is intended for scientists and researchers who would like to integrate surface depressions into hydrological modeling. It can also facilitate the identification and delineation of depressional features, such as sinkholes, detention basins, and prairie potholes. The detailed topological and geometric properties of surface depressions can be useful for terrain analysis and hydrological modeling, including the size, volume, mean depth, maximum depth, lowest elevation, spill elevation, perimeter, major axis length, minor axis length, elongatedness.

State of the Field

Currently, there are a few open-source Python packages that can perform depression filling on digital elevation data, such as RichDEM and whitebox, the Python frontend for WhiteboxTools. However, there are no Python packages offering tools for delineating the nested hierarchy of surface depressions and catchments as well as simulating inundation dynamics. The lidar Python package is intended for filling this gap.

Key Features

  • Smoothing DEMs using mean, median, and Gaussian filters.
  • Extracting depressions from DEMs.
  • Filtering out small artifact depressions based on user-specified minimum depression size.
  • Generating refined DEMs with small depressions filled but larger depressions kept intact.
  • Delineating depression nested hierarchy using the level-set method.
  • Delineating mount nested hierarchy using the level-set method.
  • Computing topological and geometric properties of depressions, including size, volume, mean depth, maximum depth, lowest elevation, spill elevation, perimeter, major axis length, minor axis length, elongatedness, eccentricity, orientation, and area-bbox-ratio.
  • Exporting depression properties as a csv file.

Installation

lidar supports a variety of platforms, including Microsoft Windows, macOS, and Linux operating systems. Note that you will need to have Python 3.x (< 3.9) installed. Python 2.x is not supported. lidar is available on both PyPI and conda-forge. lidar has a GDAL dependency, which can be challenging to install using pip on Windows. Therefore, it is highly recommended to install lidar from the conda-forge channel. If you encounter any errors, please check the Dependencies section below.

Install from PyPI

To install lidar from PyPI, run this command in your terminal:

pip install lidar

Install from conda-forage

If you have Anaconda or Miniconda installed on your computer, you can create a fresh conda environment to install lidar:

conda create -n py38 python=3.8
conda activate py38
conda install lidar -c conda-forge

Upgrade lidar

If you have installed lidar before and want to upgrade to the latest version, you can run the following command in your terminal:

pip install -U lidar

If you use conda, you can update lidar to the latest version by running the following command in your terminal:

conda update lidar -c conda-forge

To install the development version from GitHub directly using Git, run the following code:

pip install git+https://github.com/giswqs/lidar

Dependencies

lidar's Python dependencies are listed in its requirements.txt file. In addition, lidar has a C library dependency: GDAL >=1.11.2. How to install GDAL in different operating systems will be explained below. More informaton about GDAL can be found here.

Linux

Debian-based Linux

The following commands can be used to install GDAL for Debian-based Linux distributions (e.g., Ubuntu, Linux Mint).

sudo add-apt-repository ppa:ubuntugis/ppa
sudo apt-get update
sudo apt-get install gdal-bin libgdal-dev

If you encounter any compiling errors, try the following commands.

sudo apt-get install --reinstall build-essential
sudo apt-get install python3-dev
pip install wheel
Pacman-based Linux

The following commands can be used to install GDAL for Pacman-based Linux distributions (e.g., Arch Linux, Manjaro). You might need to use sudo if you encounter permission errors.

sudo pacman -S yaourt --noconfirm
yaourt -S gdal --noconfirm
yaourt -S python-gdal --noconfirm

macOS

For a Homebrew based Python environment, do the following.

brew update
brew install gdal

Alternatively, you can install GDAL binaries from kyngchaos. You will then need to add the installed location /Library/Frameworks/GDAL.framework/Programs to your system path.

Windows

The instruction below assumes that you have installed Anaconda. Open Anaconda Prompt and enter the following commands to create a conda environment and install required packages

conda create -n py38 python=3.8
conda activate py38
conda install lidar -c conda-forge

When installing the lidar package, if you encounter an error saying Microsoft Visual C++ 14.0 is required, please follow the steps below to fix the error and reinstall lidar. More infomration can be found at this link Fix Python 3 on Windows error - Microsoft Visual C++ 14.0 is required.

  • Download Microsoft Build Tools for Visual Studio 2017
  • Double click to install the downloaded installer - Microsoft Build Tools for Visual Studio 2017.
  • Open Microsoft Build Tools for Visual Studio 2017
  • Select Workloads --> Visual C++ build tools and click the install button

Usage

Launch the interactive notebook tutorial for the lidar Python package with Google Colab now:

Open In Colab

A quick example

import os
import pkg_resources
from lidar import *

# identify the sample data directory of the package
package_name = 'lidar'
data_dir = pkg_resources.resource_filename(package_name, 'data/')

# use the sample dem. Change it to your own dem if needed
in_dem = os.path.join(data_dir, 'dem.tif')
# set the output directory
out_dir = os.getcwd()

# parameters for identifying sinks and delineating nested depressions
min_size = 1000      # minimum number of pixels as a depression
min_depth = 0.5      # minimum depth as a depression
interval = 0.3       # slicing interval for the level-set method
bool_shp = True      # output shapefiles for each individual level

# extracting sinks based on user-defined minimum depression size
out_dem = os.path.join(out_dir, "median.tif")
in_dem = MedianFilter(in_dem, kernel_size=3, out_file=out_dem)
sink_path = ExtractSinks(in_dem, min_size, out_dir)
dep_id_path, dep_level_path = DelineateDepressions(sink_path, 
                                                   min_size, 
                                                   min_depth, 
                                                   interval, 
                                                   out_dir, 
                                                   bool_shp)
print('Results are saved in: {}'.format(out_dir))

lidar GUI

lidar also provides a Graphical User Interface (GUI), which can be invoked using the following Python script:

import lidar
lidar.gui()

image

lidar toolbox for ArcGIS Pro

Toolbox interface

toolbox

toolbox_ui

Video tutorials

Delineating nested surface depressions and catchments using ArcGIS Pro

demo

Delineating nested surface depressions and catchments using ArcMap

demo

A real-world example

The images below show working examples of the level set method for delineating nested depressions in the Cottonwood Lake Study Area (CLSA), North Dakota. More test datasets (e.g., the Pipestem watershed in the Prairie Pothole Region of North Dakota) can be downloaded from http://gishub.org/2019-JAWRA-Data

The following example was conducted on a 64-bit Linux machine with a quad-core Intel i7-7700 CPU and 16 GB RAM. The average running time of the algorithm for this DEM was 0.75 seconds.

image

image

image

References

The level-set algorithm was proposed by Wu et al. (2019):

  • Wu, Q., Lane, C.R., Wang, L., Vanderhoof, M.K., Christensen, J.R., & Liu, H. (2019). Efficient Delineation of Nested Depression Hierarchy in Digital Elevation Models for Hydrological Analysis Using Level-Set Method. Journal of the American Water Resources Association. DOI: 10.1111/1752-1688.12689 (PDF)

Applications of the level-set and contour-tree methods for feature extraction from LiDAR data:

  • Wu, Q., & Lane, C.R. (2017). Delineating wetland catchments and modeling hydrologic connectivity using LiDAR data and aerial imagery. Hydrology and Earth System Sciences. 21: 3579-3595. DOI: 10.5194/hess-21-3579-2017
  • Wu, Q., Deng, C., & Chen, Z. (2016). Automated delineation of karst sinkholes from LiDAR-derived digital elevation models. Geomorphology. 266: 1-10. DOI: 10.1016/j.geomorph.2016.05.006
  • Wu, Q., Su, H., Sherman, D.J., Liu, H., Wozencraft, J.M., Yu, B., & Chen, Z. (2016). A graph-based approach for assessing storm-induced coastal changes. International Journal of Remote Sensing. 37:4854-4873. DOI: 10.1080/01431161.2016.1225180
  • Wu, Q., & Lane, C.R. (2016). Delineation and quantification of wetland depressions in the Prairie Pothole Region of North Dakota. Wetlands. 36(2):215–227. DOI: 10.1007/s13157-015-0731-6
  • Wu, Q., Liu, H., Wang, S., Yu, B., Beck, R., & Hinkel, K. (2015). A localized contour tree method for deriving geometric and topological properties of complex surface depressions based on high-resolution topographic data. International Journal of Geographical Information Science. 29(12): 2041-2060. DOI: 10.1080/13658816.2015.1038719
  • Wu, Q., Lane, C.R., & Liu, H. (2014). An effective method for detecting potential woodland vernal pools using high-resolution LiDAR data and aerial imagery. Remote Sensing. 6(11):11444-11467. DOI: 10.3390/rs61111444

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given. You can contribute in many ways:

Types of Contributions

Report Bugs

Report bugs at https://github.com/giswqs/lidar/issues.

If you are reporting a bug, please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Fix Bugs

Look through the GitHub issues for bugs. Anything tagged with "bug" and "help wanted" is open to whoever wants to implement it.

Implement Features

Look through the GitHub issues for features. Anything tagged with "enhancement" and "help wanted" is open to whoever wants to implement it.

Write Documentation

lidar could always use more documentation, whether as part of the official lidar docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback

The best way to send feedback is to file an issue at https://github.com/giswqs/lidar/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that contributions are welcome.

Get Started

Ready to contribute? Here's how to set up lidar for local development.

  1. Fork the lidar repo on GitHub.

  2. Clone your fork locally:

git clone [email protected]:your_name_here/lidar.git
  1. Install your local copy into a conda env. Assuming you have conda installed, this is how you set up your fork for local development:
conda create -n lidar-test python
conda activate lidar-test
cd lidar/
pip install -e .
  1. Create a branch for local development:
git checkout -b name-of-your-bugfix-or-feature

Now you can make your changes locally.

  1. When you're done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
flake8 lidar tests
python setup.py test or pytest

To get flake8 and tox, just pip install them into your conda env.

  1. Commit your changes and push your branch to GitHub:
git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
  1. Submit a pull request through the GitHub website.

Pull Request Guidelines

Before you submit a pull request, check that it meets these guidelines:

  1. The pull request should include tests.
  2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.md.
  3. The pull request should work for Python 3.7 and 3.8. Check https://github.com/giswqs/lidar/actions and make sure that the tests pass for all supported Python versions.

Credits

Comments
  • ValueError: Values other than

    ValueError: Values other than "rc" for the "coordinates" argument to skimage.measure.regionprops are no longer supported.

    • lidar version:
    • Python version: 3.7
    • Operating System: Windows 10

    Description

    I tried running the example.py with the dem.tif but got the following error:

    # identify the sample data directory of the package
    #package_name = 'lidar'
    #data_dir = pkg_resources.resource_filename(package_name, 'data/')
    data_dir = 'lidar'
    
    # use the sample dem. Change it to your own dem if needed
    in_dem = os.path.join(data_dir, 'dem.tif')
    # set output directory. By default, use the temp directory under user's home directory
    out_dir = os.path.join(os.path.expanduser("~"), "temp")
    
    # parameters for identifying sinks and delineating nested depressions
    min_size = 1000      # minimum number of pixels as a depression
    min_depth = 0.5      # minimum depth as a depression
    interval = 0.3       # slicing interval for the level-set method
    bool_shp = True     # output shapefiles for each individual level
    
    # extracting sinks based on user-defined minimum depression size
    out_dem = os.path.join(out_dir, "median.tif")
    in_dem = lidar.MedianFilter(in_dem, kernel_size=3, out_file=out_dem)
    sink_path = lidar.ExtractSinks(in_dem, min_size, out_dir)
    dep_id_path, dep_level_path = lidar.DelineateDepressions(sink_path, min_size, min_depth, interval, out_dir, bool_shp)
    
    print('Results are saved in: {}'.format(out_dir))
    
    Median filtering ...
    Run time: 0.0290 seconds
    Saving dem ...
    Loading data ...
    min = 379.70, max = 410.72, no_data = -3.402823e+38, cell_size = 1.0
    Depression filling ...
    Saving filled dem ...
    Region grouping ...
    Computing properties ...
    
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-13-d90bdedee5d6> in <module>
         18 out_dem = os.path.join(out_dir, "median.tif")
         19 in_dem = lidar.MedianFilter(in_dem, kernel_size=3, out_file=out_dem)
    ---> 20 sink_path = lidar.ExtractSinks(in_dem, min_size, out_dir)
         21 dep_id_path, dep_level_path = lidar.DelineateDepressions(sink_path, min_size, min_depth, interval, out_dir, bool_shp)
         22 
    
    ~\Anaconda3\envs\py37\lib\site-packages\lidar\filling.py in ExtractSinks(in_dem, min_size, out_dir)
        178 
        179     print("Computing properties ...")
    --> 180     objects = measure.regionprops(label_objects, dem, coordinates='xy')
        181     dep_list = get_dep_props(objects, cell_size)
        182     write_dep_csv(dep_list, out_csv_file)
    
    ~\Anaconda3\envs\py37\lib\site-packages\skimage\measure\_regionprops.py in regionprops(label_image, intensity_image, cache, coordinates)
        855                    'stop using the "coordinates" argument, or use skimage '
        856                    'version 0.15.x or earlier.')
    --> 857             raise ValueError(msg)
        858 
        859     regions = []
    
    ValueError: Values other than "rc" for the "coordinates" argument to skimage.measure.regionprops are no longer supported. You should update your code to use "rc" coordinates and stop using the "coordinates" argument, or use skimage version 0.15.x or earlier.
    
    
    opened by maryloooh 8
  • Memory Issue

    Memory Issue

    • lidar version:
    • Python version: 3.7
    • Operating System: Windows 10

    RuntimeError: Free disk space available is 49152 bytes, whereas 2705360268 are at least necessary. You can disable this check by defining the CHECK_DISK_FREE_SPACE configuration option to FALSE.

    How can I solve this? Thanks a lot!

    opened by maryloooh 5
  • Environment issue

    Environment issue

    • lidar version: newest
    • Python version: 3.7
    • Operating System: Windows 10

    Hi. First of all, I'm quite a novice using python, so this may be a very simply issue but I'm stuck despite all the forums I checked already!

    So, I installed the lidar package according to the instructions using Anaconda Prompt. It was successful. However, when I tried installing spyder inside that new environment it does not work. I thought I installed spyder successfully also in the new environment but cannot access it (see screenshots below). I can open spyder directly but then it is still in another environment and won't let me access the lidar package. Any thoughts?

    Thanks a lot!

    python_code

    opened by maryloooh 3
  • Bump pip from 19.2 to 21.1

    Bump pip from 19.2 to 21.1

    Bumps pip from 19.2 to 21.1.

    Changelog

    Sourced from pip's changelog.

    21.1 (2021-04-24)

    Process

    • Start installation scheme migration from distutils to sysconfig. A warning is implemented to detect differences between the two implementations to encourage user reports, so we can avoid breakages before they happen.

    Features

    • Add the ability for the new resolver to process URL constraints. ([#8253](https://github.com/pypa/pip/issues/8253) <https://github.com/pypa/pip/issues/8253>_)
    • Add a feature --use-feature=in-tree-build to build local projects in-place when installing. This is expected to become the default behavior in pip 21.3; see Installing from local packages <https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages>_ for more information. ([#9091](https://github.com/pypa/pip/issues/9091) <https://github.com/pypa/pip/issues/9091>_)
    • Bring back the "(from versions: ...)" message, that was shown on resolution failures. ([#9139](https://github.com/pypa/pip/issues/9139) <https://github.com/pypa/pip/issues/9139>_)
    • Add support for editable installs for project with only setup.cfg files. ([#9547](https://github.com/pypa/pip/issues/9547) <https://github.com/pypa/pip/issues/9547>_)
    • Improve performance when picking the best file from indexes during pip install. ([#9748](https://github.com/pypa/pip/issues/9748) <https://github.com/pypa/pip/issues/9748>_)
    • Warn instead of erroring out when doing a PEP 517 build in presence of --build-option. Warn when doing a PEP 517 build in presence of --global-option. ([#9774](https://github.com/pypa/pip/issues/9774) <https://github.com/pypa/pip/issues/9774>_)

    Bug Fixes

    • Fixed --target to work with --editable installs. ([#4390](https://github.com/pypa/pip/issues/4390) <https://github.com/pypa/pip/issues/4390>_)
    • Add a warning, discouraging the usage of pip as root, outside a virtual environment. ([#6409](https://github.com/pypa/pip/issues/6409) <https://github.com/pypa/pip/issues/6409>_)
    • Ignore .dist-info directories if the stem is not a valid Python distribution name, so they don't show up in e.g. pip freeze. ([#7269](https://github.com/pypa/pip/issues/7269) <https://github.com/pypa/pip/issues/7269>_)
    • Only query the keyring for URLs that actually trigger error 401. This prevents an unnecessary keyring unlock prompt on every pip install invocation (even with default index URL which is not password protected). ([#8090](https://github.com/pypa/pip/issues/8090) <https://github.com/pypa/pip/issues/8090>_)
    • Prevent packages already-installed alongside with pip to be injected into an isolated build environment during build-time dependency population. ([#8214](https://github.com/pypa/pip/issues/8214) <https://github.com/pypa/pip/issues/8214>_)
    • Fix pip freeze permission denied error in order to display an understandable error message and offer solutions. ([#8418](https://github.com/pypa/pip/issues/8418) <https://github.com/pypa/pip/issues/8418>_)
    • Correctly uninstall script files (from setuptools' scripts argument), when installed with --user. ([#8733](https://github.com/pypa/pip/issues/8733) <https://github.com/pypa/pip/issues/8733>_)
    • New resolver: When a requirement is requested both via a direct URL (req @ URL) and via version specifier with extras (req[extra]), the resolver will now be able to use the URL to correctly resolve the requirement with extras. ([#8785](https://github.com/pypa/pip/issues/8785) <https://github.com/pypa/pip/issues/8785>_)
    • New resolver: Show relevant entries from user-supplied constraint files in the error message to improve debuggability. ([#9300](https://github.com/pypa/pip/issues/9300) <https://github.com/pypa/pip/issues/9300>_)
    • Avoid parsing version to make the version check more robust against lousily debundled downstream distributions. ([#9348](https://github.com/pypa/pip/issues/9348) <https://github.com/pypa/pip/issues/9348>_)
    • --user is no longer suggested incorrectly when pip fails with a permission error in a virtual environment. ([#9409](https://github.com/pypa/pip/issues/9409) <https://github.com/pypa/pip/issues/9409>_)
    • Fix incorrect reporting on Requires-Python conflicts. ([#9541](https://github.com/pypa/pip/issues/9541) <https://github.com/pypa/pip/issues/9541>_)

    ... (truncated)

    Commits
    • 2b2a268 Bump for release
    • ea761a6 Update AUTHORS.txt
    • 2edd3fd Postpone a deprecation to 21.2
    • 3cccfbf Rename mislabeled news fragment
    • 21cd124 Fix NEWS.rst placeholder position
    • e46bdda Merge pull request #9827 from pradyunsg/fix-git-improper-tag-handling
    • 0e4938d :newspaper:
    • ca832b2 Don't split git references on unicode separators
    • 1320bac Merge pull request #9814 from pradyunsg/revamp-ci-apr-2021-v2
    • e9cc23f Skip checks on PRs only
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • lidar gui issue

    lidar gui issue

    It seems the latest version of the dependency (PySimpleGUI) breaks the lidar gui. As a temporary solution, you can try the following:

    • conda activate py37
    • pip uninstall pysimplegui -y
    • pip install pysimplegui==2.30.0
    • python
    • import lidar
    • lidar.gui()

    You can also use the package without the gui, check the tutorial below:

    https://github.com/giswqs/lidar/blob/master/examples/lidar_colab.ipynb

    I will fix this issue in the next release.

    bug 
    opened by giswqs 1
  • Love the GUI!

    Love the GUI!

    • lidar version:
    • Python version:
    • Operating System:

    Description

    Just wanted to comment on your use of PySimpleGUI. Thanks for giving it a try and mentioning it in your docs. It is a nice looking GUI. I'm going to post it in the user's screen shots in my GitHub.

    🥇 One of the best looking layouts I've seen.

    What I Did

    Paste the command(s) you ran and the output.
    If there was a crash, please include the traceback here.
    
    opened by MikeTheWatchGuy 1
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
  • Bump pip from 9.0.1 to 19.2

    Bump pip from 9.0.1 to 19.2

    Bumps pip from 9.0.1 to 19.2.

    Changelog

    Sourced from pip's changelog.

    19.2 (2019-07-22)

    Deprecations and Removals

    • Drop support for EOL Python 3.4. ([#6685](https://github.com/pypa/pip/issues/6685) <https://github.com/pypa/pip/issues/6685>_)
    • Improve deprecation messages to include the version in which the functionality will be removed. ([#6549](https://github.com/pypa/pip/issues/6549) <https://github.com/pypa/pip/issues/6549>_)

    Features

    • Credentials will now be loaded using keyring when installed. ([#5948](https://github.com/pypa/pip/issues/5948) <https://github.com/pypa/pip/issues/5948>_)
    • Fully support using --trusted-host inside requirements files. ([#3799](https://github.com/pypa/pip/issues/3799) <https://github.com/pypa/pip/issues/3799>_)
    • Update timestamps in pip's --log file to include milliseconds. ([#6587](https://github.com/pypa/pip/issues/6587) <https://github.com/pypa/pip/issues/6587>_)
    • Respect whether a file has been marked as "yanked" from a simple repository (see PEP 592 <https://www.python.org/dev/peps/pep-0592/>__ for details). ([#6633](https://github.com/pypa/pip/issues/6633) <https://github.com/pypa/pip/issues/6633>_)
    • When choosing candidates to install, prefer candidates with a hash matching one of the user-provided hashes. ([#5874](https://github.com/pypa/pip/issues/5874) <https://github.com/pypa/pip/issues/5874>_)
    • Improve the error message when METADATA or PKG-INFO is None when accessing metadata. ([#5082](https://github.com/pypa/pip/issues/5082) <https://github.com/pypa/pip/issues/5082>_)
    • Add a new command pip debug that can display e.g. the list of compatible tags for the current Python. ([#6638](https://github.com/pypa/pip/issues/6638) <https://github.com/pypa/pip/issues/6638>_)
    • Display hint on installing with --pre when search results include pre-release versions. ([#5169](https://github.com/pypa/pip/issues/5169) <https://github.com/pypa/pip/issues/5169>_)
    • Report to Warehouse that pip is running under CI if the PIP_IS_CI environment variable is set. ([#5499](https://github.com/pypa/pip/issues/5499) <https://github.com/pypa/pip/issues/5499>_)
    • Allow --python-version to be passed as a dotted version string (e.g. 3.7 or 3.7.3). ([#6585](https://github.com/pypa/pip/issues/6585) <https://github.com/pypa/pip/issues/6585>_)
    • Log the final filename and SHA256 of a .whl file when done building a wheel. ([#5908](https://github.com/pypa/pip/issues/5908) <https://github.com/pypa/pip/issues/5908>_)
    • Include the wheel's tags in the log message explanation when a candidate wheel link is found incompatible. ([#6121](https://github.com/pypa/pip/issues/6121) <https://github.com/pypa/pip/issues/6121>_)
    • Add a --path argument to pip freeze to support --target installations. ([#6404](https://github.com/pypa/pip/issues/6404) <https://github.com/pypa/pip/issues/6404>_)
    • Add a --path argument to pip list to support --target installations. ([#6551](https://github.com/pypa/pip/issues/6551) <https://github.com/pypa/pip/issues/6551>_)

    Bug Fixes

    • Set sys.argv[0] to the underlying setup.py when invoking setup.py via the setuptools shim so setuptools doesn't think the path is -c. ([#1890](https://github.com/pypa/pip/issues/1890) <https://github.com/pypa/pip/issues/1890>_)
    • Update pip download to respect the given --python-version when checking "Requires-Python". ([#5369](https://github.com/pypa/pip/issues/5369) <https://github.com/pypa/pip/issues/5369>_)
    • Respect --global-option and --install-option when installing from a version control url (e.g. git). ([#5518](https://github.com/pypa/pip/issues/5518) <https://github.com/pypa/pip/issues/5518>_)
    • Make the "ascii" progress bar really be "ascii" and not Unicode. ([#5671](https://github.com/pypa/pip/issues/5671) <https://github.com/pypa/pip/issues/5671>_)
    • Fail elegantly when trying to set an incorrectly formatted key in config. ([#5963](https://github.com/pypa/pip/issues/5963) <https://github.com/pypa/pip/issues/5963>_)
    • Prevent DistutilsOptionError when prefix is indicated in the global environment and --target is used. ([#6008](https://github.com/pypa/pip/issues/6008) <https://github.com/pypa/pip/issues/6008>_)
    • Fix pip install to respect --ignore-requires-python when evaluating links. ([#6371](https://github.com/pypa/pip/issues/6371) <https://github.com/pypa/pip/issues/6371>_)

    ... (truncated)

    Commits
    • 0e64295 Generate NEWS
    • 0df416d Bump version for release
    • 5f0aa2a Generate AUTHORS.txt
    • 8582f7e Reduce dependency on ctypes when discovering glibc version. (#6678)
    • e308497 Merge pull request #6743 from chrahunt/maint/remove-path-copytree
    • 9281a7a Remove copytree from tests.lib.path.Path.
    • 0d28601 Remove copy from tests.lib.path.Path. (#6746)
    • c275e9d Drop a useless import in favor of explicitness
    • 3732e79 Remove normpath from tests.lib.path.Path.
    • 358e690 Remove move from tests.lib.path.Path.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • ImportError: DLL load failed: The specified module could not be found.

    ImportError: DLL load failed: The specified module could not be found.

    • lidar version:
    • Python version:3.6
    • Operating System: windows 10

    Description

    Thank you so much for creating this wonderful tool. I am testing with ArcGIS 2.5 with your dataset. But I am getting errors. Is it related python environment or version? Would you check it, please?

    What I Did

    I am testing with your provided dataset.

    ERROR 032659 updateParameters Syntax Error: Traceback (most recent call last):
      File "C:\Users\hahmad\Downloads\lidar-master\lidar\toolbox\ArcGIS Pro Hydrology Analyst.tbx#ExtrackSink.UpdateParameters.py", line 1, in <module>
      File "c:\program files\arcgis\pro\Resources\arcpy\arcpy\__init__.py", line 21, in <module>
        import numpy
      File "C:\Users\hahmad\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\numpy\__init__.py", line 140, in <module>
        from . import _distributor_init
      File "C:\Users\hahmad\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1\lib\site-packages\numpy\_distributor_init.py", line 34, in <module>
        from . import _mklinit
    ImportError: DLL load failed: The specified module could not be found.
    
    opened by hafez-ahmad 2
  • Wrong  Delineate Flow Oath

    Wrong Delineate Flow Oath

    • lidar version:
    • Python version: 3.6.9
    • Operating System: Windows x64-Conda

    Description

    I am running the example you provide in the Lidar folder. When I run script 3 I get the following error:

    Traceback (most recent call last): File "C:\lidar\lidar\toolbox\scripts\3_Flow_Path.py", line 425, in FlowPath(in_dem, in_sink, rain_intensity, out_flowpath) File "C:\lidar\lidar\toolbox\scripts\3_Flow_Path.py", line 303, in FlowPath code_block="", File "c:\program files\arcgis\pro\Resources\arcpy\arcpy\management.py", line 4530, in CalculateField raise e File "c:\program files\arcgis\pro\Resources\arcpy\arcpy\management.py", line 4527, in CalculateField retval = convertArcObjectToPythonObject(gp.CalculateField_management(*gp_fixargs((in_table, field, expression, expression_type, code_block, field_type), True))) File "c:\program files\arcgis\pro\Resources\arcpy\arcpy\geoprocessing_base.py", line 511, in return lambda *args: val(*gp_fixargs(args, True)) arcgisscripting.ExecuteError: ERROR 000539: File "", line 1 0,05 ^ SyntaxError: invalid token

    Failed to execute (CalculateField). Failed to execute (DelineateFlowPath).

    What I Did

    Paste the command(s) you ran and the output.
    If there was a crash, please include the traceback here.
    

    Captura de pantalla (69) Captura de pantalla (68)

    opened by FernandaDavi 3
  • ID mismatch between depressions / regions in depressions_info

    ID mismatch between depressions / regions in depressions_info

    • lidar version: 0.6.1
    • Python version: 3.8
    • Operating System: Ubuntu

    Description

    The following image shows the mismatch in the depressions_info.csv after executing

    sink_path = ExtractSinks(
        dem_raster,
        2,
        out_dir
    )
    DelineateDepressions(
        sink_path,
        2,
        0,
        0.2,
        out_dir,
        True
    )
    

    mismatch

    The depressions_info.csv references the region_id 143 for depression 163 while a look at the map show that actually region 173 spatially matches the depression 163. I tested in another region and here everything was correct and the region/depression IDs matched as they should. I expect this to be a bug. Any other suggestions?

    PS: I attached the SRTM image to test srtm_dem.zip

    opened by geotom 4
  • "min_depth" parameter ignored in Depression/level_set method

    • lidar version: 0.6.1
    • Python version: 3.8
    • Operating System: Ubuntu

    Description

    I noticed that the minimum depth parameter is ignored when delineating depressions with lidar. I checked the code and this seems to be the case for

    DelineateDepressions(in_sink, min_size, min_depth, interval, out_dir, bool_level_shp=False)

    See: https://github.com/giswqs/lidar/blob/66498a36c4eb184d00bc4cce3ee41de86f53a4cb/lidar/slicing.py#L570

    opened by geotom 2
  • Wrong depression ID in outputted depressions.shp and level shapefiles

    Wrong depression ID in outputted depressions.shp and level shapefiles

    • lidar version: 0.6.1 (Forced to use this one because of #21)
    • Python version: 3.8
    • Operating System: Ubuntu (Docker)

    Description

    When delineating depressions I use the following

    sink_path = ExtractSinks(
        './test/dem.tif',
        1,
        './output'
    )
    DelineateDepressions(
        sink_path,
        1,
        0.5,
        0.2,
        './output'
        True
    )
    

    This runs through and produces me the general sinks (regions.shp) and the nested depressions (depressions.shp, depressions_info.csv). I understand the relation between both and want to assign to each region all the associated depressions and their geometries by looping through all the regions (ID's) and assigning them every depression (getting info from the depressions_info.csv). But I stumbled upon an issue where I cannot associate depressions by their IDs to the according poygon geometry, because the depressions.shp IDs are wrong. While I have enumerated all depressions correctly in the CSV file, the depressions shapefile has an very strange (and I assume wrong) numbering. At least most IDs fit, but I have for the level 1 depressions suddenly hundreds of depressions all having the depression ID 1. The rest of the polygons all have the correct depression ID, which relates tothe same ID in the depressions_info.csv file.

    image

    The same can be seen when looking at the individual level shapefiles. For level 1, I have a lot of polygons all with the same depression ID = 1

    image

    This does not allow to associate a geometry to a depression via its ID. I guess 1/50 or 1/100 of my depressions (and only those of level 1) have the wrong depression ID set to 1.

    image

    Do I misunderstand something or is this a bug in lidar? I assume, the polygons in level 1 are written to the shapefiles with an inccorect ID of 1 instead of the depression ID from the CSV file

    opened by geotom 5
Releases(v0.7.1)
Owner
Qiusheng Wu
Assistant Professor of Geography at the University of Tennessee, Knoxville
Qiusheng Wu
Get Landsat surface reflectance time-series from google earth engine

geextract Google Earth Engine data extraction tool. Quickly obtain Landsat multispectral time-series for exploratory analysis and algorithm testing On

Loïc Dutrieux 50 Dec 15, 2022
Digital Earth Australia notebooks and tools repository

Repository for Digital Earth Australia Jupyter Notebooks: tools and workflows for geospatial analysis with Open Data Cube and xarray

Geoscience Australia 335 Dec 24, 2022
A package built to support working with spatial data using open source python

EarthPy EarthPy makes it easier to plot and manipulate spatial data in Python. Why EarthPy? Python is a generic programming language designed to suppo

Earth Lab 414 Dec 23, 2022
Python package for earth-observing satellite data processing

Satpy The Satpy package is a python library for reading and manipulating meteorological remote sensing data and writing it to various image and data f

PyTroll 882 Dec 27, 2022
A package to fetch sentinel 2 Satellite data from Google.

Sentinel 2 Data Fetcher Installation Create a Virtual Environment and activate it. python3 -m venv venv . venv/bin/activate Install the Package via pi

null 1 Nov 18, 2021
PySAL: Python Spatial Analysis Library Meta-Package

Python Spatial Analysis Library PySAL, the Python spatial analysis library, is an open source cross-platform library for geospatial data science with

Python Spatial Analysis Library 1.1k Dec 18, 2022
geemap - A Python package for interactive mapping with Google Earth Engine, ipyleaflet, and ipywidgets.

A Python package for interactive mapping with Google Earth Engine, ipyleaflet, and folium

Qiusheng Wu 2.4k Dec 30, 2022
leafmap - A Python package for geospatial analysis and interactive mapping in a Jupyter environment.

A Python package for geospatial analysis and interactive mapping with minimal coding in a Jupyter environment

Qiusheng Wu 1.4k Jan 2, 2023
A python package that extends Google Earth Engine.

A python package that extends Google Earth Engine GitHub: https://github.com/davemlz/eemont Documentation: https://eemont.readthedocs.io/ PyPI: https:

David Montero Loaiza 307 Jan 1, 2023
A ninja python package that unifies the Google Earth Engine ecosystem.

A Python package that unifies the Google Earth Engine ecosystem. EarthEngine.jl | rgee | rgee+ | eemont GitHub: https://github.com/r-earthengine/ee_ex

null 47 Dec 27, 2022
Open Data Cube analyses continental scale Earth Observation data through time

Open Data Cube Core Overview The Open Data Cube Core provides an integrated gridded data analysis environment for decades of analysis ready earth obse

Open Data Cube 410 Dec 13, 2022
A public data repository for datasets created from TransLink GTFS data.

TransLink Spatial Data What: TransLink is the statutory public transit authority for the Metro Vancouver region. This GitHub repository is a collectio

Henry Tang 3 Jan 14, 2022
Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Ayush Mishra 3 May 6, 2022
ESMAC diags - Earth System Model Aerosol-Cloud Diagnostics Package

Earth System Model Aerosol-Cloud Diagnostics Package This Earth System Model (ES

Pacific Northwest National Laboratory 1 Jan 4, 2022
Python Data. Leaflet.js Maps.

folium Python Data, Leaflet.js Maps folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js

null 6k Jan 2, 2023
Python tools for geographic data

GeoPandas Python tools for geographic data Introduction GeoPandas is a project to add support for geographic data to pandas objects. It currently impl

GeoPandas 3.5k Jan 3, 2023
Use Mapbox GL JS to visualize data in a Python Jupyter notebook

Location Data Visualization library for Jupyter Notebooks Library documentation at https://mapbox-mapboxgl-jupyter.readthedocs-hosted.com/en/latest/.

Mapbox 620 Dec 15, 2022
python toolbox for visualizing geographical data and making maps

geoplotlib is a python toolbox for visualizing geographical data and making maps data = read_csv('data/bus.csv') geoplotlib.dot(data) geoplotlib.show(

Andrea Cuttone 976 Dec 11, 2022
OSMnx: Python for street networks. Retrieve, model, analyze, and visualize street networks and other spatial data from OpenStreetMap.

OSMnx OSMnx is a Python package that lets you download geospatial data from OpenStreetMap and model, project, visualize, and analyze real-world street

Geoff Boeing 4k Jan 8, 2023