Tool to suck data from ArcGIS Server and spit it into PostgreSQL

Overview

chupaESRI

About

ChupaESRI is a Python module/command line tool to extract features from ArcGIS Server map services.

Name?

Think "chupacabra" or "Chupa Chups".

Dependencies

Setup

pip3 install -r requirements.txt

Usage

When run from the command line, the tool will extract all features from a Map Service Query endpoint. You must also specify a PostgreSQL connection string and the name of the table where the extracted data will be stored.

When accessed as a module, the EsriJSON2Pg class is available to you to incorporate into your own projects. See the main method for examples on its use.

How to Use

Have you encountered an ArcGIS Server instance that contains data that you would like to use, but are limited by the functionality provided by the Map Service?

By browsing the REST API structure, you can locate the Query endpoint for the Layer you desire downloading. For example, the Query endpoint looks similar to the following: http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Demographics/ESRI_Census_USA/MapServer/5/query

For example, to download the State polygons from the ArcGIS Server above into a "public.states" table in a local PostgreSQL instance:

./chupaESRI.py http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Demographics/ESRI_Census_USA/MapServer/5/query "host=localhost dbname=gisdata user=gisadmin password=P4ssW0rd" "public.states"

Optionally, you can provide an output SRID for your table to use. If not specified, the program looks for the spatial reference information using the endpoint.

./chupaESRI.py http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Demographics/ESRI_Census_USA/MapServer/5/query "host=localhost dbname=gisdata user=gisadmin password=P4ssW0rd" "public.states" -srid 4326

To-Do

  • Implement ability to export LINE features
  • More intelligent querying of server-provided statistics to better plan the download requests
  • More elegant way of identifying non-standard ArcGIS REST URLs
  • Development of an ArcGIS Toolbox to perform the extraction, transforming, loading from within ArcGIS

Disclaimer

Provided as-is. Use at your own risk. Licensed under the GPL v3. Check with the target ArcGIS Server administrator to make sure he/she allows extraction of the data using this tool.

You might also like...
Stitch image tiles into larger composite TIFs

untiler Utility to take a directory of {z}/{x}/{y}.(jpg|png) tiles, and stitch into a scenetiff (tif w/ exact merc tile bounds). Future versions will

Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.
Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Open Data Cube analyses continental scale Earth Observation data through time

Open Data Cube Core Overview The Open Data Cube Core provides an integrated gridded data analysis environment for decades of analysis ready earth obse

A public data repository for datasets created from TransLink GTFS data.
A public data repository for datasets created from TransLink GTFS data.

TransLink Spatial Data What: TransLink is the statutory public transit authority for the Metro Vancouver region. This GitHub repository is a collectio

QLUSTER is a relative orbit design tool for formation flying satellite missions and space rendezvous scenarios
QLUSTER is a relative orbit design tool for formation flying satellite missions and space rendezvous scenarios

QLUSTER is a relative orbit design tool for formation flying satellite missions and space rendezvous scenarios, that I wrote in Python 3 for my own research and visualisation. It is currently unfinished (95%) but the core algorithm and app works. Here's an example snippet of the GUI where four deputy satellites are orbiting a single chief in LEO.

This tool reads the route of a hike and generates a table of iNaturalist observations along the trails.

This tool reads the route of a hike and generates a table of iNaturalist observations along the trails. It also shows the observations and the route of the hike on a map. Moreover, it saves waypoints of the iNaturalist observations for offline navigation with a GPS device or smartphone.

Tool to display your current position and angle above your radar
Tool to display your current position and angle above your radar

🛠 Tool to display your current position and angle above your radar. As a response to the CS:GO Update on 1.2.2022, which makes cl_showpos a cheat-pro

OSMnx: Python for street networks. Retrieve, model, analyze, and visualize street networks and other spatial data from OpenStreetMap.

OSMnx OSMnx is a Python package that lets you download geospatial data from OpenStreetMap and model, project, visualize, and analyze real-world street

gjf: A tool for fixing invalid GeoJSON objects

gjf: A tool for fixing invalid GeoJSON objects The goal of this tool is to make it as easy as possible to fix invalid GeoJSON objects through Python o

Comments
  • chupaESRI crashes with

    chupaESRI crashes with "Invalid control character"

    Here's the crash dump. I'll try to narrow down the exact record that's generating the failure.

    Suggestions for patching my way around this?

    Invalid control character '\n' at: line 63227 column 678 (char 1625940)
    Traceback (most recent call last):
      File "./chupaESRI.py", line 266, in <module>
        jp = EsriJSON2Pg(arcjson)
      File "./chupaESRI.py", line 37, in __init__
        self.srcjson = json.loads(jsonstr)
      File "/usr/local/lib/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
        return _default_decoder.decode(s)
      File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
        obj, end = self.raw_decode(s)
      File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
        return self.scan_once(s, idx=_w(s, idx).end())
    JSONDecodeError: Invalid control character '\n' at: line 63227 column 678 (char 1625940)
    
    Failed on:  /arcgisshared/rest/services/WashCo_Dynamic/MapServer/110/query?where=objectid+>%3D+100001+AND+objectid+<%3D+101000&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&maxAllowableOffset=&geometryPrecision=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&returnDistinctValues=false&f=pjson
    
    opened by vielmetti 5
  • coordinates for some polygons are truncated

    coordinates for some polygons are truncated

    pointed the script from the command line at

    http://arcgisserver.semcog.org/arcgis/rest/services/BuildingFootprints/MapServer/2/query

    to test it out. here's what the PG layer looks like loaded up in QGIS:

    screen shot 2015-10-08 at 1 14 53 pm

    about 100 of the features had one point with the longitude "cut off", like so:

    {"type":"MultiPolygon",
    "coordinates":
    [[[[13506280.5777559,333401.299540699],
    [13506310.0249344,333343.036089242],
    [13506340.0593832,333358.21620737],
    [13506328.0534777,333381.971128613],
    [13506367.4163386,333401.865813643],
    [13506374.8458005,333387.16568242],
    [13506377.1305774,333388.320538059],
    [13506418.2496719,333306.963910773],
    [13506379.3402231,333287.298556462],
    [13506375.7244094,333294.452755913],
    [13506361.0564304,333287.039370075],
    [13506355.8576116,333297.324803144],
    [13506304.769685,333271.503937006],
    [13506290.8940289,333298.957349077],
    [13506260.234252,333283.461614177],
    [13506281.8251313,333240.742454067],
    [13506248.2788714,333223.787729681],
    [13506209.5357612,333300.443241477],
    [13506204.3818897,333297.838254586],
    [13506192.8385827,333320.676837265],
    [13506199.593832,333324.090879261],
    [13506184.5462598,333353.863188982],
    [13506214.8070866,333369.157480329],
    [13506217.8507218,333363.135170624],
    [13506248.6558399,333378.704724416],
    [13506231.7276903,333412.198490813],
    [13506251.5400263,333422.212270334],
    [1350624,333437.130577445], // <<< problem coordinate!
    [13506328.9796588,333480.080708683],
    [13506341.3802493,333455.545275584],
    [13506336.2562336,333452.955708668],
    [13506340.6925852,333444.177821532],
    [13506329.5485564,333438.545603678],
    [13506334.5790682,333428.592847779],
    [13506280.5777559,333401.299540699]]]]}
    
    opened by jmcbroom 3
  • error: [Errno 10060] A connection attempt failed

    error: [Errno 10060] A connection attempt failed

    I am getting an error socket.gaierror: [Errno 11004] get addrinfo failed.

    The input url parameter is - "https://server.arcgisonline.com/ArcGIS/rest/services/Specialty/Soil_Survey_Map/MapServer/0/?f=pjson"

    I have also tried from IP address but getting the error - "error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

    opened by rockmodi 1
  • Helping hand

    Helping hand

    Feel free to check out the strategies from "planning downloads" here: https://github.com/koopjs/featureservice/blob/master/index.js#L330-L371

    Hope this helps!

    enhancement 
    opened by dmfenton 0
Owner
John Reiser
John Reiser
ArcGIS Python Toolbox for WhiteboxTools

WhiteboxTools-ArcGIS ArcGIS Python Toolbox for WhiteboxTools. This repository is related to the ArcGIS Python Toolbox for WhiteboxTools, which is an A

Qiusheng Wu 190 Dec 30, 2022
Automatización de procesos geográficos con Python sobre Esri ArcGIS y QGIS.

Algoritmos y Programación GIS con Python by [email protected] Introducción Python es un potente lenguaje de programación interpretado con licencia

r.cfdtools 7 Dec 28, 2022
Mmdb-server - An open source fast API server to lookup IP addresses for their geographic location

mmdb-server mmdb-server is an open source fast API server to lookup IP addresses

Alexandre Dulaunoy 67 Nov 25, 2022
peartree: A library for converting transit data into a directed graph for sketch network analysis.

peartree ?? ?? peartree is a library for converting GTFS feed schedules into a representative directed network graph. The tool uses Partridge to conve

Kuan Butts 183 Dec 29, 2022
Imports VZD (Latvian State Land Service) open data into postgis enabled database

Python script main.py downloads and imports Latvian addresses into PostgreSQL database. Data contains parishes, counties, cities, towns, and streets.

Kaspars Foigts 7 Oct 26, 2022
A light-weight, versatile XYZ tile server, built with Flask and Rasterio :earth_africa:

Terracotta is a pure Python tile server that runs as a WSGI app on a dedicated webserver or as a serverless app on AWS Lambda. It is built on a modern

DHI GRAS 531 Dec 28, 2022
Tile Map Service and OGC Tiles API for QGIS Server

Tiles API Add tiles API to QGIS Server Tiles Map Service API OGC Tiles API Tile Map Service API - TMS The TMS API provides these URLs: /tms/? to get i

3Liz 6 Dec 1, 2021
GebPy is a Python-based, open source tool for the generation of geological data of minerals, rocks and complete lithological sequences.

GebPy is a Python-based, open source tool for the generation of geological data of minerals, rocks and complete lithological sequences. The data can be generated randomly or with respect to user-defined constraints, for example a specific element concentration within minerals and rocks or the order of units within a complete lithological profile.

Maximilian Beeskow 16 Nov 29, 2022
🌐 Local tile server for viewing geospatial raster files with ipyleaflet

?? Local Tile Server for Geospatial Rasters Need to visualize a rather large raster (gigabytes) you have locally? This is for you. A Flask application

Bane Sullivan 192 Jan 4, 2023
🌐 Local tile server for viewing geospatial raster files with ipyleaflet or folium

?? Local Tile Server for Geospatial Rasters Need to visualize a rather large (gigabytes) raster you have locally? This is for you. A Flask application

Bane Sullivan 192 Jan 4, 2023