Sorry for dropping a biggish pull request on you unannounced, but I haven't been able to get the GPU code working, and I noticed that a lot of the GPU (and CPU) Dockerfile seemed to be based on the tensorflow docker tools at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/docker, so I thought it might be worth changing the base FROM image from gdal to tensorflow.

This has made the Dockerfiles much simpler (and the gpu one is now automatically generated in the make file as it is only four characters different from the cpu version).

I have also added a very simple set of python unittests that just test module importing at the moment, and added them to the Makefile and the Travis CI configuration. All test currently pass!

I can also confirm that the CPU training works everywhere I've tried it, and the GPU training works on AWS EC2 GPU instances.

I would like to update tensorflow and tflearn, but I've left it at 0.8 and the arbitrary git checkout for now. Updating those can be a problem for a different pull request.

Thanks!

1. Installed the Docker
2. make dev command executed successfully.
3. after that when I'm executing the python bin/create_training_data.py I'm getting some errors like :

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions If the error persists, please report the following lines (removing any private info as necessary) to: [email protected]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd ls --recursive --skip-existing s3://aws-naip/in/2014/1m/rgbir/ --requester-pays Problem: gaierror: [Errno -2] Name or service not known S3cmd: 1.6.0 python: 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4] environment LANG=None

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.0-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2805, in File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.0-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2713, in main File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.0-py2.7.egg/EGG-INFO/scripts/s3cmd", line 120, in cmd_ls File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.0-py2.7.egg/EGG-INFO/scripts/s3cmd", line 153, in subcmd_bucket_list File "build/bdist.linux-x86_64/egg/S3/S3.py", line 293, in bucket_list for dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params): File "build/bdist.linux-x86_64/egg/S3/S3.py", line 320, in bucket_list_streaming response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params) File "build/bdist.linux-x86_64/egg/S3/S3.py", line 339, in bucket_list_noparse response = self.send_request(request) File "build/bdist.linux-x86_64/egg/S3/S3.py", line 1061, in send_request conn = ConnMan.get(self.get_hostname(resource['bucket'])) File "build/bdist.linux-x86_64/egg/S3/ConnMan.py", line 179, in get conn.c.connect() File "/usr/lib/python2.7/httplib.py", line 1216, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 553, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): gaierror: [Errno -2] Name or service not known

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions If the error persists, please report the above lines (removing any private info as necessary) to: [email protected] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! DOWNLOADING 1 PBFs... downloads took 3.7s EXTRACTED WAYS with locations from pbf file /DeepOSM/data/openstreetmap/delaware-latest.osm.pbf, took 7.1s

please help Using macOS Sierra

Currently, Dockerfile.devel-gpu inherits from GDAL Dockerfile, and then adds in both stuff needed for 1) data creation, and stuff needed for 2) analysis... including a nested stack of Tensorflow Dockerfiles that got copied in.

Two Dockers - Data Creation and Data Analysis

It would be cleaner if one Dockerfile created training data and saved it to disk, and one Dockerfile used training data from disk to analyze. The training data docker file could be mostly like the existing non-GPU Dockerfile, and the analysis Docker file would inherit from stock Tensorflow Dockerfile and be short, simple, and easy to maintain/deploy to AWS.

In production, I guess one Dockerfile saves data to S3, and the other mounts that data bucket for analysis. In development, it create a directory on disk, no S3 involved.

API

The data creation Docker app could also serve an API, to give people training data for their own models, or to support an MNIST like open research competition for maps.

hi~ I have installed a docker but cd /DeepOSM-master make dev then run python bin/create_training_data.py ,i have a error : File "bin/create_training_data.py", line 6, in from src.training_data import download_and_serialize ImportError: No module named src.training_data can you give me a detailed tutorial

Looks like the gdal swig binary is out of sync with newer python bindings? I'm running this inside the non-gpu docker image. I gather the GNM bits are relatively new, so it makes sense that they'd get hit.

root@f059367028e3:/DeepOSM# python ./bin/create_training_data.py 
Traceback (most recent call last):
  File "./bin/create_training_data.py", line 6, in <module>
    from src.training_data import download_and_serialize
  File "/DeepOSM/src/training_data.py", line 10, in <module>
    from osgeo import gdal
  File "/usr/local/lib/python2.7/dist-packages/osgeo/gdal.py", line 86, in <module>
    from gdalconst import *
  File "/usr/local/lib/python2.7/dist-packages/osgeo/gdalconst.py", line 148, in <module>
    OF_GNM = _gdalconst.OF_GNM
AttributeError: 'module' object has no attribute 'OF_GNM'

Also, I've tried going straight to https://hub.docker.com/r/homme/gdal/ and I can't reproduce there.

I'm wondering if one of the apt-get installs brought along an out-of-date gdal binary?

Here are a few more files:

root@f059367028e3:/usr/local/lib/python2.7/dist-packages# ls -lha /usr/local/lib/libgdal*
-rw-r--r-- 1 root root 288M Oct 28 11:24 /usr/local/lib/libgdal.a
-rwxr-xr-x 1 root root 1.6K Oct 28 11:24 /usr/local/lib/libgdal.la
lrwxrwxrwx 1 root root   17 Oct 28 11:24 /usr/local/lib/libgdal.so -> libgdal.so.20.1.0
lrwxrwxrwx 1 root root   17 Oct 28 11:24 /usr/local/lib/libgdal.so.20 -> libgdal.so.20.1.0
-rwxr-xr-x 1 root root 116M Oct 28 11:24 /usr/local/lib/libgdal.so.20.1.0
root@f059367028e3:/usr/local/lib/python2.7/dist-packages# ls -lha *gdal*
-rw-r--r-- 1 root staff 128 Oct 28 10:15 gdal.py
-rw-r--r-- 1 root staff 244 Oct 28 11:25 gdal.pyc
-rw-r--r-- 1 root staff 143 Oct 28 10:15 gdalconst.py
-rw-r--r-- 1 root staff 274 Oct 28 11:25 gdalconst.pyc
-rw-r--r-- 1 root staff 147 Oct 28 10:15 gdalnumeric.py
-rw-r--r-- 1 root staff 279 Oct 28 11:25 gdalnumeric.pyc

Hey Guys, I am B.Tech. CS graduate and I have been working on image processing since 1.5 years. I would like to contribute to this project. Can anyone assign me some task, doesn't matter how simple or tough ?

I don't understand how I can make an index of the NAIP bucket on S3. The parameters are as follows: ['md', '2013', '1m', 'rgbir', '38077']

I understand all the parameters but the last one... I assume that is some sort of grid or ordinal ID.

If I knew what the final parameter was, I could write a script to get training data from any state/year. Now it's limited to a certain place in Maryland, around Washington DC.

I‘ve configurated s3cfg, and i can download naip image via s3cmd fine, but when running bin/create_training_data.py, it ends up 403 error when downloading naip images.

“For output, DeepOSM will produce some console logs, and then JPEGs of the ways, labels, and predictions overlaid on the tiff.” i run train_neural_net.py successfully . but there no JPEGs of the ways, labels, and predictions overlaid on the tiff.

The current neural net uses something really made for handwritten digit classification (MNIST).

We could do stuff like use Alexnet instead, or implement the neural nets described in Mnih/Hinton too. Literature also describes using a sequence of pre and post processing neural nets, where you can fill in gaps in road networks.

Expanding on these vague comments, there is a whole body of literature about how to use CNNs, RNNs, global topology, lidar elevation data, and much more to improve the accuracy of satellite imagery label. We should be able to get above 90% on the pixel level, just using semi-local RGB data, and push past that with multiple neural nets, more data, and other documented improvements in the last 2-3 years since Mnih's thesis.

See the README for a list of readings.

python bin/run_analysis.py Traceback (most recent call last): File "bin/run_analysis.py", line 7, in from src.run_analysis import analyze, render_results_as_images File "/DeepOSM/src/run_analysis.py", line 6, in import label_chunks_cnn_cifar File "/DeepOSM/src/label_chunks_cnn_cifar.py", line 11, in import tflearn File "/usr/local/lib/python2.7/dist-packages/tflearn/init.py", line 21, in from .layers import normalization File "/usr/local/lib/python2.7/dist-packages/tflearn/layers/init.py", line 10, in from .recurrent import lstm, gru, simple_rnn, bidirectional_rnn, File "/usr/local/lib/python2.7/dist-packages/tflearn/layers/recurrent.py", line 8, in from tensorflow.contrib.rnn.python.ops.core_rnn import static_rnn as _rnn, ImportError: No module named core_rnn

and this

File "bin/run_analysis.py", line 7, in from src.run_analysis import analyze, render_results_as_images File "/DeepOSM/src/run_analysis.py", line 6, in import label_chunks_cnn_cifar File "/DeepOSM/src/label_chunks_cnn_cifar.py", line 11, in import tflearn File "/usr/local/lib/python2.7/dist-packages/tflearn/init.py", line 4, in from . import config File "/usr/local/lib/python2.7/dist-packages/tflearn/config.py", line 5, in from .variables import variable File "/usr/local/lib/python2.7/dist-packages/tflearn/variables.py", line 7, in from tensorflow.contrib.framework.python.ops import add_arg_scope as contrib_add_arg_scope ImportError: cannot import name add_arg_scope

while running the github project https://github.com/zilongzhong/DeepOSM In this one I am able to create training data successfully but not able to run_analysis.py and while running I'm getting this error and

..any idea? or a hint, where I might get deeper information about it?

Kind regards,

Daniel

Training samples: 199 Validation samples: 23

Training Step: 899 | total loss: 0.23639 Training Step: 899 | total loss: 0.23639 acc: 0.9115 | val_loss: 0.29569 - val_| Momentum | epoch: 001 | loss: 0.23639 - acc: 0.9115 | val_loss: 0.29569 - val_acc: 0.8261 -- iter: 199/199

Training Step: 903 | total loss: 0.36168 Training Step: 903 | total loss: 0.36168 acc: 0.8505 | val_loss: 0.23084 - val_| Momentum | epoch: 002 | loss: 0.36168 - acc: 0.8505 | val_loss: 0.23084 - val_acc: 0.9565 -- iter: 199/199

Training Step: 907 | total loss: 0.34085 Training Step: 907 | total loss: 0.34085 acc: 0.8723 | val_loss: 0.11795 - val_| Momentum | epoch: 003 | loss: 0.34085 - acc: 0.8723 | val_loss: 0.11795 - val_acc: 0.9565 -- iter: 199/199

Training Step: 911 | total loss: 0.61323 Training Step: 911 | total loss: 0.61323 acc: 0.7900 | val_loss: 0.16038 - val_| Momentum | epoch: 004 | loss: 0.61323 - acc: 0.7900 | val_loss: 0.16038 - val_acc: 0.9565 -- iter: 199/199

Training Step: 915 | total loss: 0.55020 Training Step: 915 | total loss: 0.55020 acc: 0.7925 | val_loss: 0.23164 - val_| Momentum | epoch: 005 | loss: 0.55020 - acc: 0.7925 | val_loss: 0.23164 - val_acc: 0.9565 -- iter: 199/199

WARNING:tensorflow:Error encountered when serializing data_augmentation. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'NoneType' object has no attribute 'name' WARNING:tensorflow:Error encountered when serializing summary_tags. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'dict' object has no attribute 'name' WARNING:tensorflow:Error encountered when serializing data_preprocessing. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'NoneType' object has no attribute 'name'

In geo_util.py, this line gives me the correct lat lon in web mercator.

x2, y2 = transform(in_proj, out_proj, ulon, ulat)

But this last line (just before the return) seems to transform it back to lat lon in degrees:

x2, y2 = out_proj(x2, y2, inverse=True)

which I believe is incorrect. Am I missing something here?

This appears when I'm using mac and linux centos7 both, after i do 'make dev' and working container is successfully launched, the clock in container is far away from the local clock, and seems the offset is random, and i can't set clock inside container. This may cause downloading failure when runing create_training_data.py in container. And I'm in timezone +8.

Since deeposm.org site no longger exists, so can you provide codes that generates local website contents? So user can have a presentation of the results(findings, errors .etc) other than label files.

I met this error when I rendering findings to JPEG, using render_results_for_analysis method in file src/training_visualization.py, traceback: Wrong JPEG library version: library is 90, caller expects 80 Traceback (most recent call last): File "upload_data.py", line 31, in <module> main() File "upload_data.py", line 27, in main render_findings(raster_data_paths, model, training_info, model_info['bands'], True) File "/DeepOSM/src/s3_client_deeposm.py", line 30, in render_findings training_info['tile_size']) File "/DeepOSM/src/training_visualization.py", line 34, in render_results_for_analysis tile_size) File "/DeepOSM/src/training_visualization.py", line 54, in render_predictions predictions=predictions_by_naip) File "/DeepOSM/src/training_visualization.py", line 111, in render_results_as_image im.save(outfile, "JPEG") Upgrade Pillow python library version to 4.0.0 can solve this.

btw, Since deeposm.org site no longger exists, so can you provide codes that generates local website contents? Is there ways to view the results other than JPEG with findings on map?