AnuGA for the simulation of the shallow water equation

Hello I am using anuga in my project for model simulation and successfully run my data in serial and parallel(4 cores) but when I am comparing result output timings
using DEM and creating domain by anuga.create_domain_from_regions

Geospatial_data: Reading block taking 24 mins in serial and 36 mins in parallel(4 cores)

why that much difference is there and what is the actual process while reading Geospatial data. I checked code in anuga directory geospatial_data.py

can anyone help how to reduce data reading time in parallel.

Regards, Tejaswi

I am trying to install anuga on windows 10 and with puython 2.7.5 python runtests.py gives following error

running build_py copying anuga\revision.py -> build\lib.win-amd64-2.7\anuga copying build\src.win-amd64-2.7\anuga_config_.py -> build\lib.win-amd64-2.7\anuga running build_clib customize MSVCCompiler customize MSVCCompiler using build_clib running build_ext customize MSVCCompiler customize MSVCCompiler using build_ext building 'anuga.abstract_2d_finite_volumes.neighbour_mesh_ext' extension compiling C sources C:\Users\Admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ianuga\utilities -IC:\OSGeo4W64\apps\Python27\lib\site-packages\numpy\core\include -Ianuga\pymetis\metis-4.0 -IC:\OSGeo4W64\apps\Python27\include -IC:\OSGeo4W64\apps\Python27\PC /Tcanuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c /Fobuild\temp.win-amd64-2.7\Release\anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.obj Found executable C:\Users\Admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe Running from anuga source directory. neighbour_mesh_ext.c c:\osgeo4w64\apps\python27\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION anuga\utilities\util_ext.h(55) : error C2059: syntax error : 'type' anuga\utilities\util_ext.h(63) : error C2059: syntax error : 'type' anuga\utilities\util_ext.h(320) : error C2065: 'func' : undeclared identifier anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(43) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(44) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(116) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(122) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(123) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(128) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c(134) : warning C4244: '=' : conversion from 'npy_intp' to 'int', possible loss of data error: Command "C:\Users\Admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ianuga\utilities -IC:\OSGeo4W64\apps\Python27\lib\site-packages\numpy\core\include -Ianuga\pymetis\metis-4.0 -IC:\OSGeo4W64\apps\Python27\include -IC:\OSGeo4W64\apps\Python27\PC /Tcanuga\abstract_2d_finite_volumes\neighbour_mesh_ext.c /Fobuild\temp.win-amd64-2.7\Release\anuga\abstract_2d_finite_volumes\neighbour_mesh_ext.obj" failed with exit status 2

Build failed!

Hi everyone:

Also @stoiver . Can I compile the anuga with python3.6 under ubuntu18.04? I see the python version of python is 2.7 almostly. If I use python 3.6, is there any problem? I will try and hope to get your experiences.

Best reagards dongxueyang

Hi I'm getting a segmentation fault when running the tests. I am running on ubuntu, using a conda environment. My exact installation process was as follows:

conda create -n anuga python=2
source activate anuga
conda install nose numpy scipy matplotlib netcdf4
conda install -c pingucarsti gdal
git clone https://github.com/GeoscienceAustralia/anuga_core.git
cd anuga_core/
python setup.py build
python setup.py install

Then running python runtests.py gives the following output:

$ python runtests.py 
Building, see build.log...
Build OK
Running unit tests for anuga
NumPy version 1.13.1
NumPy relaxed strides checking option: True
NumPy is installed in /home/james/miniconda3/envs/anuga/lib/python2.7/site-packages/numpy
Python version 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
nose version 1.3.7
...............................................................................................Segmentation fault (core dumped)

Hi, I am use ANUGA to simulate rainfall over DEM. It works as I used uniform size mesh anuga.rectangular_cross_domain().

ncols     = 648
nrows     = 802
xllcorner = -52.17979305225
yllcorner = -1283978.9725515
cellsize  = 290.34756669662
domain    = anuga.rectangular_cross_domain(ncols, nrows, ncols*cellsize, nrows*cellsize, [xllcorner, yllcorner])

However, it gave warning as I used non-uniform mesh, and produce zero depth in the results, also delta t = 1000, the piece of code:

bounding_polygon = anuga.read_polygon('extent.csv')  # Extent.
domain           = anuga.create_domain_from_regions(bounding_polygon,
                                    boundary_tags={'top':    [0],
                                                   'right':  [1],
                                                   'bottom': [2],
                                                   'left':   [3]},
                                    maximum_triangle_area=50000,
                                    #interior_regions=[[bounding_polygon,500000]],
                                    mesh_filename='tx_300m.msh',
                                    use_cache=False,
                                    verbose=True,
                                    fail_if_polygons_outside=True)

The warning is:

/home/wlai/anaconda3/envs/anuga/lib/python2.7/site-packages/anuga/shallow_water/shallow_water_domain.py:2219: RuntimeWarning: invalid value encountered in less
  negative_ids = num.where( num.logical_and((Stage.centroid_values - Elev.centroid_values) < 0.0 , tff > 0) )[0]
/home/wlai/anaconda3/envs/anuga/lib/python2.7/site-packages/anuga/file/sww.py:343: RuntimeWarning: invalid value encountered in greater_equal
  storable_indices = num.array(w-z >= self.minimum_storable_height)
/home/wlai/anaconda3/envs/anuga/lib/python2.7/site-packages/anuga/file_conversion/sww2dem.py:197: RuntimeWarning: invalid value encountered in greater
  q = fid.variables[name][:].flatten()

The whole code is also provided here. Thank you for your help.

"""Test with Tx300 DEM.
"""

#------------------------------------------------------------------------------
# Import necessary modules
#------------------------------------------------------------------------------
# Standard modules
import os
import time
import sys
import anuga
import numpy
from math import sin, pi, exp

time00 = time.time()

#------------------------------------------------------------------------------
# Setup computational domain
#------------------------------------------------------------------------------
anuga.asc2dem('tx_300m.asc')
anuga.dem2pts('tx_300m.dem')

# FIXME, tx300m Rainfall using non-uniform mesh doesnt work. 
bounding_polygon = anuga.read_polygon('extent.csv')  # Extent.
domain           = anuga.create_domain_from_regions(bounding_polygon,
                                    boundary_tags={'top':    [0],
                                                   'right':  [1],
                                                   'bottom': [2],
                                                   'left':   [3]},
                                    maximum_triangle_area=50000,
                                    #interior_regions=[[bounding_polygon,500000]],
                                    mesh_filename='tx_300m.msh',
                                    use_cache=False,
                                    verbose=True,
                                    fail_if_polygons_outside=True)

'''
# Try uniform mesh. This works.
ncols     = 648
nrows     = 802
xllcorner = -52.17979305225
yllcorner = -1283978.9725515
cellsize  = 290.34756669662
domain    = anuga.rectangular_cross_domain(ncols, nrows, ncols*cellsize, nrows*cellsize, [xllcorner, yllcorner])
'''
# Print some stats about mesh and domain
print 'Number of triangles = ', len(domain)
print 'The extent is ', domain.get_extent()
#print domain.statistics()
                  
#------------------------------------------------------------------------------
# Setup parameters of computational domain
#------------------------------------------------------------------------------
domain.set_name('tx_300m') # Name of sww file
domain.set_datadir('.')                       # Store sww output here
domain.set_minimum_storable_height(0.01)      # Store only depth > 1cm
domain.set_flow_algorithm('DE0')

#------------------------------------------------------------------------------
# Setup initial conditions
#------------------------------------------------------------------------------
domain.set_quantity('friction', 0.03)
domain.set_quantity('elevation', 
                    filename='tx_300m.pts',
                    use_cache=True,
                    verbose=True,
                    alpha=0.1)
h = 0.0
domain.set_quantity('stage', expression='elevation + %f' % h)  ## FIXME, set depth?

#------------------------------------------------------------------------------
# Setup boundary conditions
#------------------------------------------------------------------------------
bc_wall = anuga.Reflective_boundary(domain)      # Solid reflective wall
bc_out  = anuga.Transmissive_boundary(domain)
# Associate boundary tags with boundary objects
domain.set_boundary({'left': bc_out, 'right': bc_out, 'top': bc_out, 'bottom': bc_out})

#---------------------------------------------------------------------------------------
# Rainfall WL. Note rainfall file is daily accumulated rainfall in inch/day.
anuga.asc2dem('tx_rainfall.asc')
anuga.dem2pts('tx_rainfall.dem')

ncols            = 41
nrows            = 50
xllcorner        = -4757.5382496486
yllcorner        = -1288482.8472377
cellsize         = 4763
rainfall_domain  = anuga.rectangular_cross_domain(ncols, nrows, ncols*cellsize, nrows*cellsize, [xllcorner, yllcorner])

# Print some stats about mesh and domain
#print 'Number of triangles = ', len(rainfall_domain)
#print 'The extent is ', rainfall_domain.get_extent()
#print rainfall_domain.statistics()

rainfall_domain.set_quantity('stage', 
                    filename='tx_rainfall.pts',
                    use_cache=True,
                    verbose=True,
                    alpha=0.1)

#print 'rainfal domain ', dir(rainfall_domain)
conv      = 2.9398e-7   # convert inch/day to meter/sec.
inter_pts = domain.get_vertex_coordinates()
rf_accu   = rainfall_domain.get_quantity('stage').get_values(interpolation_points=inter_pts)
rf_accu   = numpy.reshape(rf_accu, (domain.get_number_of_triangles(), 3))
rf_rate   = conv * rf_accu
print 'number of inter_pts', len(inter_pts), ',size of inter_pts', sys.getsizeof(inter_pts), ',shape:', inter_pts.shape
print 'number of rf_rate',   len(rf_rate),   ',size of rf_rate',   sys.getsizeof(rf_rate),   ',shape:', rf_rate.shape
#print 'domain.eleveation shape : ', len(domain.quantities['elevation']), ', col', len((domain.quantities['elevation'])[0])
'''
print domain.get_number_of_nodes()
print domain.get_number_of_triangles()
print domain.get_unique_vertices()

print rainfall_domain.get_number_of_nodes()
print rainfall_domain.get_number_of_triangles()
'''
#------------------------------------------------------------------------------
# Evolve system through time
#------------------------------------------------------------------------------

dt      = 60  # Timestep in seconds.
rf_rate = abs(rf_rate * dt)
for t in domain.evolve(yieldstep=dt, finaltime=1200.0):
    print domain.timestepping_statistics()
    domain.add_quantity('stage', rf_rate)

# Print time.
print 'That took %.2f seconds' %(time.time()-time00)

# Create inundation map.
anuga.sww2dem('tx_300m.sww', 'inundation.asc', quantity='stage-elevation', cellsize=300,reduction=max,verbose=True)

I tried installing Anuga in parallel with Intel MPI. In the documentation, we have used either openmpi or mpich2. Running tests with Intel MPI fails with signal 9 always. Any plan for supporting Intel MPI?

with command-

export ANUGA_PARALLEL="openmpi"

Shall I use export ANUGA_PARALLEL="intelmpi" ?

C++ version of the neighbour_table_ext.c module using stl's unordered_map and minimalistic hash-function instead of uthash. Also changed integer-types used to long.

I currently use this and so far it works (tested up to ~1000mio triangles), but as @stoiver mentioned a C module would be preferred. Uploaded this so someone with Windows/MacOS systems can test if it would compile without having to mess with additional compiler. (I used g++ 6.3.0 on debian)

added function test_set_quantity_from_file in test_shallow_water_domain.py to test the results of set_quantity and set_values from pts, asc and dem files. The result of set_quantity and set_values should be the same.

Hi,

I was wondering if is suitable for dam breach flooding modelling? If so, can you please share a dam breach case study or application using ANUGA?

Thank you.

Dear admin,

I ve been trying to install Anuga on my MacOS but have not had much success. I have installed on my Ubuntu alright. However, as I have tried the install_conda_macos.sh file, I had no success. I have used python2 command instead of python throughout the command list and the installation seems to go all the way to the end. But as I try to runtests.py, lots of errors pop up. I haven't gone through all of them because I am not sure whether there are incompatibilities related to python scripts and Mac. If there are any specific detail I should care about while attempting an install on MacOS I would much appreciate, as I have not read anything at Github description page.

Cheers Diogo

suspecting int32 overflow and trying to migrate neighbour_table_ext.c towards larger integers. what was the reason to use uthash, and would using a different hasher with a nice license be an ok thing to do? (if actually needed, maybe uthash isn't the problem) how about rewriting the module to use c++? would open up using STL instead of custom hashtable. but one would need a c++ compiler to install ANUGA.

domain = anuga.shallow_water.shallow_water_domain.Domain(p,v,b)

[neutron:23330] *** Process received signal ***
[neutron:23330] Signal: Segmentation fault (11)
[neutron:23330] Signal code: Address not mapped (1)
[neutron:23330] Failing at address: 0x17
[neutron:23330] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7ffff7bcd0c0]
[neutron:23330] [ 1] /usr/local/lib/python2.7/dist-packages/anuga/abstract_2d_finite_volumes/neighbour_table_ext.so(add_edge+0x356)[0x7ff7e7a6f0f6]
[neutron:23330] [ 2] /usr/local/lib/python2.7/dist-packages/anuga/abstract_2d_finite_volumes/neighbour_table_ext.so(_build_neighbour_structure+0x6b)[0x7ff7e7a6fc8b]
[neutron:23330] [ 3] /usr/local/lib/python2.7/dist-packages/anuga/abstract_2d_finite_volumes/neighbour_table_ext.so(build_neighbour_structure+0xc8)[0x7ff7e7a6fe98]
[neutron:23330] [ 4] python(PyEval_EvalFrameEx+0x69a)[0x55555564f3ca]
[neutron:23330] [ 5] python(PyEval_EvalFrameEx+0x5eef)[0x555555654c1f]
[neutron:23330] [ 6] python(PyEval_EvalCodeEx+0x255)[0x55555564d535]
[neutron:23330] [ 7] python(+0x115ea8)[0x555555669ea8]
[neutron:23330] [ 8] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [ 9] python(+0x12bfee)[0x55555567ffee]
[neutron:23330] [10] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [11] python(PyEval_CallObjectWithKeywords+0x7d)[0x55555565947d]
[neutron:23330] [12] python(PyInstance_New+0x7c)[0x55555566ce3c]
[neutron:23330] [13] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [14] python(PyEval_EvalFrameEx+0x60ff)[0x555555654e2f]
[neutron:23330] [15] python(PyEval_EvalCodeEx+0x255)[0x55555564d535]
[neutron:23330] [16] python(+0x115ea8)[0x555555669ea8]
[neutron:23330] [17] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [18] python(+0x12bfee)[0x55555567ffee]
[neutron:23330] [19] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [20] python(PyEval_EvalFrameEx+0x60ff)[0x555555654e2f]
[neutron:23330] [21] python(PyEval_EvalCodeEx+0x255)[0x55555564d535]
[neutron:23330] [22] python(+0x115cee)[0x555555669cee]
[neutron:23330] [23] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [24] python(+0x12bfee)[0x55555567ffee]
[neutron:23330] [25] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [26] python(PyEval_CallObjectWithKeywords+0x30)[0x555555659430]
[neutron:23330] [27] python(PyInstance_New+0x7c)[0x55555566ce3c]
[neutron:23330] [28] python(PyObject_Call+0x43)[0x55555563b673]
[neutron:23330] [29] python(PyEval_EvalFrameEx+0x60ff)[0x555555654e2f]
[neutron:23330] *** End of error message ***

Hi .. I have installed Anuga_py3 and it runs well with a smaller number of nodes, but it fails to run on a large number of nodes with the below error.

Traceback (most recent call last):
  File "test.py", line 79, in <module>
    domain = distribute(domain, verbose=verbose)  # divide domain into no of cores
  File "/home/samir/miniconda3/envs/anuga_py3_mpich/lib/python3.7/site-packages/anuga/parallel/parallel_api.py", line 99, in distribute
  File "/home/samir/miniconda3/envs/anuga_py3_mpich/lib/python3.7/site-packages/anuga/utilities/parallel_abstraction.py", line 241, in send
  File "mpi4py/MPI/Comm.pyx", line 1406, in mpi4py.MPI.Comm.send
  File "mpi4py/MPI/msgpickle.pxi", line 185, in mpi4py.MPI.PyMPI_send
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Send(157)...................: MPI_Send(buf=0x564a90cbb350, count=130673, MPI_BYTE, dest=1052, tag=1, MPI_COMM_WORLD) failed
MPIDI_CH3_EagerContigSend(195)...: failure occurred while attempting to send an eager message
MPID_nem_tcp_iStartContigMsg(349):
MPID_nem_tcp_connect(858)........: unable to create a socket, Too many open files (errno 24)

Hello everyone, I want to simulate the viration of water quality in the river. Is the code can be used for the simulation of the solute transportation and diffusion? Or is there any other codes that can be used for? Thanks a lot! Henry

Hi sir: @stoiveroberts Now I can see that the anuga can read a topography with an asc formant file. So is it possible to use other format to generate the mesh, like the stl file?

Dong

Hi @stoiver :

I want to run a large simulation. To improve the efficiency of calculation. I want to run the program on a supercomputing platform, and use multi-nodes. So if the codes support multi-node computing mode?