So far I have failed to adapt the EncodeProcessDecode module s.t. training and testing phases can be distinguished (e.g. for the purpose of deactivating the adding of noise on latent reps during test time).

This is mainly due to the fact that a "is_training" boolean flag is typically passed when encoder, core and decoder have been already initialized (but not built) and I want to have the same EncodeProcessDecode model for train/test (i.e. no two separate train/test model objects but rather two train test tf ops). Therefore my understanding is that I somehow must be able to pass this flag into EncodeProcessDecode's _build() function. However, this function calls modules.GraphIndependent and here the number of arguments passed is pre-defined (naimly inputs and num_processing_steps)

Since this seems like a common use case to me, I was wondering how folks at DeepMind deal with this. Of course I could change the sonnet interface of GraphIndependent but that does not seem like a scalable solution to me. I could also not find a particular example for this in the documentation. Could you perhaps provide a minimum working example for this use case or provide some ideas how to do this?

Thank you!

Running against latest stable release of tensorflow (1.12) yields poor model performance for training/inference on GPU for EncodeProcessDecode-based models. It can be observed that CPU gets heavily used (and becomes the bottleneck) while GPU usage is spiky at best.

We observed that this happens because certain ops such as tf.cumsum and tf.gather were always being placed on the CPU even when allocated/running in a with tf.device("/GPU:0") block, which can be easily verified by running the session with config=tf.ConfigProto(log_device_placement=True).

The cause of this behavior is that some of these ops don't have implementations for GPU for certain data types on the latest stable release (see tensorflow/tensorflow#5445, or this comment https://github.com/tensorflow/tensorflow/issues/13164#issuecomment-332076675 related to gather_nd which also applies to cumsum per current code in https://github.com/tensorflow/tensorflow/blob/73e3215c3a2edadbf9111cca44ab3d5ca146c327/tensorflow/core/kernels/scan_ops.cc#L149 ).

The consequence of this behavior is a significant slowdown on training because tensor data has to be copied from GPU memory, processed by CPU, and copied back to GPU, during the broadcast steps which cause serious bottlenecks in the computation graph.

This fix casts the data types into something that allows these ops to run in GPU. The cast overhead is insignificant compared with the performance gain. We observed nearly 5x speedup in training with a relatively simple EncodeProcessDecode model with 6 processing steps on GPU-enabled systems.

Hi all,

first of all, I would like to say that I found the paper and this library extremely useful and with many possible real applications. However, I have a small concern about its compatibility with TF2. I tried to re-adapt the shortest_path.py example with the TF2 framework, but I found several compatibility issues.

Is there an easy way to fix it? thanks in advance

Getting this error when I run blocks_test.py, modules_test.py, and utils_tf_test.py.

2018-10-22 14:07:06.293160: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis. Error: Pack node (data_dicts_to_graphs_tuple/stack) axis attribute is out of bounds: 0

Was using tensorflow version 1.13.0-dev20181022.

launchx@launchx-System-Product-Name:~$ python Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import graph_nets as gn import sonnet as snt input_graphs = get_graphs() Traceback (most recent call last): File "", line 1, in NameError: name 'get_graphs' is not defined graph_net_module = gn.modules.GraphNetwork( ... edge_model_fn=lambda: snt.nets.MLP([32, 32]), ... node_model_fn=lambda: snt.nets.MLP([32, 32]), ... global_model_fn=lambda: snt.nets.MLP([32, 32])) output_graphs = graph_net_module(input_graphs) Traceback (most recent call last): File "", line 1, in NameError: name 'input_graphs' is not defined

Hi.

I'm trying to use nx.draw and utils_tf.data_dicts_to_graphs_tuple in the same TF2 notebook.

Whichever is executed second seems to cause a kernel restart in the notebook which i can't explain. Importing networkx is fine as long as nx.draw is not run.

@Mistobaan - I get this behaviour on your very helpful TF2 version of graph_nets_basic tutorial.

Michael.

Hi, in the EncodeProcessDecode model, I am trying to reset the node state in every rollout step in the for loop of the _build function to the ground truth node state. The networkx graphs that I used to create the create the target_op (target_graphs) and input_op (input_graph) are:

input_graph: (1,)
target_graphs: (14,)

num_processing_steps is 14 and the code I am using is the following:

       ground_truth_graphs_T = self._encoder._network._node_model(target_op.nodes)
       n_nodes = [tf.shape(ground_truth_graphs_T)[0]]
       mult = tf.constant([num_processing_steps])
       ground_truth_graphs_splits = tf.split(ground_truth_graphs_T, num_or_size_splits=tf.tile(n_nodes, mult), axis=0)


       for step in range(num_processing_steps):
           ground_truth_graphs_t = ground_truth_graphs_splits[step]
           latent = latent.replace(nodes=ground_truth_graphs_t)

           latent = self._core(latent)
           decoded_op = self._decoder(latent, is_training)
           output_ops.append(decoded_op)

I apply tf.split because I assume target_op.nodes has shape (?, x) while ? being n_node*num_processing_steps. This code fails with the following error and I fail to understand where the problem is:

InvalidArgumentError (see above for traceback): Determined shape must either match input shape along split_dim exactly if fully specified, or be less than the size of the input along split_dim if not fully specified. Got: 196 [[node EncodeProcessDecode/split (defined at /scr2/repos/GN/models/EncodeProcessDecode_one_step.py:134) = SplitV[T=DT_FLOAT, Tlen=DT_INT32, num_split=14, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_placeholders_from_networkxs_1/nodes_0_11/_63, EncodeProcessDecode/Tile/_65, gradients/EncodeProcessDecode/node_model_13/visual_and_latent_node_encoder/concat_grad/concat_1/axis)]] [[{{node EncodeProcessDecode/MLPDecoderGraphIndependent_13/graph_independent_1/global_model/mlp_decoder_global/sequential/linear/add/_6037}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_11958...linear/add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Question

I assume the number of nodes is not correctly inferred. How do I get it from GraphsTuple input_op or target_op after graph definition time? In the tutorial I found this piece of information:

nodes (shape=[total_num_nodes] + node_feature_dimensions) --> What does the "+" here mean? Assume a graph with 3 total nodes and a node feature dimension of 200. What will the shape look like? Thank you.

Hi, I've been trying to understand the EncodeProcessDecode model parameters, however, the documentation is a little scarce on the parameters (edge_output_size, node_output_size, global_output_size) in this case.

In the code I found the following comment # Transforms the outputs into the appropriate shapes. but that doesn't clarify the exact reshaping. Can anybody clarify?

Thanks a lot!

Hi, is it possible that this library one day will be usable in the new generation of architecture Apple Silicon M1?

I tried to install it in a conda environment (through miniforge) after having installed also TensorFlow 2.4 benchmark on Mac M1, but it doesn't work! It gives an error related with bazel.

The paper is great. But I wanted to ask if there is any plan to develop this project? If I wanted to learn a graph framework, should I invest time to learn this or others like https://github.com/dmlc/dgl ? last commit was 4 months ago.

Hi! I'm trying to run a model that predicts node attributes based on global and edge inputs. I've been largely following the shortest_path.ipynb demo to write my code, and my code at the moment looks as follows (happy to include more if need be!):

# train_input, train_target, test_input etc. are all lists containing nxgraphs

train_input_ph, train_target_ph = create_placeholders(train_input, train_target)
test_input_ph, test_target_ph = create_placeholders(test_input, test_target)

output_train_graphs = graph_net_module(train_input_ph)
output_test_graphs = graph_net_module(test_input_ph)

loss_train = create_loss_ops(train_target_ph, output_train_graphs)
loss_test = create_loss_ops(test_target_ph, output_test_graphs)
....

train_input_ph, train_target_ph, output_train_graphs, output_test_graphs = make_all_runnable_in_session(train_input_ph, train_target_ph, output_train_graphs, output_test_graphs)

In the running training section of code, I then have:

for iteration in range(last_iteration, num_training_iterations):
    last_iteration = iteration
    train_values = sess.run({
        "step": step_op,
        "target": train_target_ph,
        "train_loss": loss_train,
        "outputs": output_train_graphs
    },
    feed_dict={train_input_ph: gn.utils_np.networkxs_to_graphs_tuple(train_input),
               train_target_ph: gn.utils_np.networkxs_to_graphs_tuple(train_target)}
    )

However, when I try to run the second set of code, I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1091             subfeed_t = self.graph.as_graph_element(
-> 1092                 subfeed, allow_tensor=True, allow_operation=False)
   1093           except Exception as e:

/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in as_graph_element(self, obj, allow_tensor, allow_operation)
   3477     with self._lock:
-> 3478       return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
   3479 

/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation)
   3566       raise TypeError("Can not convert a %s into a %s." % (type(obj).__name__,
-> 3567                                                            types_str))
   3568 

TypeError: Can not convert a Operation into a Tensor.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-1103-fddee2f34548> in <module>
     16     },
     17     feed_dict={train_input_ph: gn.utils_np.networkxs_to_graphs_tuple(train_input),
---> 18                train_target_ph: gn.utils_np.networkxs_to_graphs_tuple(train_target)}
     19     )
     20     the_time = time.time()

/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    927     try:
    928       result = self._run(None, fetches, feed_dict, options_ptr,
--> 929                          run_metadata_ptr)
    930       if run_metadata:
    931         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1093           except Exception as e:
   1094             raise TypeError(
-> 1095                 'Cannot interpret feed_dict key as Tensor: ' + e.args[0])
   1096 
   1097           if isinstance(subfeed_val, ops.Tensor):

TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a Operation into a Tensor.

I notice this is a similar thing to #24 but when I tried the solution there of reducing make_all_runnable_in_session to only act on output_train_graphs and output_test_graphs, I get the following error instead:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1091             subfeed_t = self.graph.as_graph_element(
-> 1092                 subfeed, allow_tensor=True, allow_operation=False)
   1093           except Exception as e:

/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in as_graph_element(self, obj, allow_tensor, allow_operation)
   3477     with self._lock:
-> 3478       return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
   3479 

/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation)
   3566       raise TypeError("Can not convert a %s into a %s." % (type(obj).__name__,
-> 3567                                                            types_str))
   3568 

TypeError: Can not convert a NoneType into a Tensor.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-1106-fddee2f34548> in <module>
     16     },
     17     feed_dict={train_input_ph: gn.utils_np.networkxs_to_graphs_tuple(train_input),
---> 18                train_target_ph: gn.utils_np.networkxs_to_graphs_tuple(train_target)}
     19     )
     20     the_time = time.time()

/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    927     try:
    928       result = self._run(None, fetches, feed_dict, options_ptr,
--> 929                          run_metadata_ptr)
    930       if run_metadata:
    931         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/miniconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1093           except Exception as e:
   1094             raise TypeError(
-> 1095                 'Cannot interpret feed_dict key as Tensor: ' + e.args[0])
   1096 
   1097           if isinstance(subfeed_val, ops.Tensor):

TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a NoneType into a Tensor.

What can I do with the feed_dict and session variables to make this all work?

Hey, in the last 2 years I've been working with graph_nets in my work (specially in my PhD). I even implement new blocks to work with recurrent layers and other minor features (I didn't send a pull request yet, because I cannot find time to organize the code and implement the tests).

Anyway, there are not any commit in the last 2 years, is there any plan to continue this project?

If graph_nets will not be continue, do you have any advice regarding other frameworks? I am not a big fan of Pytorch, I really like to use Tensorflow and Sonnet, then I am thinking to try DGL. However, I am not sure if I have the same degree of freedom that graph_nets gives to me.

Let's say I want to train an LSTM or transformer on sequences of graphs using Sonnet2/TF2:

I want to represent the graphs in each sequence as one GraphsTuple, which means my batches are essentially an iterable of GraphsTuples, each with a variable number of graphs. This is great until it's time to get the input signature and compile the update step. It's unclear to me how to define the tensorspec for this type of input. Is my best route to subclass collections.namedtuple() similar to how you define a GraphsTuple, or can you suggest a more elegant solution?

Thanks

I am using the graph module to solve an ETA estimation problem. Could anybody tell me how to make the training faster by using all the available cores on my computer? I tried many things in TensorFlow 2.5, such as setting the number of thread by tf.config.threading.set_inter_op_parallelism_threads and tf.config.threading.set_intra_op_parallelism_threads, but nothing works. Training the MLP networks took a very long time.

Hi, I'm interested in your novel work. I tried to change some condition in 'demo/physics.ipynb'. I wonder whether it also predict velocity well when I change initial position of spring. For example, I changed that the only first mass is fixed in function 'base_graph'.(it means the last mass is not fixed.) And I added damping force with hooke's law in Springsimulator. I thought this is so small condition for affecting but, the result of inference didn't follow target and spread out.

Is there relationship between model structure and form of initial condition(such as base graph)? and how can I change it to fit the target?

Hello, graph-nets Authors!! Following the discussion in #115 I wanted to contribute and adapted the shortest_path example to run in TF2 and implementing the sonnet Adam optimizer. I added a couple of functions to make it runnable. Ideally, they should probably live inside utils_np or utils_tf but I kept them here for simplicity. I hope it is useful!