Pytorch implementation of "A simple neural network module for relational reasoning" (Relational Networks)

Train Epoch: 18 [193280/196000 (99%)] Relations accuracy: 95% | Non-relations accuracy: 100% Train Epoch: 18 [194560/196000 (99%)] Relations accuracy: 86% | Non-relations accuracy: 100% Train Epoch: 18 [195840/196000 (100%)] Relations accuracy: 89% | Non-relations accuracy: 100%

Test set: Relation accuracy: 89% | Non-relation accuracy: 100% Train Epoch: 19 [192000/196000 (98%)] Relations accuracy: 94% | Non-relations accuracy: 100% Train Epoch: 19 [193280/196000 (99%)] Relations accuracy: 80% | Non-relations accuracy: 100% Train Epoch: 19 [194560/196000 (99%)] Relations accuracy: 89% | Non-relations accuracy: 100% Train Epoch: 19 [195840/196000 (100%)] Relations accuracy: 91% | Non-relations accuracy: 100%

Test set: Relation accuracy: 90% | Non-relation accuracy: 99% Train Epoch: 20 [193280/196000 (99%)] Relations accuracy: 91% | Non-relations accuracy: 100% Train Epoch: 20 [194560/196000 (99%)] Relations accuracy: 97% | Non-relations accuracy: 100% Train Epoch: 20 [195840/196000 (100%)] Relations accuracy: 95% | Non-relations accuracy: 100%

Test set: Relation accuracy: 89% | Non-relation accuracy: 99%

Hi, the code is almost completely self-explanatory. However, I couldn't understand this part. Could you explain there ? Why you're creating coord_tensor and np_coord_tensor and what is the number 25 there ? I also would like to hear about lines 48-49-50. Edit: I have also another question about the implementation. Eventhough it is completely differ'rent, I don't want to open one more issue :) I guess translator.py file and function inside of it isn't used anywhere in the code. What was the aim of that file ?

I have trained RN for 10 epochs. The final test set accuracy is 73% for relational question and 72% for non-relational question. It seems that there is no significant improvement for relational questions.

Hi, I'm trying to find a source for the Sort-of-CLEVR Dataset. The provided code in this repository seems to be what I'm looking for, but I need help understanding how to set it up for training, validation, and testing. Could you provide a brief example of how the included code could be used to generate a training, validation, and testing set, and then from this, iterate through these datasets in batches of chosen size?

Hi, is there any public AMI on amazon so that we can try the code ? I cannot run your code a couple of different pytorch AMIs since I got a couple of errors inside the main.py file.

From the article in the "Dealing with pixels" case:

So, after convolving the image, each of the d^2 k-dimensional cells in the d × d feature maps was tagged with an arbitrary coordinate indicating its relative spatial position, and was treated as an object for the RN.

Also, the author (/u/asantoro) confirmed on reddit that objects were of the form: [x, y, v_1, v_2, ..., v_k] where k is the number of kernels and the range of the coordinates x,y doesn't matter. (Reddit link)

So I think in the model, object coordinates should be added to oi and oj. https://github.com/kimhc6028/relational-networks/blob/master/model.py#L53

for i in range(25):
    oi = x[:,:,i/5,i%5]
    for j in range(25):
        oj = x[:,:,j/5,j%5]
        x_ = torch.cat((oi,oj,qst), 1)
        x_ = self.g_fc1(x_)

I believe this should improve performance on questions where the spatial relationship between objects is important (closest, furthest, ...).

To have a clean A/B kind of model, this PR factors out the CNN input stage and the final FC stage. This highlights the differences between the two models clearly.

The specific CNN-MLP implementation has been taken from yangky11's fork. However, since my branch was taken after the '10x speedup' patch, it includes that too.

RN @ epoch 20 :  
   Test set: Relation accuracy: 87% | Non-relation accuracy: 99%
CNN-MLP  @ epoch 100 :  
   Test set: Relation accuracy: 66% | Non-relation accuracy: 69%

It's also now Python3-ready, and there are some other minor tweaks.

What do you think?

Why all of your variables here and here's requires_grad argumand set to False ? You set requires_grad parameter of coord_tensor variable to false and then you concatenate it with the output of cnn (defined as x),whose requires_grad arguman is true by default I guess, at line 78. In this case what is the requires_grad parameter of concatenated variable (output of line 78 which is also x_flat)?

Hello sir, I am reading the paper and your code. However there is one thing I dont understand from your implementation. In the paper, at the Section 4 (Dealing with pixels), the author said that: "So after convolving the image, each of the d2 k-dimensional cells in the d x d feature maps was tagged with an arbitrary coordinate ...". So my question is in your code, which part is referring to this and would you please explain more about it since it is kind a difficult for me to understand it. Thanks !

Hi, in line 21 I had a hard time understanding the calculation (24+2)*2+11 The input of the linear layer should be two objects each encoded as a vector size of 24. Where did the +2 and +11 came from?

Thanks!

Regarding the training procedure for the entire CLEVR, how did you manage to train pixel and state description stages? i.e., did you train end-to-end the whole system (LSTMs, ConvInputModel, and RN)? Or was it by stages?

Another question off the topic: What is the purpose of coord_oi and coord_oj

Thank you! Great implementation by the way.

I wanted to report that I didn't manage to replicate the results in the paper or in the repo. Relation accuracy: 80% Non-Relation accuracy: 93%

trained for 20 epochs, with all default arguments

Hi, thanks for your work and sharing of the code!!! I have on question on data generation part, I know the questions and answers are represented in one-hot vectors where questions = 2 x (6 for one-hot vector of color), 3 for question type(binary, ternary, norel), 3 for question subtype answers = yes, no, rectangle, circle, r, g, b, o, k, y

My question is why you use count+4 in here bianry question-subtype 3, which is as follows:

  elif subtype == 2:
      """count->1~6"""
      my_obj = objects[color][2]
      count = -1
      for obj in objects:
          if obj[2] == my_obj:
              count +=1 
      answer = count+4

As I understand, the count is already the number of Number of objects which have the same shape with the certain colored object. The +4 in [ yes, no, rectangle, circle, r, g, b, o, k, y] means the colors?

Any help would be appreciated and thanks for your time

Say, if in the training data the RN saw many green circles, and many yellow triangles, but it never saw green triangles or yellow circles, would the RN perform well if asked a question about those shape-color combinations it never saw in the training data? In other words, can the RN learn the abstract concept of "color", the abstract concept of "shape" and generalize those concepts to understand new questions involving novel color-shape combinations?

I do not understand well how you pair the objects. My question is if each pair of objects is feed as input once or twice. If it is fed once then how do you choose the order? to be independent of the order each pair should be fed twice, right? Otherwise there is some break of symmetry in the input pair. I could not understand this from the original article, and the code is still obscure to me to understand this. thanks! UPDATE: I think I got it know, you input all pairs twice (in different order each time), and also each object with itself, thus n^2 instead of n(n-1)/2, Is this correct?