Tutorial 7 - Hey thanks for these wonderful notebooks. I was going through the GNN code especially the GAT section and just to be sure I understood everything correctly i replicated everything for your special case.

    a_input = torch.cat([torch.index_select(input=node_feats_flat, index=edge_indices_row, 
           dim=0), torch.index_select(input=node_feats_flat, index=edge_indices_col, dim=0)
            ], dim=-1) 

    # Calculate attention MLP output (independent for each head)
    attn_logits = torch.einsum('bhc,hc->bh', a_input, self.a)
    attn_logits = self.leakyrelu(attn_logits)

In this line you stack the features of the nodes in each edge so say we have two nodes i, j we get a 2x2 matrix corresponding to them represented as [[a, b] , [c, d]] and we have the attention weights [[w, x] , [y, z]]. When we do the einsum operation we get a 2x1 matrix [[aw+bx, cy+dz]]. This is the first doubt shouldnt it be [aw+bx+cy+dz] according to the equation as for each i, j we have one value. If we have two heads then the a matrix should have shape 2x2*d where d=2 for our case

But going further down keeping the same calculations as above. After the attention probabilities.

node_feats = torch.einsum('bijh,bjhc->bihc', attn_probs, node_feats) which is this line

which can be expanded into where ap is the attention probabilites and feats is the node features after the linear projection

       for i in range(4):
           p1 = ap[i, :, 0]
           p2 = ap[i, :, 1]
  
           f1 = feats[:, 0, :].squeeze() ## dimension 1 
           f2 = feats[:, 1, :].squeeze() ## dimension 2 
           p1.shape , f1.shape, f2.shape

           print(torch.tensor([(torch.dot(p1, f1), torch.dot(p2, f2))]))

we see that the results is obtained by taking the two different probabilites from different heads and taking the dot product with two different dimensions of the feature matrix, but each head should operate on both the dimensions of the node features or atleast I hope it should. I check the output at each intermediate stage to be sure that it matches the results from the notebook you provide. Am i missing something

Tutorial:-1 (6)

Describe the bug This is more of a clarification question than a bug. First of all, thanks for the excellent tutorial documentation. It's been very clear overall.

The reason I'm reaching out is to ask if a little more explanation could be provided on how and where to insert and apply the key padding mask to the attention_weights. Specifically, I have a Tensor of the form [True True True False False] for every sequence in the batch ([Batch, SeqLen]), with False marking padding tokens.

However, scaled_dot_product shown below wants the mask to have the following dimensions: [Batch, Head, SeqLen, SeqLen]. To this end, I have simply expanded key padding mask in the row dimension (using, key_padding_mask.view(bsz, 1, 1, seqlen).expand(-1, num_heads, seqlen, -1)), yielding the following square [SeqLen, SeqLen] mask for a sequence:

[[True True True False False], [True True True False False], [True True True False False], [True True True False False], [True True True False False]]

I do this somewhere upstream, in the forward definition of TransformerPredictor. Next, the same mask is fed all the way down to scaled_dot_product where it is then used to mask out False tokens, rendering the attn_logits -9e15 where there used to be a False. However, in contrast to a previous attempt using length-normalized sequences, the model does not manage to learn. This makes me wonder whether the above implementation is not how it was meant to be designed. Am I missing anything important here?

def scaled_dot_product(q, k, v, mask=None): d_k = q.size()[-1] attn_logits = torch.matmul(q, k.transpose(-2, -1)) attn_logits = attn_logits / math.sqrt(d_k) if mask is not None: attn_logits = attn_logits.masked_fill(mask == 0, -9e15) attention = F.softmax(attn_logits, dim=-1) values = torch.matmul(attention, v) return values, attention

Thank you for your great tutorials!

I'm tring tutorial 11 and have 2 questions on the dequatization and the quantization process (codes in 6th - 8th cells).

• You mentioned, between 7th and 8th cells, that the test fails because of numerical inaccuracy. Is it really correct?

I found 3 ldj updates for dequantization process and 2 for quantization process. I guess this means the quantization process is not theoritically invert of the dequantization process. This is because scaling to prevent boundaries 0 and 1 for the dequantization process in sigmoid function in the 6th cell.

z = z * (1 - self.alpha) + 0.5 * self.alpha

I add codes in sigmoid function for quantization process:

ldj -= np.log(1 - self.alpha) * np.prod(z.shape[1:]) z = (z - 0.5 * self.alpha) / (1 - self.alpha)

With these code, the test succeeded.

Smaller values(z < self.alpha) can also be shifted to z = self.alpha, I guess. This does not require ldj update. And, of course, because the test fail is not serious and ldj update is very small, we can ignore this.

• The figure, output of 8th cell, shows the probability distribution after dequantization. Is the figure is correct?

The area of -0.5 < z < 0.5 is larger than 1.5, I guess. It means all area is much larger than 1. And I found plotted "prob" array in 8th cell is [1, 1, ..., 1]. In the cell, prior array is assumed 1 for each value means uniform distribution. So, the "plot" array should be normalized before muliplied by the "prob" array:

prob = prob * prior[out] / quants

Theoritically, the figure shows prob = e^{-z}/(1+e^{-z})^2. The output of the modification looks like e^{-z}/(1+e^{-z})^2.

Thank you.

Hi, Thanks for sharing such a great notebook! I have a question about the vertical and horizontal convolution stacks in tutorial 12. Based on your explanation:

The vertical convolution is not allowed to work on features from the horizontal convolution. In the feature map of the horizontal convolutions, a pixel contains information about all of the "true" pixels on the left. If we apply a vertical convolution which also uses features from the right, we effectively expand our receptive field to the true input which we want to prevent. Thus, the feature maps can only be merged for the horizontal convolution.

I'm still confused about why for horizontal convolution we need to add horiz_conv(horiz_img) + vert_img but for vertical convolution, we only need vert_conv(vert_img).

Would appreciate if you can explain more about this!

Tutorial: 8

Describe the bug The loss function used in the implementation of 'DeepEnergyModel()' in tutorial 8 (cdiv_loss) does not match the loss described earlier in the algorithm. It should be: cdiv_loss = real_out.mean() - fake_out.mean(), not the other way around.

To Reproduce (if any steps necessary) Steps to reproduce the behavior:

1. Go to Tutorial 8
2. Scroll down to Training Algorithm
3. See error in Algorithm 2 vs code in cell 6.

Expected behavior cdiv_loss = real_out.mean() - fake_out.mean()

In the Tutorial 9 when comparing latent dimensionality, the plots show reconstruction results on the train dataset. If the model overfits (for example, if someone decides to make the model bigger) train image reconstruction quality might become misleading. I recommend using test dataset images for that experiment.

Hi,

the dataset (https://surfdrive.surf.nl/files/index.php/s/6YWMO1eiVXI4EkB/download) that is used in https://github.com/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/DL2/sampling/graphs.ipynb seems to be unavailable. Is there any way to still download (or generate) it?

Best and thanks for your help, Gerrit

It seems like the implementation of MultiheadAttention is not consistent with the "Multi-Head Attention" figure. In particular, the projection: self.qkv_proj = nn.Dense(3*self.embed_dim,...) Should actually be: self.qkv_proj = nn.Dense(3*self.embed_dim*self.num_heads,...) Am I missing something?

[this would also require to change the line: values = values.reshape(batch_size, seq_length, self.embed_dim) to: values = values.reshape(batch_size, seq_length, -1) ] Thanks.

In Tutorial 17, SimCLR implementation; you've mentioned that you didn't use color distortion incase of train image transforms. Because it changes the color distribution which is an important feature for classification. But you've used RandomGrayscale(p=0.2) in the train image transforms. Converting an RGB image to Grayscale image changes the color distribution right?

Also can you point to the resource which tells that color distribution is an important feature?

Thank you for your great tutorials!

I tried tutorial 11 on my laptop (it has no gpu.) and I got a runtime error in train_flow function. Its error message said map_location in torch.load should be set. So I modified ckpt = torch.load(pretrained_filename) to ckpt = torch.load(pretrained_filename, map_location=device).

I guess this modification is good for PCs without gpu.

Thank you.

Tutorial: 6

Describe the bug In the MultiheadAttention.forward method, the line:

        values = values.reshape(batch_size, seq_length, embed_dim)

should read:

        values = values.reshape(batch_size, seq_length, self.embed_dim)

The embed_dim should not come from the input tensor, i.e. instead of:

        batch_size, seq_length, embed_dim = x.size()

we should probably have something like:

        batch_size, seq_length, _ = x.size()

or

        batch_size, seq_length, input_dim = x.size()

To Reproduce (if any steps necessary) Steps to reproduce the behavior:

1. Go to the In [5]: cell, the one containing class MultiheadAttention(nn.Module):
2. Run it
3. Insert a cell under it
4. Run the following:

batch_size = 3
seq_len = 11
input_dim = 13
num_heads = 19
embed_dim = 17 * num_heads

mha = MultiheadAttention(input_dim, embed_dim, num_heads)

input_tensor = torch.rand((batch_size, seq_len, input_dim))
values = mha(input_tensor)

values.shape

which yields the following error:

RuntimeError                              Traceback (most recent call last)
[<ipython-input-50-38c850c37259>](https://localhost:8080/#) in <module>
      8 
      9 input_tensor = torch.rand((batch_size, seq_len, input_dim))
---> 10 values = mha(input_tensor)
     11 
     12 values.shape

1 frames
[<ipython-input-49-45be71448f04>](https://localhost:8080/#) in forward(self, x, mask, return_attention)
     36         values = values.permute(0, 2, 1, 3) # [Batch, SeqLen, Head, Dims]
     37         # values = values.reshape(batch_size, seq_length, embed_dim)
---> 38         values = values.reshape(batch_size, seq_length, embed_dim)
     39         o = self.o_proj(values)
     40 

RuntimeError: shape '[3, 11, 13]' is invalid for input of size 10659

Expected behavior After making the suggested change, the output is:

torch.Size([3, 11, 323])

which is what I was expecting to get.

Runtime environment (please complete the following information): Google Colab, both CPU and GPU.