Hi folks.

I am willing to work on a tutorial that shows how to extend graph regularization example in the same way it's done for text-based problems. Is there a scope for this tutorial inside this repo?

Hi @arjung as discussed. Please find the adversarial learning notebook example.

I have added it in the neural_structured_learning/examples/notebooks as you had mentioned. Feel free to suggest the necessary edits and changes to keep in line with Google & TF standards and I will make the necessary changes based on feedback.

Hi,

I’m interested in neural structure learning(graph neural network) and want to apply some basic gnn models such as GCN, GAT using Tenserflow2.0. I have some ideas on how to implement it, but I am not sure whether such a model architecture is clear. Here is the code structure of gnn I roughly thought:

• Create a folder named ‘models’ to design various gnn models structure
• In ‘models’ folder, create a file named sublayers.py which defines a single graph layer as a class called: GraphConvLayer, GraphAttenLayer, etc…
• Design base node & edge models to let other gnn models inherit them (increase flexibility, readability)
• Bulid functions in utils.py(or a folder named utils) such as evaluate, inference, calculate_loss, etc… to make users more convenience

About the above structure, I have slightly referred to the repository of Deep Graph Library(dgl). As mentioned before, I want to apply some gnn model application but don’t know if my idea is appropriate. My initial thought is maybe we can create a folder just called gnn-survey-paper or somewhere else to put these implementations of gnn models. This is the first time I tried to post an issue and hope to contribute to the open-source code. If there is anything unclear above or there are some recommendations from your team, just feel free to let me know. Thanks:)

Best regards, Josh

Hi, I'm implementing a Keras binary image classifier using VGG16 with Adversarial Regularization. After initialization of the VGG16 model layers, I'm configuring the Adversarial Regularizer using the following code -

import neural_structured_learning as nsl

adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(custom_vgg_model, adv_config)
adv_model.compile(tf.keras.optimizers.SGD(learning_rate=2e-5), loss='categorical_crossentropy', metrics=['accuracy'])

When I execute the code, I'm getting the following error -

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-32-bb6bdecb015d>](https://localhost:8080/#) in <module>()
      1 adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
      2 adv_model = nsl.keras.AdversarialRegularization(custom_vgg_model, adv_config)
----> 3 adv_model.compile(tf.keras.optimizers.SGD(learning_rate=2e-5), loss='categorical_crossentropy', metrics=['accuracy'])

2 frames
[/usr/local/lib/python3.7/dist-packages/neural_structured_learning/keras/adversarial_regularization.py](https://localhost:8080/#) in _build_labeled_losses(self, output_names)
    554       return  # Losses are already populated.
    555 
--> 556     if len(output_names) != len(self.label_keys):
    557       raise ValueError('The model has different number of outputs and labels. '
    558                        '({} vs. {})'.format(

TypeError: object of type 'AdvRegConfig' has no len()

How do I resolve this issue?

@arjung I reckoned that adding the example under g3docs/tutorials might be more appropriate because it would be the first one that shows how to build a synthetic graph for an image dataset and build a model.

Let me know your thoughts. A parallel Colab Notebook is available here for the results. For reference, the corresponding issue thread is available here.

Hello

The ValueError error Insufficient elements in branch_graphs[0].outputs. is found in both Neural Graph Learning tutorials (sentiment and document classification).

Both happen during the training of graph_reg_model.fit() process. The error messages are shown below, I think it is due to the same issue.

• Graph regularization for document classification using natural graphs

Epoch 1/100
      1/Unknown - 0s 303ms/step

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-20-0c7d19de6181> in <module>()
     10     loss='sparse_categorical_crossentropy',
     11     metrics=['accuracy'])
---> 12 graph_reg_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)

25 frames

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/cond_v2.py in _make_indexed_slices_indices_types_match(op_type, branch_graphs)
    650                      "Expected: %i\n"
    651                      "Actual: %i" %
--> 652                      (current_index, len(branch_graphs[0].outputs)))
    653 
    654   # Cast indices with mismatching types to int64.

ValueError: Insufficient elements in branch_graphs[0].outputs.
Expected: 11
Actual: 10

• Graph regularization for sentiment classification using synthesized graphs

Epoch 1/10
      1/Unknown - 2s 2s/step

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-30-e49eed0ffe51> in <module>()
      3     validation_data=validation_dataset,
      4     epochs=HPARAMS.train_epochs,
----> 5     verbose=1)

25 frames

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/cond_v2.py in _make_indexed_slices_indices_types_match(op_type, branch_graphs)
    650                      "Expected: %i\n"
    651                      "Actual: %i" %
--> 652                      (current_index, len(branch_graphs[0].outputs)))
    653 
    654   # Cast indices with mismatching types to int64.

ValueError: Insufficient elements in branch_graphs[0].outputs.
Expected: 18
Actual: 17

Thanks

Is there a particular format when combining with ImageDataGenerator? Error: OperatorNotAllowedInGraphError: iterating over tf.Tensor is not allowed in Graph execution

This PR adds Denoised Smoothing under the research folder.

The success of Randomized Smoothing is proven and it works for many different scenarios. But it also operates under the assumption that the underlying classifier is able to perform well under Gaussian perturbations. Won't it be better if we could just take our standard pre-trained image classifiers (including the Cloud APIs) and have the benefits of Randomized Smoothing inside of them in an easy manner?

That is preciously what Denoised Smoothing does by prepending a Denoiser to an image classifier and still maintains the theoretical guarantees of robustness against L2 attacks.

Besides, the implementation includes a suite of utilities that may be helpful to generate robustness certificates. I think that gives us a unique opportunity to actually design an API inside NSL that would allow easy generation of robustness certificates. To the best of my knowledge, there does not exist any such framework as we speak.

Hi,

I have been thinking about using NSL in the context of link prediction. It can definitely be reframed as a classification problem I believe. The only thing I am wondering about is if anyone has thought of an elegant way to add the neighbours (I guess in this case of both nodes which form the link ). Has anyone been working on this?

I guess a decent way of going about it would be changing the parse_example function:

def parse_example(example_proto):
  """Extracts relevant fields from the `example_proto`.

  Args:
    example_proto: An instance of `tf.train.Example`.

  Returns:
    A pair whose first value is a dictionary containing relevant features
    and whose second value contains the ground truth labels.
  """

so, that it can take a pair of examples which are part of a link.

Thanks, George

Hi @arjung as discussed. Please find the adversarial learning notebook example updated based on feedback from your side.

You can review it and add in any comments as needed to fix any pending issues. Also would be great if you can close PR: https://github.com/tensorflow/neural-structured-learning/pull/67

Zoey and I are partners working on a project to help improve NSL documentation (specifically this tutorial).

I noticed in online communities and from personal experience, while the corresponding Youtube videos verbally explain concepts well for beginners, the recommended interactive collabs were a bit difficult to follow for beginners. That's why I wanted to better connect the video content with the collab content thus I created a recap section for beginners that includes all the concepts explained in the Youtube video. Also for some, they may prefer reading content over watching, some may also just want to have a written reference integrated into the collab tutorial.

Hello, I was trying to use NSL to implement adversarial training on my custom model, so I followed the default steps in the tutorial video, which worked like a charm. While studying the code, I noticed that the call to make_adv_reg_config() has a parameter called pgd_epsilon, which is "...Only used in Projected Gradient Descent (PGD) attack".

This statement suggests that NSL can use different attacks in adversarial training; however, it is not clear how to select which attack to use, or which attack is currently in use. Up till now I had assumed that PGD was being used by default, as this is common in literature, but I would like to know if this is actually the case, and by extension if it is possible to use a different attack and how that can be done.

Thanks!

I want to train the model without adversarial attack at the first two epochs. After that, I would like to train the model with adversarial learning.

In summary,

1 epoch: Training w/o adversarial 2 epoch: Training w/o adversarial 3 epoch: Training with adversarial 4 epoch: Training with adversarial ....

Is it possible to adjust the start epoch of the adversarial training? I couldn't find any related parameter in nsl.configs.make_adv_reg_config

The current implementation of nsl.tools.pack_nbrs does not support reading and writing to multiple tfrecord files. Given the extensive optimizations made available by the tf.data API when working with multiple tfrecords, supporting this would yield significant performance gain in distributed training. I would be willing to contribute to this

Relevant parts of the code

• for reading https://github.com/tensorflow/neural-structured-learning/blob/c21dad4feff187cdec041a564193ea7b619b8906/neural_structured_learning/tools/pack_nbrs.py#L63-L71
• for writing https://github.com/tensorflow/neural-structured-learning/blob/c21dad4feff187cdec041a564193ea7b619b8906/neural_structured_learning/tools/pack_nbrs.py#L264-L270