Thanks for the great work! I was wondering whether you have tried to relax the constraint that eps must be greater than zero. I would be grateful if you could tell me.

The basic procedure sounds like

• (a) set \epsilon = 0, get grads to update \theta
• (b) evaluate new \theta on clean validation set .
• (c) set - grads (validation loss w.r.t \epsilon) as \epsilon
• (d) use new \epsilon to evaluate noisy data and update parameters again

My question is when \epsilon=0, the derivative of loss w.r.t. \theta =0? That means we don't update \theta actually in (a)?

Hi, thank you for open-sourcing learning-to-reweight-examples, interesting work!

When I tired the code about CIFAR experiments, there is a file missing exception:

File "~/projects/cifar/cifar_train.py", line 54, in from models.cnn.configs.resnet_model_config_pb2 import ResnetModelConfig ModuleNotFoundError: No module named 'models.cnn.configs.resnet_model_config_pb2'

many scripts in cifar folder import this script resnet_model_config_pb2, but the file did not exist. is it a public file that I should download from the internet?

Hi, thanks for sharing your implementation. I have some questions about it:

1. Does it also work on tabular data?
2. Is the code tailored to the datasets used in the paper or can one apply it to any data?
3. Is it possible to identify the noisy instances (return the noisy IDs or the clean set)?

Thanks!

In Tensorflow 2, eager execution seems to remove epsilon (ex_wts_a) from the computational graph, and the derivative of Lg with respect to epsilon is none. Is there a way to make it work in Tensorflow 2?

epsilon = tf.Variable([0.0 for i in range(batch_size)], dtype=tf.float32) with tf.GradientTape(persistent=True) as tape: tape.reset() yf_hat = damage_classifier_model(Xf) binaryCrossEntropy = tf.keras.losses.BinaryCrossentropy() loss_f = binaryCrossEntropy(yf,yf_hat) Lf = tf.reduce_sum(loss_f * epsilon) trainable_vars = damage_classifier_model.trainable_variables delta_theta = tape.gradient(Lf, trainable_vars) damage_classifier_model.optimizer.apply_gradients(zip(delta_theta, trainable_vars)) yg_hat = damage_classifier_model(Xg) Lg = damage_classifier_model.compiled_loss(yg,yg_hat)

delta_epsilon = tape.gradient(Lg, epsilon) delta_epsilon

If Adam optimizer is used, can it still work ? (Line 7. indicates a standard gradient desent method) Or, this just fit into the SGD based optimizer? 2. I would like to use this reweighting strategy in more complicated neural framework such as LSTM, BERT for other downstream tasks. Whether I must modify these to the 'meta-style' structures ? It seems to be trivial.

In file generate_nosi_cifar_data.py, there are some issues I don't understand:

1. The paper claim using 10 images per class, but in fact the code choose randomly first 100 images, so it is not guaranteed to have 10 images per class. When I test the number of image per class is around 9-10-11-12
2. The flip_data_background function: It does not flip data to background, but instead flip 40% data to another random class among number of class, and there is no guarantee they flip to wrong class as well.

Please help me clarify these issues

Thanks very much for taking the effort to make your code available. While reproducing your results, I found that the default number of clean examples in the Cifar experiments is 100, while the paper reports using 1000. Is the number 1000 in the paper correct? If so, maybe you could change the defaults in your code.

https://github.com/uber-research/learning-to-reweight-examples/blob/0b616c99ecf8a1c99925322167694272a966ed00/cifar/cifar_train.py#L698

https://github.com/uber-research/learning-to-reweight-examples/blob/0b616c99ecf8a1c99925322167694272a966ed00/cifar/cifar_train_background.py#L715

PS: does your Cifar code also include the MentorNet baseline somewhere?