In the File "/media/iiau/LiGong4T2/zwb/code/l2p/train_continual.py", line 42, I can see:

from models import resnet_v1

but in "./models" folder, i can't find resnet_v1, so I get error like this:

ImportError: cannot import name 'resnet_v1' from 'models' (unknown location)

How to solve this problem? Thanks!

Hi, Thanks for your nice work!

I am currently trying to follow your Split ImageNet-R benchmark but encounter some problems. Firstly, I am not able to download the ImageNet-R dataset. I've got errors like this. 2022-10-25 16:43:39.764732: E tensorflow/core/platform/cloud/curl_http_request.cc:614] The transmission of request 0x668a800 (URI: https://www.googleapis.com/storage/v1/b/tfds-data/o/dataset_info%2Fimagenet_r%2F0.2.0?fields=size%2Cgeneration%2Cupdated) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.004201 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error) It seems like a networking issue, so I access the uri(https://www.googleapis.com/storage/v1/b/tfds-data/o/dataset_info%2Fimagenet_r%2F0.2.0?fields=size%2Cgeneration%2Cupdated) through my web browser, but I got such a info:

{ "error": { "code": 404, "message": "No such object: tfds-data/dataset_info/imagenet_r/0.2.0", "errors": [ { "message": "No such object: tfds-data/dataset_info/imagenet_r/0.2.0", "domain": "global", "reason": "notFound" } ] } }

It seems that the dataset is not accessible at this time.

Secound, I tried to understand the codes you priveded in libml/input_pipeline.py and try to understand 2 things: (1) How to split the training and testing set from the original dataset. (2) What is the training order of the classes in continual learning tasks.

For (1), I found that the codes using TFDS split, in Line414-424 in input_pipeline.py. I am not familiar with TFDS, Is this a stratified sampling (based on classes) or just a split based on the whole data?

For (2), I found that the codes permutate class order here, if the config.continual.rand_seed is set. However in configs/imr_dualprompt.py, config.continual.rand_seed is set to $-1$. Does this mean that the dualprompt conducts experiments in the order of natural numbers？ (i.e,. 0, 1, 2, 3, ...)

Hope you counld kindly help me :)

Looking forward to your early reply and thanks again!

I am trying to reproduce the results of the tables included in your nice paper. And I don't think that the code you have provided allows me to reproduce the results of Table 3. Table 3 uses CORe50, and I was curious to know when the code that produces the results of Table 3 will be available.

Sorry to take your time with this, and hope you are doing well.

Dear author.

Thank you for your great work.

I'm having a little problem with reproducing L2P.

First, please modify the environment setup of README.md.

The link for adjust the jax version in README does not support CUDA version. I think the link

https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

seems to have changed to the link above.

The other thing is that even if batch_size is set to 1, I can't run the code completely in my 4 A5000 GPUs because of out of memory issue. Can you also provide small models such as ViT-Tiny or ViT-Small?

I look forward to hearing from you.

Thank you.

Dear author,

Thank you for your excellent work!

I am a little curious about the pretrained model, it is trained on the entire ImageNet-21k dataset, and is fixed during training. But will this lead to information leakage?

Take the class incremental setting as an example, I think all 100 classes of CIFAR100 can be found in ImageNet-21k so it is possible that the model has already learned all the features necessary for CIFAR100. But in practice, the model is expected to learn new features. We can not assume the classes in new tasks have already been observed by the backbone, right?

Have you tried to remove CIFAR100 classes in ImageNet and pretrained a model or evaluate the model on some datasets disjoint with ImageNet?

Thank you very much!

Thanks for the great idea and the result!

As the title says, I'd like to know how to use optionally diversifying prompt-selection, I don't see where to use the arguments for this method, nor do I see an implementation of it in . /models/prompt.py

I would like to ask about how to normalize the frequency of each prompt into a penalty factor, I don't see a specific description in the paper.

Hi, thanks for the great work!

With the script you provided, I successfully reproduced the L2P result (of Split CIFAR-100 dataset) without replay.

I am now trying to reproduce the results with replays, but even using replay buffer storing 50 samples/class, I got much lower results (acc 80.10/forgetting 9.13) compared to the reported ones (acc 86.31/forgetting 5.83).

It doesn't make sense that using replay leads to higher forgetting. I guess I might miss something.

Since there is no examples how to use replays in given configuration file (i.e., cifar100_l2p.py), I added some lines to handle replays as below.

  # add for replay
  config.continual.replay = ml_collections.ConfigDict()
  config.continual.replay.num_samples_per_class = 50
  config.continual.replay_no_mask = True
  config.continual.replay_reverse_mask = False
  config.continual.replay.include_new_task = True
  config.continual.review_trick = False
  config.continual.num_review_steps = -1
  config.continual.num_review_epochs = 5

I guess review_trick is for fine-tuning the model with balanced dataset.

Strangely, when I set review_trick=True, I got much lower result (especially, very low learning accuracy).

And when I set review_trick=False, then the model is kept updating with replays, but still it shows much low accuracy (acc 80.10/forgetting 9.13)

Do you have any advice on what I am missing or where to modify in your code?

Or can you share the correct configuration files to reproduce the result of Table 1 using replay buffer?

Thank you.

I've read the paper and have some small questions about the paper's comparison to other models like EWC, ER and DER++ because I've not gained or maybe missed the information about the details of conducting these methods in the pretrained ViT model. Which part of the ViT is trained or finetuned in Upper-bound and those methods and which parts are using pretrained weights? I guess only the classifier is trained but need some confirmation.

The constraint_coefficient of surrogate loss is 0.5 in the paper.

But, in the codes, that is set to -0.1.

What is correct? Which is better?

Thank you.

Hi, I am trying to run the model on a CIFAR100 dataset. I am getting the following error. I have 4 Tesla V100 GPUs.

2022-08-05 10:00:26.833817: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:479] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184)requested by op 
2022-08-05 10:00:26.835182: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:491] *********************************************************************************x**************x***
2022-08-05 10:00:26.835281: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2130] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 9437184 bytes.
BufferAssignment OOM Debugging.

The complete running logs can be found here. Please help me with solving the issue.

===============

For your information, I was getting a RuntimeError: Visible devices cannot be modified after being initialized error. Hence, I added the following code snippet in main.py from https://www.tensorflow.org/guide/gpu, and it solved the issue.

"""Main file for running the example."""

import os
os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false"

imports ...

FLAGS = flags.FLAGS
...

def main(argv):
  del argv

  # Hide any GPUs form TensorFlow. Otherwise TF might reserve memory and make
  # it unavailable to JAX.
  # tf.config.experimental.set_visible_devices([], "GPU")

  gpus = tf.config.list_physical_devices('GPU')
  if gpus:
    # Restrict TensorFlow to only use the first GPU
    try:
      tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
      logical_gpus = tf.config.list_logical_devices('GPU')
      print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU")
    except RuntimeError as e:
      # Visible devices must be set before GPUs have been initialized
      print(e)

  # if gpus:
  #   # Create 2 virtual GPUs with 1GB memory each
  #   try:
  #     tf.config.set_logical_device_configuration(
  #         gpus[0],
  #         [tf.config.LogicalDeviceConfiguration(memory_limit=1024),
  #         tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
  #     logical_gpus = tf.config.list_logical_devices('GPU')
  #     print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs")
  #   except RuntimeError as e:
  #     # Virtual devices must be set before GPUs have been initialized
  #     print(e)


  if FLAGS.exp_id:
     ...

Dear authors,

Thanks for your great job in building CIL learners with pre-trained models. I have a simple question regarding the pre-trained ViT. I noticed there are several versions of pre-trained ViT on the market. Seeing the fact that the current repo suggests downloading the pre-trained model from https://storage.googleapis.com/vit_models/imagenet21k/ViT-B_16.npz, does it mean the model is pre-trained with IN21K?

Thx in advance.

In 'Learning to prompt for continual learning' paper, I understand 'FT-seq-Frozen' in Table 1 as a naive prompt tuning at the input token feature.

To implement the FT-seq-Frozen setting in CIFAR100, I set prompt pool_size as 1. The result shows Acc@1 81.49 with Forgetting 6.3667.

Any point that I missed? How did you set the hyperparamters for FT-seq-Frozen? Specifically, did you set the argument 'train_mask = False' for FT-seq-Frozen?

Dear authors,

Thank you for open-sourcing your work. I am slightly confused about the metrics you guys used for the evaluation. Here is my understanding from your readme.

1. accuracy_n: accuracy evaluation on only the n-th task after training for the n-th task.
2. forgetting: Average forgetting up until the current task.
3. avg_acc: Average evaluation accuracy after training for the n-th task.

for (3), after training for the n-th task, we got,
acc_per_task = [a1, a2,...an];
avg_acc = average(acc_per_task);

Is it right?

Hi, thanks for the interesting work.

I have one question regarding the choice of prompt during the testing.

It seems that both DualPrompt and L2P use batch mode during testing, and each batch will choose the same prompt using majority voting. However, assuming that every test sample per batch is from the same task is questionable. Do you have any thoughts about this? Looking forward to hearing from you!

Bests, Yuansheng

I sincerely question the reproducibility of the code and the results of the paper, in this repo issue:

1. I have seen people using the same V100 GPU as the author in time, but unable to run through the code and experiencing OOM errors. No reply was forthcoming #1 #20
2. My own reproduction of the code does not achieve the results shown in the left panel of Figure 3 in the paper, and I do not understand why catastrophic forgetting does not occur when such statistical results occur. And even without using the Optionally diversifying prompt-selection method, I can't get this statistic, same as in the #18 #24 This issue comes with detailed statistics logs. By looking at the Histogram records, we can see that only four prompts were selected and that all tasks share these prompts. I think this is inevitably going to cause catastrophic forgetting.
3. The use of pre-trained ViT may have caused an information leak #11.
4. The given requirement.txt does not directly install the required runtime environment, and even if it does, it will only run on the CPU #1.

And, for myself:

1. This code is really hard to run on my RTX 3090 GPU, and even after a lot of effort and without any error reporting, the program is stuck at training step 5 of the first task.
2. I have not seen anyone in the issues who has successfully reproduced the results.

I very sincerely hope that the author will answer the above questions.