In line 49 of replay_buffer.py time_step is an ExtendedTimeStepWrapper objective, not a list.

Why you use indecs to extract information?

value = time_step[spec.name]

The error is

TypeError: tuple indices must be integers or slices, not str

Dear Misha Laskin,

I am very grateful for open-sourcing the well-written code.

It is really helpful for my research!

However, I have one question about the implementation of fine-tuning APS.

https://github.com/rll-research/url_benchmark/blob/710c3eb04e60ef559525bc90136ee4e1acae4c97/finetune.py#L196-L197

As shown in the code block in finetune.py, the task vector (named meta) is updated periodically "after" initial seed frames.

However, in the original paper of APS, it is said that the task vector is searched using initial seed frames, and is "fixed" during fine-tuning phase.

Therefore, I understand that the code should be revised as follows (the inequality sign is reversed): if self.global_step < ( init_step // repeat) and self.global_step % every == 0:

I wonder whether I miss something,

and I hope you provide some explanation about my question.

Best,

Junsu Kim

I couldn't find the values used for normalized score calculation neither in paper nor in repo. It would be convenient if we'd be able to compare new methods based on the same metric (mean normalized return). Also the values themselves do not appear anywhere in the paper, only on figures, which is a bit confusing.

Hi all,

I'm trying to get some data to work with ExORL, the buffer directory appears to be empty when saving the dataset. Were there any edits to urlb to generate the datasets used with ExORL? Thanks for all help :smiley:

I am running pre_training.py with: python pretrain.py agent=aps domain=walker

I believe this function should be saving the .npz buffer file:

https://github.com/rll-research/url_benchmark/blob/bb98f0c6d78b3c467fb5a9fa5bbba3b7c0250397/replay_buffer.py#L18

I would like to try out using finetuning after pretraining. I follow the instructions and use:

python finetune.py pretrained_agent=icm task=walker_run snapshot_ts=1000000 obs_type=states

Unfortunately, this gives me the following error:

Could not override 'pretrained_agent'.
To append to your config use +pretrained_agent=icm
Key 'pretrained_agent' is not in struct
    full_key: pretrained_agent
    object_type=dict\
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

What should I do here? Thanks!

Hi,

I had a few questions/comments regarding the ICM implementation.

Unlike the original "Curiosity-driven exploration by self-supervised prediction" paper, this implementation doesn't use the inverse dynamics model to learn a feature space in which forward predictions are made. In fact, it seems the inverse model is not being used for anything?

Also, is it problematic that the same encoder is being used (i) in the process of producing intrinsic rewards, and (ii) in the process of predicting intrinsic rewards (i.e. when predicting DDPG Q-values during pre-training)? I believe in the original paper the ICM module and the RL agent use a separate encoder.

I'm just wondering if you had any useful insights regarding these design choices. Apologies if I misinterpreted anything from the code or paper.

Hi, thank you very much for such wonderful work and implementation In the implementation, the encoder has its separate optimiser and does a separate update on top of agent optimiser step. Doesn't the ICM (or ddpg/AC) update the encoder parameters?

I'm wondering if there is any advantage of using separate optimiser/update for the encoder, and if it's necessary for the model ?

Thank you

    def update(self, replay_iter, step):
        metrics = dict()

        if step % self.update_every_steps != 0:
            return metrics

        batch = next(replay_iter)
        obs, action, extr_reward, discount, next_obs = utils.to_torch(
            batch, self.device)

        # augment and encode
        obs = self.aug_and_encode(obs)
        with torch.no_grad():
            next_obs = self.aug_and_encode(next_obs)

        if self.reward_free:
            metrics.update(self.update_icm(obs, action, next_obs, step))

def update_icm(self, obs, action, next_obs, step):
        metrics = dict()

        forward_error, backward_error = self.icm(obs, action, next_obs)

        loss = forward_error.mean() + backward_error.mean()

        self.icm_opt.zero_grad(set_to_none=True)
        if self.encoder_opt is not None:
            self.encoder_opt.zero_grad(set_to_none=True)
        loss.backward()
        self.icm_opt.step()
        if self.encoder_opt is not None:
            self.encoder_opt.step()

        if self.use_tb or self.use_wandb:
            metrics['icm_loss'] = loss.item()

        return metrics

When I try to fine-tune the agent after the pretraining, by below command as mentioned in README.md,

python finetune.py pretrained_agent=icm task=walker_run snapshot_ts=1000000 obs_type=states

This error comes.

Could not override 'pretrained_agent'.
To append to your config use +pretrained_agent=icm
Key 'pretrained_agent' is not in struct
    full_key: pretrained_agent
    object_type=dict

Should I modify finetune.yaml?

Hey,

Not sure if anyone can clarify just wanted to check on signs with intrinsic reward for SMM

intr_reward = pred_log_ratios + self.latent_ent_coef * h_z + self.latent_cond_ent_coef * h_z_s.detach()

The original paper in equation 3 has:

r_z(s) = log(p*(s)) - log(rho_pi(s|z)) + log(p(z|s)) - log(p(z))

Why do we add the log(rho_pi(s|z)) == pred_log_ratios and log(p(z)) == self.latent_ent_coef and not subtract them as in equation 3, sorry if this is obvious 😄

Hello! I noticed that the representation dim of some models (such as ICM) is 512 in Table 3. However, in the code, the representation of these models are 39200. When I use these models, should I add a linear layer after the conv layers to project the representation to 512 dim?

Nice work for this benchmark, and I am working on this transfering this benchmark to my custom environment. I want to enquire that how can I identify that the unsupervised RL algorithm truely work rather than some random trajectories? Any metrics that can help me identify that?

I was wondering if there was ever discussion on using the URL agents in a package. For example, I'm working in an environment with discrete actions spaces, so I need a different training script, but would like an easy way to port over the reward models.

Or, is there another exploration agent library that is better suited for that?

cc @aliciafmachado

Hi, When reading the paper and checking issue #1 , I found that the numerical results in Appendix C&F are inconsistent with the provided expert scores. For example, in Figure 7, ICM with 10^5 pretraining on walker_walk has about 50% normalized score, and the numerical result in Table 5 is (302+-45). As the expert score is 971 mentioned in #1 , this is equivalent to about 31% normalized score. Did I miss something here? Also, it seems that the score for pretraining methods, in general, cannot compete with SOTA methods like CURL and DrQ, which do not require any pretraining. Is there any explanation for this? Thanks!

When I run python pretrain.py agent=icm domain=walker save_train_video=true, I encounter an error

  File "/root/url_benchmark/video.py", line 83, in record
    frame = cv2.resize(obs[-3:].transpose(1, 2, 0),

But recording evaluation video is okay. Then I found that we call .record method differently between training and evaluating.

When we record eval video,

self.video_recorder.record(self.eval_env)

we pass 'env'.

When we record train video,

self.train_video_recorder.record(time_step.observation)

we pass observation.

Is this intended?