Thank you for your apply a few days ago. I'm doing something about control the medicine dose with rl. I woder whether the td-mpc can against the mpc with biologic model. If possible, what's the advantage of td-mpc compared with mpc in this field. It's just my own question.

Firstly, thank you for the awesome paper!

I was trying to implement this using OpenAI's Atari gym instead of the physics sims and I was having some trouble getting it set up. Could you give me some tips on how I should modify the functions to make it work? Thanks!

Hi, congratulations on the amazing work!

I wanted to ask a question, the paper mentions that multimodal data [RGB + proprioception] can be used as input of the model

In the code, the observations are sent to an encoder that process them in different ways depending if it's pixels or another modality, nevertheless I'm not sure that any of those options apply to multimodal data containing both pixels and state information. Given the experiments you made in the paper, how would you recommend processing such data in the encoder?

I saw that Meta-World is used in the paper, but it doesn't seem to be in the code. How do I change the code if I want to use a Meta-World environment?

The result of my experiment on Reacher-easy is only 318±172(random seed is 1 to 5), which the article is 628±105. All settings are the same as in the article. Maybe it is because of the setting of the random seed. Could you share with us the exact value of the random seed? Thank you very much!

I ran the command python src/train.py task=dog-run save_video=True save_model=True When finished the train, I only got the model.pt, there are not video saved.

My platform is ubuntu20.04

Hi! thanks for this excellent paper and it inspired me a lot!

In my experiment, I'm confused about the device setting in cfg.py:

https://github.com/nicklashansen/tdmpc/blob/96cb7036ecf06f75d5ffd64a0454bbab7d0d3e17/src/cfg.py#L45

I don't understand why we need to make pixel's environment(modality=='pixels') based on cpu and state's environment(modality=='state') based on gpu(cuda), is there any special purpose for this setting?

Hi, while I was reading through the code implementation, I found that LayerNorm is only used after the input layer of Q-network. What is the reason for this?

I guess it stabilizes the Q-value estimation that is susceptible to wrong value estimations.

Thanks in advance!

Hi, thanks for brining a new perspective to this field with this paper. I really enjoyed reading it.

While reading through the paper, I found a typo in the inline equation that describes the MPC.

In the Model predictive control subsection under Preliminaries, it is stated that the globally optimal policy \Pi_{\theta} is proportional to the expectation of the negated Q-values. I think the negation should be removed, intuitively.

Another question regarding the description of MPPI in Sec. 3 is about Eq.(4), that describes the CEM.

Here, the mean/var of the j-th policy is computed based on weighted/shifted \Gamma, where \Gamma is denoted as sampled trajectory in the paragraph. I guess the authors meant the state-action sequence as \Gamma. Thus, \Gamma^{\star}_i in Eq.(4) should be replaced with the action I guess. Maybe the code snippet here corresponds to this equation?

I wonder if I understood it correctly. Thanks in advance!

Hi! First of all, thanks for such an interesting and well written paper!

For my experiments, I would really like to try this approach on Crafter to compare it with DreamerV2. The problem is that Crafter only supports discrete action space.

I'm mostly familiar with model-free algorithms, so it's not quite obvious to me how this algorithm can be adapted. As far as I know MCTS is used for discrete action space, but writing it efficiently in python is a challenge. Is it reasonable to try to just change the distribution in CEM from normal to categorical and sample from it? Could it work?

Hi, while I was reading through the code implementation, I found that you use zero initialization in critic and reward model last layer. What is the reason for this? I guess this can avoid some overestimation, is that right? Other question is: why you choose the orthogonal_init except last layer, is there any theory behind this? zero initialization Thanks in advance!