When loading a pretrained model for further training, the performance during training seems to have resetted with respect to the pretrained model. Bypassing the train function

policy.train(replay_buffer, args.batch_size,mean_action,scale)

does result in the performance corresponding to the trained model. policy.train seems to reset parameters somehow, but i can't seem to find how or where.

I run the source code of TD3: ourddpg algorithm and find performance in most games(apart from HalfCheetah-v3_0) is inconsistent with the original paper? Could you illustrate what might be the problem, please?

hi, I tried several experiments and found that even after setting the same random seed, the results are different each time. This confuses me. After debugging, I found that in the main.py

action = env.action_space.sample()

This line will sample different actions every time, so I want to ask whether gym.action_space should also set the same random seed?

Thank you for your outstanding work. In your paper, the estimated value and the real value are mentioned. I would like to ask about the specific calculation method. Thank you。

Could you confirm that the new hyperparameters for TD3 (i.e. network size from [400, 300] to [256, 256], batch size from 100 to 256, learning rate from 1e-3 to 3e-4) really improve the performance?

In my experiment, it does not demonstrate a consistent improvement.

The current algorithm assumes env.action_space.low = -env.action_space.high and env.action_space.high[0] = env.action_space.high[i] for all i, which is not the case for all environments.

Hi,

Thanks for your elegant codes. Here I'm asking to confirm how TD3 performs in mujoco Humanoid-v2.

In SAC paper and also other papers, they reported TD3 almost cannot make any progress on Humanoid-v2 (the learning curve is like y=0). However, we tried to reproduce this phenomenon using your code and found that TD3 could reach 6000+ scores in testing returns in 10M steps (just one seed though).

Since you didn't report humanoid results in your paper, and you might change hyperparameters to improve the results, I'm wondering which result might be true? We use the latest hyperparams and set start_timesteps=10000 in reference to https://github.com/sfujim/TD3/blob/master/run_experiments.sh#L11

Thanks!

I notice that in your paper C. DDPG Network and Hyper-parameter Comparison mention that the DDPG architecture is different from OpenAI Baseline. Is there any specific reason or intuition behind this? Thank you.

Fujimoto, S., Van Hoof, H., & Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. 35th International Conference on Machine Learning, ICML 2018, 4, 2587–2601.

Thank you very much for your code, which is very clean and effective. But I have a question: DDPG mentioned the use of noise, why didn't you add noise to DDPG?

Great work! Thanks for the codes.

Just quick questions on your three great contributions (DP, TPN, CDQ), have you ever tried the three on discrete-action-space games (just typical ones like SpaceInvader or KungFu or Seaquest etc.)? And are the three tricks providing improvement as well ? Since in the last paragraph of your paper it is mentioned that the three contributions can also be utilized for discrete-action games.

Thanks again.

hello and thank you for sharing such a great project I'm trying to solve Humanoid-V3 and I am having a problem to set target_policy_noise and target_noise_clip the default paper for 0.2 and -0.5,0.5 doesn't work for gym Humanoid-V3 because the action space is -.04 to 0.4 best regards

python main.py --expl_noise=0.1 would result in

Traceback (most recent call last):
  File "main.py", line 122, in <module>
    + np.random.normal(0, max_action * args.expl_noise, size=action_dim)
TypeError: can't multiply sequence by non-int of type 'float'

argparse takes in 0.1 as str not float, so type needs to be specified for float arguments in main.py.

1. Added a minimum action variable, included it as the lower bound for action clipping, and modified necessary range calculations.
2. Changed minimum and maximum action variables to be array-like instead of single floating point variables. torch.clamp doesn't support tensor min and max inputs therefore torch.min and torch.max have to be used instead.

The changes have been tested on Pendulum-v0 and some custom environments with array-like action spaces to ensure clipping and minimum action are handled properly.