Hi. Thanks for publishing implementation of trpo.

I have question about get_kl().

I thought what get_kl() is supposed to do is to calculate the kl divergence of old policy and new policy, but this get_kl() seems always returning 0.

Also,I do not see kl constraining part in the parameters updating process.

Is this code the modification of trpo or do I have some misunderstanding?

Thanks,

like eg, imagine I have my own policy, that takes in a state, and outputs an action, or perhaps a distribution over actions; and I have a world that takes an action, and returns a reward and a new state, how would I plug these into this TRPO implementation?

Hi, thanks once again for implementing a really interesting algorithm in PyTorch :+1: ,

I was wondering how to modify the code to be able to use it for environments which require discrete actions, (say cartpole as in the other pytorch trpo implementation, or maybe even Atari games)?

I'm new to pytorch and am having a hard time getting used to handling the variables properly on cpu and gpu. As we are calculating our own losses here, I am having trouble understanding what and when to send to the device (gpu). Would really appreciate an explanation of how to go about this. The code is quite well written and easy to understand btw.

Dear author: I found your code very helpful. However, I have problems trying to read the following code:

https://github.com/ikostrikov/pytorch-trpo/blob/eb26e29ed75b7c7b46b0c717331cc7488ab16b8d/main.py#L111

I wonder the usage of volatile flag. I want to when u set volatile to True/False.

Hello， I wanna ask that in line 67 in your trpo.py, you will get two terms, and in the TRPO paper, he said the second term vanishes ?, and you add v*damping, I guess its function is to make sure the positive definiteness? , could you explain it in detail? thank you very much! and in your line 117 in your main.py, could you explain why this can approximate the average KL in detail? thank you very much!

hello, so I notice your code is about mujoco, and I wonder how to modify it to fit other env, I have tried but failed. thx a lot! ikostrikov, thx very much. I have tried one continuous control game "MountainCarContinuous-v0" in classical control and it succeeds.

Thanks for your great code! I notice that in the function get_kl(), you use policy net to generate the mean, log_std and std, then copy these three parameters and calculate the KL divergence between the original parameters and the copied parameters, which is obviously zero all the time. Is this a bug or a intended behavior?

Thank you very much for the code you provided！I learn a lot from it . I would like to ask what is the function of these lines of code, is there any mathematical proof or the like, thank you very much！！！these are different from the original paper？Thanks！！！

neggdotstepdir = (-loss_grad * stepdir).sum(0, keepdim=True)
expected_improve = expected_improve_rate * stepfrac
ratio = actual_improve / expected_improve
 if ratio.item() > accept_ratio and actual_improve.item() > 0:

It would be nice if the agent was an object (with methods "get_action" and "remember" or similar) so that it could be more easily reused.

Currently, the target for the value function is the discounted sum of all future rewards. This gives unbiased estimate but will result in higher variance. An alternative is to use bootstrapped estimate, i.e. something like target[i] = rewards[i] + gamma * prev_values * masks[i]

Bootstrapping is often preferred due to low variance, even though it results in biased gradient estimate.

Hi, I am a newcomer to drl. When I try to read trpo_step in trpo.py, I notice that you use a linesearch method instead of trust region for numerical optimization. So I want to know why you choose that method and dose it conflict with a "trust region" policy gradient algorithm?

In your main.py, line 147: for t in range(10000): # Don't infinite loop while learning But actually, the t ends at 50, because the env is done in 50 steps. so the range(10000) is so big and not necessary.