Implementation of algorithms for continuous control (DDPG and NAF).

I tried running the code with HalfCheetah and commented out the wrappers.Monitor(...) line and any line that rendered the result. I get the error:

Traceback (most recent call last): File "main.py", line 98, in agent.update_parameters(batch) File "/home/sbhupatiraju/pytorch-ddpg-naf/naf.py", line 126, in update_parameters loss.backward() File "/home/sbhupatiraju/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 156, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables) File "/home/sbhupatiraju/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py", line 98, in backward variables, grad_variables, retain_graph) RuntimeError: element 0 of variables tuple is volatile

Any idea on what might be going on?

I implemented parameter noise for DDPG, as described here: https://blog.openai.com/better-exploration-with-parameter-noise/

I found that the trained policies resulting from DDPG with OU noise were similar to the one shown in the blog post above; the agent flipped onto its head and scooted forward, with the reward plateauing around 1000. With parameter noise, the learned policy looked more like running and ended with rewards of 3000+.

I modified the policy network architecture for DDPG to include layernorm, which is necessary for using parameter noise.

Parameter noise is not implemented yet for NAF, because I am not too familiar with the algorithm. The only change in the NAF code is an extra argument to select_action to make the interface the same as DDPG’s.

I also added utility functions for saving and loading networks.

To-do: decay parameter noise like how it’s done for OU noise

https://github.com/ikostrikov/pytorch-ddpg-naf/blob/d2e587a741e5220dfc6eceb7896ba7d6fe5c2630/naf.py#L121 current Target: r_t + \gamma * mask + v_{t+1} correct Target: r_t + \gamma * mask * v_{t+1}

Implemented DDPG and tried to make it as close as possible to your implementation of NAF.

I've also made some changes in NAF (seems to work better now, at least for Pendulum-v0).

(Both algorithms seem to give similar results for Pendulum-v0 but NAF has a bit more variance compared to DDPG).

Hi,

I am doing some work about RL, and very interested in the two algorithms. I have tried to train your models both on CPU and GPU, however, both outputted "out of memory" error. The memory in use was keeping increasing. It seems that the data and/or the model in former steps are not released . And the code is very similar to the example, as follows:

    action = agent.select_action(state, ounoise, param_noise)
    next_state, reward, done, info = env.step(action.cpu().numpy()[0])
    total_numsteps += 1
    episode_reward += reward

    action = torch.Tensor(action.cpu())
    mask = torch.Tensor([not done])
    next_state = torch.Tensor(next_state.cpu())
    reward = torch.Tensor([reward])

    # pdb.set_trace()
    memory.push(state, action, mask, next_state, reward)
    state = next_state

    if len(memory) > args.batch_size:
        for _ in range(args.updates_per_step):
            transitions = memory.sample(args.batch_size)
            batch = Transition(*zip(*transitions))

            value_loss, policy_loss = agent.update_parameters(batch)

            writer.add_scalar('loss/value', value_loss, updates)
            writer.add_scalar('loss/policy', policy_loss, updates)

            updates += 1

Would you please help to solve the problem? Thanks in advance

https://github.com/ikostrikov/pytorch-ddpg-naf/blob/7870655178fe93a4eba5450257323f00ff8a227f/normalized_actions.py#L16

Should this return action, not actions?

The NAF algorithm does not work on Pendulum or any of the PyBullet environments. @ikostrikov Do you have any guesses why that might be the case? Which environments did you experiment with this code on? In case you used different hyperparameters than the default values, could you mention the changes that need to be made to get the NAF algorithm working.

For adaptive noise estimation, need to get some states for the expectation operator and compute the ddpg distance metric between perturbed and non perturbed action but getting samples index "t" was not defined, assuming it is batch size we can now compute it

Hi Ilya,

Thanks for your open-source implementation of DDPG/NAF in pytorch.

We spotted a typo in NAF: the discount factor (and the done mask) should multiply the next_state_values instead of adding up to it.

Cheers, Jacques

Hi @ikostrikov , I appreciate your implementation, and I wonder if you've benchmarked your implementation? If so, can I have some roughly results. Many thanks!

hi ikostrikov, I got this error when running your code

Traceback (most recent call last):
  File "main.py", line 89, in <module>
    agent.update_parameters(batch)
  File "/home/andrewliao11/Work/pytorch-naf/naf.py", line 121, in update_parameters
    param.grad.data.clamp(-1, 1)
AttributeError: 'NoneType' object has no attribute 'data'

the original code is:

for param in self.model.parameters():
            param.grad.data.clamp(-1, 1)

maybe we should modify in into:

torch.nn.utils.clip_grad_norm(self.model.parameters(), 1)

I'm just a newbie to pytorch, not sure if it's right, thx!