The rmsprop statistics are created in the optimizer's step function, but that is only called after the optimizer is copied into the worker processes. Because of this, all the copies will create separate statistics for themselves and putting these into shared memory. Moving the state initialization into __init__ should work. Not sure if doing it there is good practice though, since all the pytorch optimizer implementations do state initialization in the step function.

Take this with a grain of salt, I am new to PyTorch.

Hi, thanks for releasing this project,

I get the following error, when running run_a3c.py.

[2017-02-19 18:18:07,137] Making new env: Breakout-v0
Process Process-1:
Traceback (most recent call last):
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "run_a3c.py", line 38, in train
    p, v = local_policy(Variable(torch.from_numpy(o)).unsqueeze(0))
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 210, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ajay/PythonProjects/pytorch_a3c-master/policy.py", line 61, in forward
    x = F.relu(self.head(x))
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 210, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 54, in forward
    return self._backend.Linear()(input, self.weight, self.bias)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/nn/_functions/linear.py", line 10, in forward
    output.addmm_(0, 1, input, weight.t())
TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.ByteTensor, torch.FloatTensor), but expected one of:
 * (torch.ByteTensor mat1, torch.ByteTensor mat2)
 * (torch.SparseByteTensor mat1, torch.ByteTensor mat2)
 * (int beta, torch.ByteTensor mat1, torch.ByteTensor mat2)
 * (int alpha, torch.ByteTensor mat1, torch.ByteTensor mat2)
 * (int beta, torch.SparseByteTensor mat1, torch.ByteTensor mat2)
 * (int alpha, torch.SparseByteTensor mat1, torch.ByteTensor mat2)
 * (int beta, int alpha, torch.ByteTensor mat1, torch.ByteTensor mat2)
 * (int beta, int alpha, torch.SparseByteTensor mat1, torch.ByteTensor mat2)

Also can you tell me where reinforce() is defined? It's called on line 80 as,

a.reinforce(r - v.data.squeeze())

Thanks for your help

Hi @rarilurelo,

can I ask if you have been able to modify your code to work with continuous actions - eg pendulum or mountain car? I tired to modify @ikostrikov 's implementation, see here

https://discuss.pytorch.org/t/continuous-action-a3c/1033

but could not get it too work? I think @pfre00 has tried too, but he said training was not stable, see here

https://github.com/pfre00/a3c/issues/1

Have you got any advice?

Kind regards,

Ajay

Hi, thanks for this project. I have a question about the optimizer, Is it enough to simply use the original optimizer provided by pytorch? Before opt.step(). if we simply do sth like:

opt = SGD(shared_para, lr=0.01)
for l_p, s_p in zip(local_para, shared_para):
     l_p.grad.data = s_p.grad.data
opt.step()

will this be enough? Thanks!

I am new to pyTorch, just cloned your codes and ran them, but got an error. I hope you to point me to the right direction to fix this issue.

More specifics:

1. used conda env with python 3.6
2. Ran 'run_a3c.py' with the default Breakout-v0 env till the end and ran 'python test_a3c.py --render --monitor --env Breakout-v0'
3. got the below error message -

=== File "test_a3c.py", line 71, in test(policy, args) File "test_a3c.py", line 25, in test p, v = policy(o) ... File "/home/john/anaconda3/envs/th/lib/python3.6/site-packages/torch/nn/functional.py", line 37, in conv2d return f(input, weight, bias) if bias is not None else f(input, weight) RuntimeError: expected a Variable argument, but got numpy.ndarray

Could you tell me what could be the issue(s) here ?

Many thanks,

John