Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...

Hello, when I am trying to run a tutorial script, e.g. the your_first_program example, I always encounter this AttributeError during the imports:

AttributeError: module 'torch.distributed.rpc' has no attribute 'rpc_sync'

However, I fulfill the listed requirements. Is there anything I am missing or have can I solve this?

Hello,

Does machin support Multi Discrete Action Spaces? (two different actions in the same time step) I've looked through the documentation but cannot find anything related to that

João

Hey, I'm trying to implement hybrid action space with A2C agent, maybe you have some advice. My expected output are two actions: one discrete, one continuous. Network predicts 3 things:

• logits for discrete action (sampling from Categorical dist)
• mean for distribution of continuous action
• std for same (sampling from Normal)

Net outputs sum of log probabilities of actions from both distributions (same for entropy). Network successfully learns the mean and std but the weight for the logits layers are not updated at all. What can be the reason?

File "D:\Anaconda3\envs\universe\lib\site-packages\torch\distributed\rendezvous.py", line 9, in from . import FileStore, TCPStore ImportError: cannot import name 'FileStore'

After installing machin, run PPO. py，than report an error, and try others to report the same error.As follow:

D:\Anaconda3\envs\universe\python.exe F:/machin/machin-master/examples/framework_examples/dqn.py Traceback (most recent call last): File "F:/machin/machin-master/examples/framework_examples/dqn.py", line 1, in from machin.frame.algorithms import DQN File "D:\Anaconda3\envs\universe\lib\site-packages\machin_init_.py", line 1, in from . import env, frame, model, parallel, utils File "D:\Anaconda3\envs\universe\lib\site-packages\machin\env_init_.py", line 1, in from . import utils, wrappers File "D:\Anaconda3\envs\universe\lib\site-packages\machin\env\wrappers_init_.py", line 1, in from . import base, openai_gym File "D:\Anaconda3\envs\universe\lib\site-packages\machin\env\wrappers\openai_gym.py", line 8, in from machin.parallel.exception import ExceptionWithTraceback File "D:\Anaconda3\envs\universe\lib\site-packages\machin\parallel_init_.py", line 2, in from . import distributed, server, assigner, exception, pickle, thread, pool, queue File "D:\Anaconda3\envs\universe\lib\site-packages\machin\parallel\distributed_init_.py", line 1, in from .world import ( File "D:\Anaconda3\envs\universe\lib\site-packages\machin\parallel\distributed\world.py", line 14, in import torch.distributed.distributed_c10d as dist_c10d File "D:\Anaconda3\envs\universe\lib\site-packages\torch\distributed\distributed_c10d.py", line 10, in from .rendezvous import rendezvous, register_rendezvous_handler # noqa: F401 File "D:\Anaconda3\envs\universe\lib\site-packages\torch\distributed\rendezvous.py", line 9, in from . import FileStore, TCPStore ImportError: cannot import name 'FileStore'

Process finished with exit code 1

Hi,

I was using machin until today, but I think it there was a super recent update and now my code stopped working (only for PPO, DQN still works).

With version 0.4.0 I get the error (when importing "from machin.frame.algorithms import PPO"): ModuleNotFoundError: No module named 'machin.frame.helpers'

And if I install version 0.3.4 I get the error (when updating PPO): AttributeError: 'PPO' object has no attribute 'grad_max'

Joao

Where to alter It's best to alter throughout the entire library, but my specific pain started with ppo.py.

Why to alter

• Custom formatting is less readable than standard black formatting.
• Standard formatting means minimal diffs on any future edits - easier to review

How to alter

1. Run black https://github.com/psf/black on entire repository
2. [OPTIONAL] add this commit to list of ignored commits for git blame as described here https://akrabat.com/ignoring-revisions-with-git-blame/ - I think this can be skipped because so far you are the main author of the library and git history is not that rich.
3. [OPTIONAL] add pre-commit hook to run black on all future commits automatically https://black.readthedocs.io/en/stable/version_control_integration.html

Example: originally, this is how code looks that I needed to debug:

            batch_size, (state, action, advantage) = \
                self.replay_buffer.sample_batch(self.batch_size,
                                                sample_method="random_unique",
                                                concatenate=concatenate_samples,
                                                sample_attrs=[
                                                    "state", "action", "gae"],
                                                additional_concat_attrs=[
                                                    "gae"
                                                ])

• that's a festival of different indentation levels.

Here is how it looks after black reformatting:

            batch_size, (state, target_value) = self.replay_buffer.sample_batch(
                self.batch_size,
                sample_method="random_unique",
                concatenate=concatenate_samples,
                sample_attrs=["state", "value"],
                additional_concat_attrs=["value"],
            )

Hey, i'm using DQN and my q-values are variable lengths sequences as I have different amount of actions for each state (my states have different shapes also). When sampling batches default Buffer concatenating them which leads to a tensor error. But when using update() with concatenate_samples=False it stills doesn't solve the problem as now samples are just lists, and all torch operations fail. Of course, I can pad the sequences, but then argmax() can return one of the padded indexes, as it's not possible to pass original lengths of each sample in batch in update() function. Is there any way to solve the problem right now, or it yet to be implemented?

Hi, this project is really awesome and the codes is well structured!

Is your feature request related to a problem? Please describe. I have run some codes in examples/tutorials, but can not find some about MARL algorithms such as maddpg.

Describe the solution you'd like Since you have already implemented the maddpg.py and test_maddpg.py, I am wondering could you implement a tutorial for maddpg.py too? (It would be better to implement more MARL algorithms, such as COMA, QMIX, VDN)

Is your feature request related to a problem? Please describe.

1. Transition batch size

I read the tutorial about RL in spiningup. I found that for on-policy RL, they have a step to collect a set of trajectories in their pseudocode. However, in your documentation Data flow in Machin, you point out that

Currently, the constructor of the default transition implementation Transition requires batch size to be 1

and

Buffer.store_episode(): If you pass in a dict type transition object, it will be automatically converted to Transition

In your PPO examples/ppo.py it seems that you only save one trajectory per iteration and update it. What should I do if I want to save a set of trajectories? Will such a change affect the update() part?

2. Multiple trajectories with one reward

In general scenarios, one trajectory (episode) will have one total reward. However, I encountered a case where multiple trajectories with only one total reward. For example:

(trajectory1: [s,a,0,s,a,0,...,s,a], trajectory2 [s,a,0,s,a,0,...,s,a], trajectory3[s,a,0,s,a,0,...,s,a] ) ---> final reward Imagine that many football players are playing the same game. They receive the same reward only when goal, Imagine that generating a batch of noise sequences to attack the neural network will get only one reward which indicates the degraded performance of the neural network.

I give two solutions to this problem in the next section, but I am not sure which one is better. Could you give me some advice?

Describe the solution you'd like For feature 1, it may realized as below(I am not sure whether this affect the update() part):

while episode< max_episodes:
    for i in range(sub_episode_size) # add a loop here
        episode += 1
        tmp_observations = []
        while not terminal and step <= max_steps:
               # ....
               tmp_observations.append(...) # store transition
         ppo.store_episode(tmp_observations)
     ppo.update()
     # clean buffer

For feature 2, it may have two solutions. One is to assign the final reward to every trajectory:

from collection import defaultdict
while episode< max_episodes:
    batch_trajectory = defaultdict[List]
    episode += batch_size
    while not terminal and step <= max_steps:
           # using batch state to generate batch action
           reward = env.step(batch_action)
           for i in range(batch_size):
               batch_trajectory[i] += batch_state[i] + batch_action[i] + reward # assign the same reward to each trajectory

     for i in range(batch_size):
         ppo.store_episode(batch_trajectory[i])
     ppo.update()
     # clean buffer

Describe alternatives you've considered

For feature 2, another solution may resort to the multi-agent RL. Each agent manages one trajectory and they receive the same reward from the environment. I found that Machin has a multi-agent algorithm implementation called MADDPG. From spiningup I found that this algorithm is only for continuous action space. Is there any plan to implement other multi-agent RL algorithms such as multi-agent PPO for discrete action space?

Additional context

Hi! Thanks for your excellent work! I tried several RL frameworks based on PyTorch, machin is one of the few libraries that discusses hybrid action space.

I'm diving into a complex environment which is a hierarchical action space problem. I hope you could give me some advice!

To explain the meaning of hierarchical action space more clearly, here is an example in the paper Generalising Discrete Action Spaces with Conditional Action Trees. Figure2 in the paper shows that the actions are decomposed as an action tree. One should first select the first level actions, then select the second level actions. The action space of the first level is 3 and the action space of the second level depends on the first level.

I try to give one possible solution to solve this:

1. change the transition part:

transition = { "state": {"some_state": old_state} , ...  } # old
transition = {"state": {"some_state": old_state, "valid_actions": valid_action_set } , ... }

here, the "valid_actions" contains all of the possible second-level actions based on the first-level action.

2. change the agent sampling(explore) flow:

state = env.reset() # state have key words "some_state" and "valid_actions", here, for initial, the valid_actions are pre-defined manually. For example, we choose the first-level action 'use', and then the valid_actions will be 'food'
while not done:
    action2 = agent.act2(state) # choose action2 from valid action set. Here, action2 is 'food'
    action1 = agent.act1(state) # choose action1 which denotes the first level action space. Here, suppose action1 is 'move'.
    env.first_level(action1) # tell env, the valid action set in next step is 'up, down, right, left' under the 'move' branch
    next_state, reward, done = env.step(action2) # next_state have key words "some_state" and "valid_actions", the valid_actions have 'up, down, right, left'.

3. change the actor network agent.act2 to:

class act2(nn.Module):
  def forward(self, state = {'some_state': <...> , 'valid_actions': <...>}  ):
    state, second_level_actions = state.state, state.valid_actions
    ....  # calculate the similarity between state and valid actions, and output logits
    return (...), None

Do you think this solution is reasonable? Is there any better way to support such a conditional hierarchical action space?

Hi,

I guess the entropy in A2C is wrong:

if new_action_entropy is not None:
    act_policy_loss += self.entropy_weight * new_action_entropy.mean()

instead it should be:

if new_action_entropy is not None:
    act_policy_loss -= self.entropy_weight * new_action_entropy.mean()

Best,

Lorenzo

It seems that your code produce error if the len of your trajectory < 2 ( len(tmp_observations) < 2). I tested this on PPO I don't know if this happens with all algorithms.

The error:

ValueError: The parameter probs has invalid values