Currently, we are unable to do the following:

from salina import Workspace
ws = Workspace()
batch_size = 5
ws.set("obs", 0, torch.zeros(batch_size, 3))
ws.set("obs", 1, torch.zeros(batch_size, 5))

due to https://github.com/facebookresearch/salina/blob/10d09bb80f78e05ddd7de58e9e24ff0f302877fb/salina/workspace.py#L45

Since tensors from sequential timesteps are stored as lists, I suspect variable-sized features should be possible. This would be immensely useful in multiagent systems as well as recurrent models (e.g. building/expanding a map as the agent explores).

This is a really interesting framework, I'd love to move some of our code to it. The workspace abstraction should greatly simplify our multiagent and recurrent models. I am a bit worried about the readability/correctness though -- are there any plans to write up proper documentation (e.g. https://readthedocs.org) and unit tests?

cc @ludc, not a lot of time but this should work. TODOs:

• [x] handle dimensions not being powers of two (max episodes -> 1024 vs 1000). Either change the episodes or pad
• [ ] (be smarter in how to choose in between sparse and blocksparse. If the time span is small enough, use sparse, else blocksparse)

let me know what you think. Requires a TensorCore enabled GPU, should be faster than sparse for some regimes and fully fp16 aware

Fixes #16.

There were a few more instances than I expected, but I think I got everything. I fixed a couple of other little issues at the same time which would have thrown errors.

I also noticed that some of the files weren't black-formatted even though the README indicated to run isort + black through pre-commit. I ran pre-commit run --all-files to fix these formatting issues. There are several unused imports (and some unused variables) too, but I didn't want to go looking for all of these, so I left the ones I saw.

I just cloned the repo and tried to run some of the provided examples. Unfortunately I couldn't run them as the requirements are not clear. I tried python 3.7 - 3.9 to see if it was an issue of the python version. For each I did the following:

• create clean anaconda environments (e.g. conda create -n salina-test python=3.7)
• install torch and torchvision (conda install pytorch torchvision torchaudio cpuonly -c pytorch)
• pip install -r requirements.txt
• python setup.py install

For individual examples I first get ModuleNotFoundErrors. E.g., when running salina_examples/rl/a2c/mono_cpu/main.py I encounter the following errors: ModuleNotFoundError: No module named 'graphviz' and ModuleNotFoundError: No module named 'pandas' After fixing those I get ModuleNotFoundError: No module named 'salina_examples.rl.a2c'

Similarly, when trying to run salina_examples/rl/ppo_continuous/ppo.py I first encounter ModuleNotFoundError: No module named 'cv2' and after fixing it I get the following error:

Traceback (most recent call last):
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/gym/envs/registration.py", line 158, in spec
    return self.env_specs[id]
KeyError: 'Pendulum-v0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "salina_examples/rl/ppo_continuous/ppo.py", line 180, in main
    action_agent = instantiate_class(cfg.action_agent)
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/salina-1.0-py3.8.egg/salina/__init__.py", line 27, in instantiate_class
    return c(**d)
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/salina-1.0-py3.8.egg/salina_examples/rl/ppo_continuous/agents.py", line 92, in __init__
    env = instantiate_class(args["env"])
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/salina-1.0-py3.8.egg/salina/__init__.py", line 27, in instantiate_class
    return c(**d)
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/salina-1.0-py3.8.egg/salina_examples/rl/ppo_continuous/agents.py", line 33, in make_gym_env
    e = gym.make(env_args["env_name"])
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/gym/envs/registration.py", line 128, in make
    spec = self.spec(path)
  File "/home/biedenka/anaconda3/envs/salina-test/lib/python3.8/site-packages/gym/envs/registration.py", line 185, in spec
    raise error.DeprecatedEnv(
gym.error.DeprecatedEnv: Env Pendulum-v0 not found (valid versions include ['Pendulum-v1'])

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

which is caused by a recent change in gym. For reference see:

• https://github.com/openai/gym/pull/2423

To me it looks like these are all dependency issues (and for the ModuleNotFoundError: No module named 'salina_examples.rl.a2c' it mightbe fixed by including an init.py in the folder). Could you please state the minimal version number for which salina works as well as the python version (more prominently than in the setup.py)?

Hi,

Thanks for sharing your interesting project! I recently have been interested in the project related to the paper, Learning a Subspace of Policies for Online Adaptation in RL. In the paper, they commented the code is released in this repository, and I tried to run the code, subspace_of_policies. I succeeded in running the training code, train.py, after slightly modifying it. Also, after relatively much modifying the evaluation code, I checked the code was working. However, I think to confirm the contents represented in the paper the evaluation configuration should be given, which has the components such as the torso, thig, shin, foot, gravity, and friction.

I think the test_cfgs is needed. from salina_examples.rl.subspace_of_policies.envs import test_cfgs

Could you share the code?

Best regards,

Hello,

I'm interested with loading and storing recurrent states for training over longer episodes. This is generally called truncated back propagation through time (BPTT). For example, in the following case we break each trajectory into 80-timestep chunks:

    env = AutoResetGymAgent(
        make_cartpole,
        n_envs=2,
    )
    actor = Agents(
        LSTMAgent(hidden_size=32),
        QNetworkAgent(input_handle="state", num_actions=2),
        EpsilonGreedyActorAgent(epsilon=0.02),
    )
    collector = TemporalAgent(Agents(env, actor))
    
    ws = Workspace()
    for epoch in range(10):
        collector(ws, t=0, n_steps=80)

Currently, if an episode is > 80 timesteps, it will receive a recurrent state of zeros. Does Salina provide a way to load the previous recurrent state?

Hey there, i find it somewhat counterintuitive that the framework uses a default reward at t=0 of 0 (see gyma.py line 279 & 292). Note that the gym interface only returns the initial state on reset (https://github.com/openai/gym/blob/103b7633f564a60062a25cc640ed1e189e99ddb7/gym/core.py#L8). Isn't it more common to assume that r_t = R(s_t, a_t) and consequently r_t is the outcome of \pi(s_t)? Currently, r_{t+1} is the outcome of \pi(s_t). In the A2C example this leads to some confusion where reward[1:] is the reward at t and critic[1:] the state value at t+1 (but both use a 1)

 target = reward[1:] + cfg.algorithm.discount_factor * critic[1:].detach() * (1 - done[1:].float())

Best regards

edit: Fig. 13 & Fig. 14 in the ArXiv Paper use set.get(...) , i believe it should be self.get(...) :-)

adding another Transformer-based agent, using xformers under the hood. For masks sparse enough (few enough time slices), this means that the computation will be naturally sparse, saving time and memory

NRemoteAgent's create method has the signature def create(agent, num_processes=0, time_size=None, **extra_args):. Should this be def create(self, agent, num_processes=0, time_size=None, **extra_args):? Right now, it seems the agent is making copies of itself to put in remote processes instead of making copies of an agent that's passed to create.

https://github.com/facebookresearch/salina/blob/f231c77e44e87713d54984fa08ef4b38be47f644/salina/agents/remote.py#L193

Some functions that accept arbitrary keyword arguments (e.g. Agents.__call__) name the keyword arguments args (e.g. __call__(self, workspace, **args)). What would you think of renaming this to __call__(self, workspace, **kwargs)? Even with the ** making it clear these were keyword arguments, I was confused for a bit thinking that these were positional arguments, since it's common to use f(*args, **kwargs) in Python. I'd be happy to submit a PR renaming any **args to **kwargs if you'll accept it; it shouldn't change anything for users since that name can only be used inside the functions. Otherwise, feel free to close this.

Hi,

I've recently been using your library for a reinforcement learning project. More precisely, I'm interested in multi-agent systems that provide multiple rewards for them. In my environment, after the one-step, it provides n_agents x 1 size of rewards. However, in the workspace, it initializes the reward as "torch.tensor([0.0]).float()" at the first time step(t=0). So, at the second time step, it could not set the new reward whose size is "n_agents x 1" because the occupied reward size is torch.Size([1]). I may think there are many cases that have multiple dimensions of reward not only in my environment but also in other environments.

I carefully think it could be more useful if your library supports a flexible reward size.

Best regards,

Hi,

I've been working with the a2c example agent you provide, and haven't found any game other than CartPole-v0 that it can learn well. Is there any other game that it is good at?

Thank you so much.

Best, Karin