See #59

Todo Items:

• [ ] logging other types of metrics such as WinLossRewardFunction
• [ ] better docs
• [ ] add test cases (at least make sure it runs)

On my local checkout, I can't seem to be able to run pytest because of an import error.

⋊> ~/s/enn-incubator on main ◦ poetry run pytest                                                                                                                    (base) 21:50:44
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/clemens/src/enn-incubator
collected 3 items / 2 errors / 1 selected

===================================================================================== ERRORS ======================================================================================
____________________________________________________________ ERROR collecting enn_ppo/enn_ppo/test_sample_recorder.py _____________________________________________________________
ImportError while importing test module '/home/clemens/src/enn-incubator/enn_ppo/enn_ppo/test_sample_recorder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
enn_ppo/enn_ppo/test_sample_recorder.py:3: in <module>
    from enn_ppo.sample_recorder import SampleRecorder, Sample, Trace
enn_ppo/enn_ppo/sample_recorder.py:4: in <module>
    from entity_gym.environment import ActionSpace, ObsSpace
entity_gym/entity_gym/environment.py:6: in <module>
    from ragged_buffer import RaggedBufferF32, RaggedBufferI64
../../.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.7/lib/python3.7/site-packages/ragged_buffer/__init__.py:1: in <module>
    from typing import Any, Generic, Protocol, Type, TypeVar, Union, cast, overload
E   ImportError: cannot import name 'Protocol' from 'typing' (/home/clemens/miniconda3/lib/python3.7/typing.py)
________________________________________________________________ ERROR collecting enn_ppo/enn_ppo/test_training.py ________________________________________________________________
ImportError while importing test module '/home/clemens/src/enn-incubator/enn_ppo/enn_ppo/test_training.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
enn_ppo/enn_ppo/test_training.py:1: in <module>
    from enn_ppo import train
enn_ppo/enn_ppo/train.py:21: in <module>
    from entity_gym.environment import (
entity_gym/entity_gym/environment.py:6: in <module>
    from ragged_buffer import RaggedBufferF32, RaggedBufferI64
../../.cache/pypoetry/virtualenvs/incubator-Qe8CM38i-py3.7/lib/python3.7/site-packages/ragged_buffer/__init__.py:1: in <module>
    from typing import Any, Generic, Protocol, Type, TypeVar, Union, cast, overload
E   ImportError: cannot import name 'Protocol' from 'typing' (/home/clemens/miniconda3/lib/python3.7/typing.py)
============================================================================= short test summary info =============================================================================
ERROR enn_ppo/enn_ppo/test_sample_recorder.py
ERROR enn_ppo/enn_ppo/test_training.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================ 2 errors in 1.16s ================================================================================

Rethink action mask API

The current API for specifying actors is very confusing and poorly documented (I myself was very confused by it after not looking at it for a month after implementing it and forgetting how it works 😆). I think we can come up with something better.

Running example

Suppose we have an environment with two entity types, Worker and Resource, both with features x and y, a harvest action, and a move action. The observation and action spaces might look like this:

obs_space = {
  "resource": Entity(["x", "y"]),
  "worker": Entity(["x", "y"]),
}
action_space = {
  "harvest": CategoricalAction(["harvest", "don't harvest"]),
  "move": CategoricalAction(["up", "down", "left", "right"]),
}

Suppose that on a particular timestep, we have two resource entities r1 and r2, and three worker entities w1, w2, and w3. Only w1 and w3 can perform the harvest action on this timestep. All resources and workers can perform the move action, but w2 can't move down or left.

Observations

Observations are currently defined like this:

https://github.com/entity-neural-network/incubator/blob/c887f4a2f6601c67e8feb03ba9b16f27f74a223f/entity_gym/entity_gym/environment/environment.py#L82-L92

entities

The observation consists of a dictionary mapping each entity type to a numpy array with the features of all entities of that type:

entities = {
  "resource": [
    [10, 4], # r1
    [11, 6], # r2
  ],
  "worker": [
    [10, 4], # w1
    [0, 0],  # w2
    [11, 6], # w3
  ],
}

This seems fine, or at least I can't think of any obvious simplification or more ergonomic API.

ids

The ids are meant to allow the environment to designate an opaque identifier for each entity, which then makes it easy to later lookup that entity when executing actions chosen by the policy. For example, we could use the following list of strings to identify our entities:

ids = ["r1", "r2", "w1", "w2", "w3"],

In practice, this mostly adds runtime overhead and API complexity, and it's unclear whether it is actually particularly useful for most environments. One issue is also that the correspondence between the ids and entities is not obvious since it relies on the implicit ordering of entity types defined by the observation space.

action_masks

The environment must specify which entities can perform each action. This is done by returning an ActionMask for each action that specifies an actors list that specifies the indices of the entities that can perform the action:

action_masks = {
  "harvest": DenseCategoricalActionMask(actors=[2, 4]),
  "move": DenseCategoricalActionMask(
    actors=[0, 1, 2, 3, 4],
    mask=[
      [True, True, True, True],
      [True, True, True, True],
      [True, True, True, True],
      [True, False, False, True],
      [True, True, True, True],
    ],
  ),
}

The actors field is very unintuitive. In contrast to the observation, it's a single list rather than a list per entity, and it relies on the implicit ordering of entity types given by the observation space dictionary. The mask is also inconsistent with the ids, since it uses indices rather than the ids to specify actors.

action

Once we have sampled actions from the policy, we call the step method again and supply a action: Mapping[str, Action] with the choices made by the policy for each action. For categorical actions, this is represented by a actions: List[Tuple[EntityID, int]] which pairs each actor with the index of the chosen action:

actions = {
  "move": CategoricalAction([
    "r1": 0,
    "r2": 0,
    "w1": 2,
    "w2", 0,
    "w3", 3
  ]),
  "harvest": CategoricalAction([
    "w1": 0,
    "w3": 3,
  ]),
}

Solutions

The current API is pretty bad, not sure what the best solution is but here are some ideas:

Proposal 1: Group action_masks and ids by entity type

The indices would now be per entity, corresponding to the features in entities. This would make it a little more obvious what's going on. And in the common case of all entities of some type being able to execute an action, we wouldn't need to specify the actors field at all.

action_masks = {
  "worker": {
    "harvest": DenseCategoricalActionMask(actors=[0, 2]),
    "move": DenseCategoricalActionMask(
      mask=[
        [True, True, True, True],
        [True, False, False, True],
        [True, True, True, True],
      ],
    ),
  },
  "resource": {
    "move": DenseCategoricalActionMask(
      mask=[
        [True, True, True, True],
        [True, True, True, True],
      ],
    )
  },
}

ids = {
  "worker": ["w1", "w2", "w3"],
  "resource": ["r1", "r2"],
}

Proposal 2: Use EntityIDs to specify actors

Instead of indices, use EntiyIDs for actors:

action_masks = {
  "harvest": DenseCategoricalActionMask(actors=["w1", "w2"]),
  "move": DenseCategoricalActionMask(
    actors=["r1", "r2", "w1", "w2", "w3"],
    mask=[
      [True, True, True, True],
      [True, True, True, True],
      [True, True, True, True],
      [True, False, False, True],
      [True, True, True, True],
    ],
  ),
}

Proposal 3: Just get rid of ids

The EntityIDs haven't proved particularly useful so far. We could just get rid of the ids field and remove the EntityID from the actions passed to act. Something similar to the old interface that works with environment-supplied ids rather than indices could still be achieved with an env wrapper or helper methods.

actions = {
  "move": CategoricalAction([0, 0, 2, 0, 3]),
  "harvest": CategoricalAction([0, 3]),
}

If combined with (1):

actions = {
  "worker": {
    "move": CategoricalAction([2, 0, 3]),
    "harvest": CategoricalAction([0, 3]),
   },
  "resource": {
    "move": CategoricalAction([0, 0]),
  },
}

Fix the ids field on Observation/BatchObs to allow the EntityIDs to be arbitrary objects, and add a new test that makes sure we don't break that property.

I'm Python 3.8.12 which should be supported according to the pyproject.toml. I'm also using Poetry 1.1.7 with poetry-core==1.0.4, because of this related issue" https://stackoverflow.com/questions/69836936/poetry-attributeerror-link-object-has-no-attribute-name

However, I get weird errors concerning the concurrent installation of torch and torch-scatter. It looks like torch-scatter is trying to be installed before torch. See log.txt for full stack trace. Tried installing torch first and then re-installing torch-scatter but got a different error: log3.txt

Help appreciated

#115 but with a fresh start.

Currently has few environments. ViZDoom, being a "visual" learning env, is not the best fit for entity-gym but the agent should still learn something.

TODO:

• [x] Add poetry/etc stuff
• [x] Add quick readme docs
• [x] Get positive results in all envs (try out relpos encoding, study params, double-check env)

Single runs with different envs:

• Defend the Center: https://wandb.ai/entity-neural-network/enn-ppo/runs/2wabrli9
• Health gathering: https://wandb.ai/entity-neural-network/enn-ppo/runs/37rfxti0
• Health-gathering supreme: https://wandb.ai/entity-neural-network/enn-ppo/runs/ok8sx1w1 (this one probably fails to learn because walls are not included as part of entities etc.)

Hello, reporting a bug here. The current implementation seems to assume sequential entity IDs.

In #123, I have the following observations:

where the entities have IDs [4, 5, 6, 7, 8, 9, 10, 11] and the entities have mask [ 8 10]. I can use the main.py to interact with the games to harvest resources.

However, there is an issue during training. Basically, the mask [ 8 10] is used to select actor_embeds, however, the idx 8 and 10 are out of bounds for the x.data, which only has embeddings corresponding to 8 objects.

Add an option similar to --translate that configures absolute positional encoding for environments with discrete positions with one embedding for each grid point.

Taking inspiration from stable-baselines' vector implementation, but with some additional optimizations... the ParallelListEnv takes a num_processes and num_envs parameter. and will split the envs into equal proportions on each process.

This is better than having a process per-environment as it reduces the latency/observations ratio. (having 10 environments per process, means we can get 10 observations with only 1 send/receive from the process).

Setting num_processes==num_envs is equivalent to SubProcVecEnv. This has a overhead of 1 observation per send/receive.

Setting num_processes==1 means you have no parallelization.

Setting num_processes==8 and num_envs=64 will give you 8 envs per-process. This has a reduced overhead of 8 observations per send/recieve.

With Griddly I can achieve about 6K SPS, up from previous 3K SPS, when the envs are split across 8 processes.

This PR adapts the entire project to use poetry as the package manager. See the following demo.

One issue is when doing poetry install torch-scatter throws an error. I think this is because torch-scatter never explicitly says it depends on torch so when doing parallelized installation it throws the error. To fix it, simply run poetry install again.

We'll want some generic way of normalizing input features. The best way to do this might be as an environment wrapper. There are still various details to figure out how exactly that will look like, and how the normalization will be incorporated into network architectures in a way that can be checkpointed.

This issue tracks python 3.7.1+ support

• [x] https://github.com/cswinter/hyperstate/pull/2
• [x] https://github.com/entity-neural-network/entity-gym/pull/9
• [x] https://github.com/entity-neural-network/rogue-net/pull/9
• [ ] https://github.com/entity-neural-network/enn-trainer/pull/16

This PR adds submodules linking the separate repositories. We still need a way to work with all other repos at the same time in the same virtual poetry environment.