Good day, thank you for open sourcing your code. I am very excited to experiment with it.

I ran the following:

python3 src/main.py --config=offpg_smac --env-config=sc2 with env_args.map_name=3s_vs_3z

The code seemed to start but then hangs indefinitely. I would really appreciate if you could help me get the code running.

This is the only output I get in the terminal:

INFO - pymarl - Started run with ID "2"
INFO - my_main - Experiment Parameters:
INFO - my_main - 

{   'action_selector': 'multinomial',
    'agent': 'rnn',
    'agent_output_type': 'pi_logits',
    'batch_size': 16,
    'batch_size_run': 10,
    'buffer_cpu_only': True,
    'buffer_size': 32,
    'checkpoint_path': '',
    'critic_baseline_fn': 'coma',
    'critic_lr': 0.0001,
    'critic_q_fn': 'coma',
    'critic_train_mode': 'seq',
    'critic_train_reps': 1,
    'env': 'sc2',
    'env_args': {   'continuing_episode': False,
                    'debug': False,
                    'difficulty': '7',
                    'game_version': None,
                    'map_name': '3s_vs_3z',
                    'move_amount': 2,
                    'obs_all_health': True,
                    'obs_instead_of_state': False,
                    'obs_last_action': False,
                    'obs_own_health': True,
                    'obs_pathing_grid': False,
                    'obs_terrain_height': False,
                    'obs_timestep_number': False,
                    'replay_dir': '',
                    'replay_prefix': '',
                    'reward_death_value': 10,
                    'reward_defeat': 0,
                    'reward_negative_scale': 0.5,
                    'reward_only_positive': True,
                    'reward_scale': True,
                    'reward_scale_rate': 20,
                    'reward_sparse': False,
                    'reward_win': 200,
                    'seed': 972368347,
                    'state_last_action': False,
                    'state_timestep_number': False,
                    'step_mul': 8},
    'epsilon_anneal_time': 500000,
    'epsilon_finish': 0.05,
    'epsilon_start': 0.5,
    'evaluate': False,
    'gamma': 0.99,
    'grad_norm_clip': 20,
    'label': 'default_label',
    'learner': 'offpg_learner',
    'learner_log_interval': 20000,
    'load_step': 0,
    'local_results_path': 'results',
    'log_interval': 20000,
    'lr': 0.0005,
    'mac': 'basic_mac',
    'mask_before_softmax': False,
    'mixing_embed_dim': 32,
    'name': 'offpg_smac',
    'obs_agent_id': True,
    'obs_last_action': True,
    'off_batch_size': 32,
    'off_buffer_size': 70000,
    'optim_alpha': 0.99,
    'optim_eps': 1e-05,
    'q_nstep': 0,
    'repeat_id': 1,
    'rnn_hidden_dim': 64,
    'runner': 'parallel',
    'runner_log_interval': 20000,
    'save_model': True,
    'save_model_interval': 1000000,
    'save_replay': False,
    'seed': 972368347,
    'step': 5,
    't_max': 10050000,
    'target_update_interval': 600,
    'tb_lambda': 0.93,
    'td_lambda': 0.8,
    'test_greedy': False,
    'test_interval': 20000,
    'test_nepisode': 20,
    'use_cuda': True,
    'use_tensorboard': True}

When I interrupt the terminal with ctrl+c I get the following:

Process Process-10:
Process Process-3:
Process Process-5:
Process Process-7:
Process Process-2:
Process Process-8:
Process Process-9:
Process Process-1:
Process Process-6:
Process Process-4:
Traceback (most recent call last):
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/claude/Documents/ICQ/ICQ-MA/src/runners/parallel_runner.py", line 228, in env_worker
    cmd, data = remote.recv()
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

Traceback (most recent call last):
  File "src/main.py", line 96, in <module>
    ex.run_commandline(params)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/sacred/experiment.py", line 250, in run_commandline
    return self.run(cmd_name, config_updates, named_configs, {}, args)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/sacred/experiment.py", line 199, in run
    run()
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/sacred/run.py", line 229, in __call__
    self.result = self.main_function(*args)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/sacred/config/captured_function.py", line 48, in captured_function
    result = wrapped(*args, **kwargs)
  File "src/main.py", line 34, in my_main
    run(_run, config, _log)
  File "/home/claude/Documents/ICQ/ICQ-MA/src/run.py", line 48, in run
    run_sequential(args=args, logger=logger)
  File "/home/claude/Documents/ICQ/ICQ-MA/src/run.py", line 101, in run_sequential
    learner.cuda()
  File "/home/claude/Documents/ICQ/ICQ-MA/src/learners/offpg_learner.py", line 195, in cuda
    self.mac.cuda()
  File "/home/claude/Documents/ICQ/ICQ-MA/src/controllers/basic_controller.py", line 72, in cuda
    self.agent.cuda()
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 265, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 199, in _apply
    param.data = fn(param.data)
  File "/home/claude/miniconda3/envs/icq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 265, in <lambda>
    return self._apply(lambda t: t.cuda(device))

您好，我在拜读您的代码时，遇到一个问题： 在ICA-antmaze_mu中ica.py文件中test_bc()函数中，tensor_o仅包含observation，而在使用self.vae.decode(tensor_o)时，维度应该是self.obs_dim+2，所以导致以下问题： RuntimeError: size mismatch, m1: [1 x 45], m2: [47 x 750] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:961

请问该怎么解决？

The definition of the advantage function is A(s,a) = Q(s,a) - V(s). It seems that V(s) is not explicitly calculated in the code (here), as

# V(s), why, isn't this the Q of at the current action?
pi, logp_pi = self.ac.pi(o)
q1_pi = self.ac.q1(o, pi)
q2_pi = self.ac.q2(o, pi)
v_pi = torch.min(q1_pi, q2_pi)

# Q(s,a)
q1_old_actions = self.ac.q1(o, data['act'])
q2_old_actions = self.ac.q2(o, data['act'])
q_old_actions = torch.min(q1_old_actions, q2_old_actions)

# A(s,a)
adv_pi = q_old_actions - v_pi

Looking forward to your reply

Hi，

Thanks for providing the code.

When I run python3 src/main.py --config=offpg_smac --env-config=sc2 with env_args.map_name=3s_vs_3z I got a error about the tensor in different devices, when I used torch==1.1.0 or torch==1.10.

I notice the off_batch （in run.py line 155) is on the CPU and all the Net is on the GPU. Could you check this problem?

Thanks!

Hi,

Thanks for providing the code.

Other configurations (e.g., --config=qmix_smac) cannot run. I was wondering if you plan to fix these other configurations?

Dear author, thanks for making the code available.

I have two questions regarding ICQ_softmax, where the weight is approximated by the softmax over the minibatch:

1. Why len(weights), e.g., this line, is needed to scale the softmax distribution?
2. Why the softmax is performed wrt the TD error in this line, instead of wrt the Q value suggested in the paper?

Thanks!