这个工程主要是在对A2V网络进行预训练和微调。而在Generate_detection.py文件中直接引入了TIN的检测结果，并且通过 sHO = (((sH + sO) * ssp + sP + sA + sL) * score_I[im_index[keys[obj_index]], x:y]) * hod 这条语句融合TIN和A2V的结果。针对此我有三点疑惑：

1： 这样做需要TIN和A2V Net的联合训练吗？还是分别独立训练之后，直接在这里进行score的融合就可以了？

2： 我将sHO融合式子中的sP + sA + sL去掉之后（只用TIN检测结果），发现在HICO上的性能依然很好，和TIN+PaStaNet的效果只差1两个百分点。这是不是说明A2V Net并没有为检测结果提供很大的贡献呢？ ('Default: ', 0.22116964775393347) ('Default rare: ', 0.20193286258301021) ('Default non-rare: ', 0.22691570046732615) ('Known object: ', 0.24059134635026103) ('Known object, rare: ', 0.2219354706179083) ('Known object, non-rare: ', 0.24616388065992487)

3：Generate_detection.py文件中，对score_P,score_A,score_L都先除以各自的fac，再通过sigmoid函数转化为概率。score_P,score_A,score_L这两个在A2V中取出的时候已经是概率了，除以fac的意义是什么？为什么要再次通过Sigmoid函数？ score_P[obj_index][:, hoi_index - x] /= P_fac[hoi_index] score_A[obj_index][:, hoi_index - x] /= A_fac[hoi_index] score_L[obj_index][:, hoi_index - x] /= L_fac[hoi_index]

sP = torch.sigmoid(torch.from_numpy(score_P[obj_index])).cpu().numpy() * P_weight sA = torch.sigmoid(torch.from_numpy(score_A[obj_index])).cpu().numpy() * A_weight sL = torch.sigmoid(torch.from_numpy(score_L[obj_index])).cpu().numpy() * L_weight

期待您的回复！

Hi,

It seems that this code does not include the part of human part or object detection. I don't understand why iCAN is used as the object detection? iCAN seems to use Faster-RCNN for object detection.

Is there any way to input an image and output the corresponding instance-level human interaction behavior as a demo?

Thanks, Tairan Chen

I noticed the Generate_detection.py that there are P_fac, A_fac, L_fac for factorize score_P, score_A and score_L. Also, there is sigmoid function for calculating score_P, score_A and score_L. But in image-level output, there is no sigmoid function and factorization. Do I need them as well?

When I execute python scripts/train.py --pasta-mode linear in docker container, language_model.log is generated. And I checked this file，there are some problems:

I0916 11:37:38.907941   553 caffe.cpp:217] Using GPUs 0
I0916 11:37:38.949647   553 caffe.cpp:222] GPU 0: GeForce GTX 1080 Ti
I0916 11:37:39.380765   553 solver.cpp:51] Initializing solver from parameters: 
test_iter: 2000
test_interval: 1000000
base_lr: 1e-05
display: 20
max_iter: 600000
lr_policy: "cosine"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 30000
snapshot: 10000
snapshot_prefix: "snaps/language-model"
device_id: 0
net: "models/Language-Models/train_language-model.prototxt"
train_state {
  level: 0
  stage: ""
}
test_initialization: false
average_loss: 100
iter_size: 16
snapshot_format: HDF5
train_size: 38116
te: 2
multfactor: 2
tt: 2.05884722928
tenext: 2
I0916 11:37:39.381000   553 solver.cpp:94] Creating training net from net file: models/Language-Models/train_language-model.prototxt
[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 3124:7: Message type "caffe.LayerParameter" has no field named "layer".
F0916 11:37:39.382678   553 upgrade_proto.cpp:79] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: models/Language-Models/train_language-model.prototxt
*** Check failure stack trace: ***
    @     0x7feae92775cd  google::LogMessage::Fail()
    @     0x7feae9279433  google::LogMessage::SendToLog()
    @     0x7feae927715b  google::LogMessage::Flush()
    @     0x7feae9279e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7feae9b77ac1  caffe::ReadNetParamsFromTextFileOrDie()
    @     0x7feae996e6eb  caffe::Solver<>::InitTrainNet()
    @     0x7feae996f9c7  caffe::Solver<>::Init()
    @     0x7feae996fd6a  caffe::Solver<>::Solver()
    @     0x7feae995a6c3  caffe::Creator_SGDSolver<>()
    @           0x40afb9  train()
    @           0x4077c8  main
    @     0x7feae7a0d830  __libc_start_main
    @           0x408099  _start
    @              (nil)  (unknown)

I googled this problem, it seems to be the reason for the different version of caffe. So I want to ask what is the version of caffe for this project. My caffe version is:

>>> caffe.__version__
'1.0.0-rc3'

Thanks for your released code. However, I faces a strage problem that the HAKE-Action seems do not need to train. I run the code using the command:

python tools/Train_pasta_HICO_DET.py --data 0 --init_weight 1 --train_module 2 --num_iteration 11 --model test_pastanet

Then, I test the model 'test_pastanet/HOI_iter_10.ckpt'.

Unbelievably! The result is:

Default:  0.2194696378436301                                                                                                                                                                                       
Default rare:  0.20412592656443512                                                                                                                                                                                 
Default non-rare:  0.22405282432962342                                                                                                           
Known object:  0.2382635240705128                                                                                                               
Known object, rare:  0.22197228395195176                                                                                                                                                                           
Known object, non-rare:  0.24312973865138168

Have you ever tested your code like this? This result is ....... I'm trying to test snapshot 1. I do not change the code.

Does iCAN detection result contains human-object relation? then you needn't estimate which object one person interacts with, Only need to predict which part interacts with this object?

iamge-level中的evaluate用matlab部分中用的测试文件是哪一个。我测试得到的最后结果是linear-final.csv或者fused.csv，但是我看了matlab中的代码的标注label格式是1,0,-1，（分别代表什么意思这里我也不懂），但是linear-final.csv或者fused.csv的格式是一个600维的数组，看起来是激活值，并不是1,0，-1这样的格式，那么最后的结果我怎么得到1,0,-1呢

I have been able to run PaStaNet to generate bboxes, hdet, keys, odet, scores_A, scores_L, and scores_P on custom images, but in the Generate_detection.py file there are a bunch of TIN scores used to generate the detections:

score_H = pickle.load(open('TIN/score_H.pkl', 'rb')) score_O = pickle.load(open('TIN/score_O.pkl', 'rb')) score_sp = pickle.load(open('TIN/score_sp.pkl', 'rb')) hdet = pickle.load(open('TIN/hdet.pkl', 'rb')) odet = pickle.load(open('TIN/odet.pkl', 'rb')) keys = pickle.load(open('TIN/keys.pkl', 'rb')) pos = pickle.load(open('TIN/pos.pkl', 'rb')) neg = pickle.load(open('TIN/neg.pkl', 'rb')) bboxes = pickle.load(open('TIN/bboxes.pkl', 'rb'))

There is also a score_I pulled from the hico_caffe600.h5 file, a key_mapping.pkl, and the generation args file.

I am hoping y'all can release a description of how to generate these files / the formatting, or release the code used to generate these from TIN results / any other necessary sources. I already have TIN running but am unsure of how to format the results for Generate_detection.py. Thank you!

Thanks for your great job on both HOI recognition/detection! I downloaded your code and tried to reproduce your results in the paper, but in the 75L~77L of -Results/Generate_detection.py I felt confused about the details:

sP = torch.sigmoid(torch.from_numpy(score_P[obj_index]).cuda()).cpu().numpy() * P_weight
sA = torch.sigmoid(torch.from_numpy(score_A[obj_index]).cuda()).cpu().numpy() * A_weight
sL = torch.sigmoid(torch.from_numpy(score_L[obj_index]).cuda()).cpu().numpy() * L_weight

As shown, in your code sP, sA, sL are multiplied with P_weight, A_weight and L_weight respectively. The three weights are defined in 42L:

h_fac, o_fac, sp_fac, P_fac, A_fac, L_fac, hthresh, othresh, athresh, bthresh, P_weight, A_weight, L_weight = pickle.load(open('generation_args.pkl', 'rb'))

I carefully checked the content of generation_args.pkl and found that only P_weight has the value of 1.0, while A_weight and L_weight are both 1e-5. And after I removed sA and sL from the final score, the mAP didn't decrease, which means both Attention branch and Language branch will not contribute to the final results with the weight 1e-5. Is 1e-5 the best weight that you found? Or did I make anything wrong? The problem is important since the reproduction of your ablation study strongly relies on this part.

When I run the testing command of HICO-DET in Instance-level-detection, I check the Test_all_part.pkl file carefully. I found that all part boxes in this file are all the same. There must be something wrong when running the PaStaNet. How does this happen?

Here is the screenshot.:

fused.py: for ii in range(1, len(sys.argv)): print sys.argv[ii] a1 = np.loadtxt(sys.argv[ii], delimiter = ',', dtype = str,usecols=range(600)) arr1 = a1.astype(np.float) for i in range(arr1.shape[0]): for j in range(arr1.shape[1]): if ii == 3: global_array[i][j] += arr1[i][j] * 2

why only ii == 3 here? it seems it only uses pairwise.csv result.