Hi, thank you very much for sharing your training and testing codes. Could you please also share the training data? I have applied for the training data on the website but still get no reply. Thank you very much.

We use CK+ dataset, but when I use test dataset image to test ,the result doesn’t expect,The following is the process of my train. Can you give me some suggestions?

1、Train GANimation

2、The GPU working status

3、The test result

4、Train loss

5、The train image result

First，I use the below command line to generation AU E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -out_dir E:\out

I generate CSV file of AU is from 679 column to 696，like below

Finally, I use this to train GANimation. But my train result is not expectation. I suspect that I generation is false.

Today ,I use FeatureExtraction.exe to generate AU ,but I choice different parameter，the command line and generation CSV of Au in below

1. generation AU from column 6 to column 25 E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -out_dir E:\

E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -au_static -out_dir E:\out

But ,today I use 1 and 2 openface command line to generate AU ,the result is different. (I used the same picture for the AU generation test)

Can you tell me how to use the right openface command parameter to generate CSV of AU.

Hi, everyone!

I came across this error when running the train process for several epochs. I know that it is a PyTorch error, however I want to know is there some problems with my dataset that raise this error, and whether this error can be resolved by this project.

I tried to track the problem, this was caused by "src->nDimention <= 0" in PyTorch, but I don't know what was wrong with my data.

Any reply will be appreciated!

I tried running the training on about 110k images. I used all the default parameters for the training. But after around 11 epochs, I get a segfault in the training. I don't know where this error is originating from.

Also, since I had 11 epochs already saved, I just tried running a test on the sample images provided to see the performance. But the results seems very bad. Did you notice something similar? Is it because of the epoch that I used and it can be resolved with using a higher epoch?

Along the same lines, what would you say as the optimum epoch for the training? The default is about 30 epochs.

Thank you for your time.

python train.py --data_dir mydata --name experiment_1 --batch_size 25 ------------ Options ------------- D_adam_b1: 0.5 D_adam_b2: 0.999 G_adam_b1: 0.5 G_adam_b2: 0.999 aus_file: aus_openface.pkl batch_size: 25 checkpoints_dir: ./checkpoints cond_nc: 17 data_dir: mydata dataset_mode: aus display_freq_s: 300 do_saturate_mask: False gpu_ids: [0] image_size: 128 images_folder: imgs is_train: True lambda_D_cond: 4000 lambda_D_gp: 10 lambda_D_prob: 1 lambda_cyc: 10 lambda_mask: 0.1 lambda_mask_smooth: 1e-05 load_epoch: 0 lr_D: 0.0001 lr_G: 0.0001 model: ganimation n_threads_test: 1 n_threads_train: 4 name: experiment_1 nepochs_decay: 10 nepochs_no_decay: 20 num_iters_validate: 1 poses_g_sigma: 0.06 print_freq_s: 60 save_latest_freq_s: 3600 serial_batches: False test_ids_file: test_ids.csv train_G_every_n_iterations: 5 train_ids_file: train_ids.csv -------------- End ---------------- ./checkpoints/experiment_1 Dataset AusDataset was created Dataset AusDataset was created #train images = 91 #test images = 91 Network generator_wasserstein_gan was created Network discriminator_wasserstein_gan was created Model GANimation was created Traceback (most recent call last): File "train.py", line 141, in Train() File "train.py", line 27, in init self._train() File "train.py", line 40, in _train self._train_epoch(i_epoch) File "train.py", line 69, in _train_epoch self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator) File "/data/GANimation/models/ganimation.py", line 207, in optimize_parameters loss_D, fake_imgs_masked = self._forward_D() File "/data/GANimation/models/ganimation.py", line 278, in _forward_D d_real_img_prob, d_real_img_cond = self._D.forward(self._real_img) File "/data/GANimation/networks/discriminator_wasserstein_gan.py", line 29, in forward out_aux = self.conv2(h) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 477, in call Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers result = self.forward(*input, **kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward finalizer() File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call res = self._callback(*self._args, **self._kwargs) File "/usr/lib/python2.7/shutil.py", line 252, in rmtree self.padding, self.dilation, self.groups) RuntimeError: sizes must be non-negative

saving the latest model (epoch 3, total_steps 375825) saved net: ./checkpoints/experiment_1/net_epoch_3_id_G.pth saved net: ./checkpoints/experiment_1/net_epoch_3_id_D.pth [27/08/2018 18:08:54] (T, epoch: 3, it: 121/7516, t/smpl: 0.017s) g_fake:0.000 g_cond:0.000 g_mskd_fake:-0.242 g_mskd_cond:3.193 g_cyc:1.215 g_rgb:0.000 g_rgb_un:0.000 g_rgb_s:0.000 g_m1:0.067 g_m2:0.067 g_m1_s:0.134 g_m2_s:0.129 g_idt:0.000 d_real:-0.856 d_cond:2.661 d_fake:0.491 d_gp:0.107 Traceback (most recent call last): File "train.py", line 141, in Train() File "train.py", line 27, in init self._train() File "train.py", line 40, in _train self._train_epoch(i_epoch) File "train.py", line 59, in _train_epoch for i_train_batch, train_batch in enumerate(self._dataset_train): File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 210, in next return self._process_next_batch(batch) File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 230, in _process_next_batch raise batch.exc_type(batch.exc_msg) RuntimeError: Traceback (most recent call last): File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in default_collate return {key: default_collate([d[key] for d in batch]) for key in batch[0]} File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in return {key: default_collate([d[key] for d in batch]) for key in batch[0]} File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 105, in default_collate return torch.stack([torch.from_numpy(b) for b in batch], 0) RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch

Hello, I get an error like this. Can you give me some suggestions? Maybe the pytorch version is not right?

dear friend: thank you for sharing your code,however when i "import torchvision.transforms as transforms" and "from utils import face_utils",there was a "Segmentation fault (core dumped)".when i run only one of them it 's fine.i seems that there is a conflict .Would you mind take some suggest.

I have trained the model but I can see that desired condition in test.py file is expression = np.random.uniform(0, 1, opt.cond_nc)

I am running it as python test.py --input_path ds/imgcelebaprocessed/000009.jpg --name experiment_128v2

How can I modify expression of generated output? Have you used different test file which is not shared?

@albertpumarola Thank you for your contribution on this work ! When I run the test like this: python test.py --input_path ./sample_dataset/imgs/N_0000001507_00202.jpg It came :

And there is an image in output/ like this :

Is the model not public available? Thank you! Looking forward to your reply.

Any idea why this error occurs? I am using 224x224 celeba aligned database.

Traceback (most recent call last):
  File "train.py", line 141, in <module>
    Train()
  File "train.py", line 27, in __init__
    self._train()
  File "train.py", line 40, in _train
    self._train_epoch(i_epoch)
  File "train.py", line 69, in _train_epoch
    self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
  File "/home/aidas/GANimation/models/ganimation.py", line 208, in optimize_parameters
    loss_D, fake_imgs_masked = self._forward_D()
  File "/home/aidas/GANimation/models/ganimation.py", line 281, in _forward_D
    self._loss_d_cond = self._criterion_D_cond(d_real_img_cond, self._real_cond) / self._B * self._opt.lambda_D_cond
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/loss.py", line 421, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1716, in mse_loss
    return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss, input, target, reduction)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1674, in _pointwise_loss
    return lambd_optimized(input, target, reduction)
RuntimeError: input and target shapes do not match: input [25 x 17 x 2 x 2], target [25 x 17] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12

While executing the train.py file,

IndexError: invalid index of a 0-dim tensor. Use tensor.item() in Python or tensor.item<T>() in C++ to convert a 0-dim tensor to a number

The complete log has been attached below:

D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 4
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: GANimation-master/sample_dataset1
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: experiment_1
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 15
#test images = 15
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/content/GANimation-master/train.py in <module>()
    139 
    140 if __name__ == "__main__":
--> 141     Train()

4 frames
/content/GANimation-master/models/ganimation.py in get_current_errors(self)
    318         loss_dict = OrderedDict([('g_fake', self._loss_g_fake.data[0]),
    319                                  ('g_cond', self._loss_g_cond.data[0]),
--> 320                                  ('g_mskd_fake', self._loss_g_masked_fake.data[0]),
    321                                  ('g_mskd_cond', self._loss_g_masked_cond.data[0]),
    322                                  ('g_cyc', self._loss_g_cyc.data[0]),

IndexError: invalid index of a 0-dim tensor. Use tensor.item() in Python or tensor.item<T>() in C++ to convert a 0-dim tensor to a number

I have run all the previous commands, including the requirements.txt file. While looking for this over the net, I found that it is due to the update in the Pytorch version which does not allow accessing the last element as [0], we need to use .item(). But, even after changing it, I am getting the same error.

Anyone facing the same issues?

Fake image generation in ganimation.py is like this: # generate fake images fake_imgs, fake_img_mask = self._G.forward(real_img_a, real_cond_b) fake_img_mask = self._do_if_necessary_saturate_mask(fake_img_mask, saturate=self._opt.do_saturate_mask) fake_imgs_masked = fake_img_mask * real_img_a + (1 - fake_img_mask) * fake_imgs The last line seems to be different from Eq. (1) in the paper. Did I miss something or both work?

The result is amazing! I have two questions about loss.

1. But I have a question that the three loss below will influence each other？some positive some negative.

“self._loss_d_real + self._loss_d_cond + self._loss_d_fake”

2. when I am training “loss_d_real is negative value” is right" and why

self._loss_d_real = self._compute_loss_D(d_real_img_prob, True) * self._opt.lambda_D_prob

def _compute_loss_D(self, estim, is_real):
        return -torch.mean(estim) if is_real else torch.mean(estim)

print(target.shape,type(target))
(40, 40, 9) <class 'numpy.ndarray'>
m = Image.fromarray(target.astype('uint8'))


<ipython-input-31-b037b3c8b026> in make_dataset()
     26         print(target.shape,type(target))
     27         #rescaled = (255.0/feature.max() * (feature - feature.min())).astype(np.uint8)
---> 28         im = Image.fromarray(target.astype('uint8'))
     29         #im.save(train_path)
     30 

~/venv/lib/python3.7/site-packages/PIL/Image.py in fromarray(obj, mode)
   2645             mode, rawmode = _fromarray_typemap[typekey]
   2646         except KeyError:
-> 2647             raise TypeError("Cannot handle this data type")
   2648     else:
   2649         rawmode = mode

TypeError: Cannot handle this data type