Hello. I just tried to run this code with Ubuntu 16.04 LTS, Geforce TITAN X GPU with Pytorch 0.4.1 While running the code with mpirun -n 4 python plot_surface.py --mpi --cuda --model vgg9 --x=-0.5:1.5:401 --dir_type states --model_file cifar10/trained_nets/vgg9_sgd_lr=0.1_bs=128_wd=0.0_save_epoch=1/model_300.t7 --model_file2 cifar10/trained_nets/vgg9_sgd_lr=0.1_bs=8192_wd=0.0_save_epoch=1/model_300.t7 --plot --show , nothing has happened.

But just delete mpirun -n 4, then code starts running. I think it is in the training process. And after the training process, I can see the plotted results.

Can I run this code without 'OpenMPI'?? I only know that openmpi is just for parallel computation. So can I use the code with python plot_surface.py --mpi --cuda --model vgg9 --x=-0.5:1.5:401 --dir_type states --model_file cifar10/trained_nets/vgg9_sgd_lr=0.1_bs=128_wd=0.0_save_epoch=1/model_300.t7 --model_file2 cifar10/trained_nets/vgg9_sgd_lr=0.1_bs=8192_wd=0.0_save_epoch=1/model_300.t7 --plot --show?

hi

I download Resnet h5 files from https://drive.google.com/a/cs.umd.edu/file/d/12oxkvfaKcPyyHiOevVNTBzaQ1zAFlNPX/view?usp=sharing

then I try the conversion python h52vtp.py --surf_file path_to_surf_file --surf_name train_loss --zmax 10 --log

but I get this error

Traceback (most recent call last): File "loss-landscape/h52vtp.py", line 259, in h5_to_vtp(args.surf_file, args.surf_name, log=args.log, zmax=args.zmax, interp=args.interp) File "loss-landscape/h52vtp.py", line 38, in h5_to_vtp [xcoordinates, ycoordinates] = np.meshgrid(f['xcoordinates'][:], f['ycoordinates'][:][:]) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/group.py", line 177, in getitem oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5o.pyx", line 190, in h5py.h5o.open KeyError: "Unable to open object (object 'xcoordinates' doesn't exist)"

any tips?

and most importantly, can you advice me how to convert .h5 files to .obj mesh files? thank you so much

flake8 testing of https://github.com/tomgoldstein/loss-landscape on Python 3.7.1

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./cifar10/dataloader.py:7:16: F821 undefined name 'os'
        assert os.path.exists(trainloader_path), 'trainloader does not exist'
               ^
./cifar10/dataloader.py:7:31: F821 undefined name 'trainloader_path'
        assert os.path.exists(trainloader_path), 'trainloader does not exist'
                              ^
./cifar10/dataloader.py:8:16: F821 undefined name 'os'
        assert os.path.exists(testloader_path), 'testloader does not exist'
               ^
./cifar10/dataloader.py:8:31: F821 undefined name 'testloader_path'
        assert os.path.exists(testloader_path), 'testloader does not exist'
                              ^
./cifar10/dataloader.py:9:34: F821 undefined name 'trainloader_path'
        trainloader = torch.load(trainloader_path)
                                 ^
./cifar10/dataloader.py:10:33: F821 undefined name 'testloader_path'
        testloader = torch.load(testloader_path)
                                ^
6     F821 undefined name 'os'
6

So newer versions of h5py throw the following error when f = h5py.File(surf_file, 'r+' if rank == 0 else 'r') is called:

OSError: Unable to open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

This fix prevents this error from happening and the project can run on newer versions of h5py. This will also finally fix issue https://github.com/tomgoldstein/loss-landscape/issues/4 .

I run the plot_surface code like so:

    /usr/bin/python -u /local/mnt/workspace/ikarmano/Gitlab/sagd/loss-landscape/plot_surface.py --cuda \
    --x=-1:1:51 --y=-1:1:51 --model_file models/32_32_32_32_32_32_32_32_32_32_32_32_32_32_32cnn.t \
    --dir_type weights --xnorm filter --xignore biasbn --ynorm filter --yignore biasbn --plot

And it seem to calculate the loss fine:

Evaluating rank 2  90/2601  (3.5%)  coord=[ 0.56 -0.96] 	train_loss= 21.470 	train_acc=14.54 	time=5.28 	sync=0.00
Evaluating rank 2  91/2601  (3.5%)  coord=[ 0.6  -0.96] 	train_loss= 22.225 	train_acc=14.10 	time=5.65 	sync=0.00
Evaluating rank 2  92/2601  (3.5%)  coord=[ 0.64 -0.96] 	train_loss= 23.044 	train_acc=13.67 	time=5.92 	sync=0.00
Evaluating rank 2  93/2601  (3.6%)  coord=[ 0.68 -0.96] 	train_loss= 23.935 	train_acc=13.33 	time=5.71 	sync=0.00
Evaluating rank 2  94/2601  (3.6%)  coord=[ 0.72 -0.96] 	train_loss= 24.905 	train_acc=13.02 	time=5.65 	sync=0.00
Evaluating rank 2  95/2601  (3.7%)  coord=[ 0.76 -0.96] 	train_loss= 25.958 	train_acc=12.66 	time=5.50 	sync=0.00
Evaluating rank 2  96/2601  (3.7%)  coord=[ 0.8  -0.96] 	train_loss= 27.100 	train_acc=12.37 	time=5.99 	sync=0.00
Evaluating rank 2  97/2601  (3.7%)  coord=[ 0.84 -0.96] 	train_loss= 28.334 	train_acc=12.13 	time=5.85 	sync=0.00
Evaluating rank 2  98/2601  (3.8%)  coord=[ 0.88 -0.96] 	train_loss= 29.666 	train_acc=11.91 	time=5.71 	sync=0.00
Evaluating rank 2  99/2601  (3.8%)  coord=[ 0.92 -0.96] 	train_loss= 31.101 	train_acc=11.69 	time=5.58 	sync=0.00

However, the plot functions do not work because 'train_loss' is not found:

train_loss is not found in ../models/32_32_32_32_32_32_32_32_32_32_32_32_32_32_32cnn.t_weights_xignore=biasbn_xnorm=filter_yignore=biasbn_ynorm=filter.h5_[-1.0,1.0,51]x[-1.0,1.0,51].h5

And if I print the keys(), it's just:

<KeysViewHDF5 ['dir_file', 'xcoordinates', 'ycoordinates']>

Not sure what I'm doing wrong?

To plot the figure 10 in your paper, I am assuming I should generate the PCA directions from plot_trajectory.py first then use the PCA directions to plot the loss contours of the final model.

I had to write my own code because of some technical difficulties. However, I notice that for my data, the trajectory should start from loss ~= 0.9 but the loss contour of the final model is far from 0.9 at the trajectory starting point. This makes me think that actually there is no guarantee that loss contours which the plotted trajectory comes across reflect the real loss, in other words, the loss contours of the final model do not show the "loss landscape" along the trajectory. However, when I reduced the number of models used to perform the PCA, the loss at the trajectory starting point is near 0.9 loss contour of the final model.

This is reasonable since the final model is perturbed along pc1 and pc2, while the trajectory is projected to pc1 and pc2 and a model can actually be far away from the projection, thus the loss corresponding to the trajectory can be far from the loss contours of the final model.

I understand that pc1 and pc2 can explain most of the variance among the parameters of all the models, but there is no guarantee that it can explain the most difference between any given model and the final model. That is probably why I got more "accurate" results when I use less models to estimate the principal components?

There is an implicit float16 -> float32 conversion in the original code if net weights are float16

p.data = w + torch.Tensor(d).type(type(w))

thus results in inaccurate loss computation. The following code can avoid this problem since it take the exact w.dtype when doing the conversion

p.data = w + torch.Tensor(d).type_as(w.dtype)

Hi, I'm not sure if I'm missing something simple, but using either VGG9 or Resnet56 for both the 1D and 2D visualizations gives nan losses when they should be smaller. Using Pytorch 0.4.1, models downloaded from provided links

We have some nodes which only support command interface, and there is no GUI display. If we run the code, then we will get

_tkinter.TclError: no display name and no $DISPLAY environment variable

Can we save figures just into files such as pdf files?

Hi guys,

I have already get the contour and trajectory plot, but in two different pdfs. I really want to plot the trajectory on the contour (just like the figure 9 in paper). Do anyone have successfully done that?

Thanks!

I can implement plot the surface of the function using the trained networks attached. If I want to apply the code on a new model, could you provide more details about how to generate the .t7 and .h5 files?

Are there any plans to add support for h5py 3.7 and hdf5 1.12.0 in the near future? At this point, it is very difficult to find and compile hdf 1.8.16 on MacOS Monterey, as it is obsolete. Also, h5py 2.7 (the required version) does not work with later versions of python (3.9+) either.

In the figure 9 of your paper, I noticed that by using L2 norm, the landscape becomes more narrow around the minimal point. Which is different from previous figures.

I do know that you are using a different way of choosing vectors by PCA. And it can be understood by a way from-result-to-cause -- that is, L2 norm makes it harder to train, so the convex part is smaller. However, I curious if you have any deeper insight of this pattern? Thanks!

When I re-run the code for visualizing 2D loss contours, the generated 2D loss contour and 3d surface are different from the figures provided in google drive. I use the same setting and pre-trained models:

mpirun -n 4 python plot_surface.py --mpi --cuda --model resnet56 --x=-1:1:51 --y=-1:1:51 \
--model_file cifar10/trained_nets/resnet56_sgd_lr=0.1_bs=128_wd=0.0005/model_300.t7 \
--dir_type weights --xnorm filter --xignore biasbn --ynorm filter --yignore biasbn  --plot 

--dir_type states indicates the direction contains dimensions for all parameters as well as the statistics of the BN layers (running_mean and running_var). Note that ignoring running_mean and running_var cannot produce correct loss values when plotting two solutions together in the same figure.

Why running mean and var important for plotting two solutions in the same figure? Could anyone help me with this one?

I'm trying to plot two or three solutions in one surface plot.

Thanks