Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters.

Overview

Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters.

Overview

This project is a Torch implementation for our CVPR 2016 paper, which performs jointly unsupervised learning of deep CNN and image clusters. The intuition behind this is that better image representation will facilitate clustering, while better clustering results will help representation learning. Given a unlabeled dataset, it will iteratively learn CNN parameters unsupervisedly and cluster images.

Disclaimer

This is a torch version reimplementation to the code used in our CVPR paper. There is a slight difference between the code used to report the results in our paper. The Caffe version code can be found here.

License

This code is released under the MIT License (refer to the LICENSE file for details).

Citation

If you find our code is useful in your researches, please consider citing:

@inproceedings{yangCVPR2016joint,
    Author = {Yang, Jianwei and Parikh, Devi and Batra, Dhruv},
    Title = {Joint Unsupervised Learning of Deep Representations and Image Clusters},
    Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    Year = {2016}
}

Dependencies

  1. Torch. Install Torch by:

    $ curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-deps | bash
    $ git clone https://github.com/torch/distro.git ~/torch --recursive
    $ cd ~/torch; 
    $ ./install.sh      # and enter "yes" at the end to modify your bashrc
    $ source ~/.bashrc

    After installing torch, you may also need install some packages using LuaRocks:

    $ luarocks install nn
    $ luarocks install image 

    It is preferred to run the code on GPU. Thus you need to install cunn:

    $ luarocks install cunn
  2. lua-knn. It is used to compute the distance between neighbor samples. Go into the folder, and then compile it with:

    $ luarocks make

Typically, you can run our code after installing the above two packages. Please let me know if error occurs.

Installation Using Nvidia-Docker

  1. Run docker build -t .
  2. Run nvidia-docker run -it /bin/bash

Train model

  1. It is very simple to run the code for training model. For example, if you want to train on USPS dataset, you can run:

    $ th train.lua -dataset USPS -eta 0.9

    Note that it runs on fast mode by default. You can change it to regular mode by setting "-use_fast 0". In the above command, eta is the unfolding rate. For face dataset, we recommand 0.2, while for other datasets, it is set to 0.9 to save training time. During training, you will see the normalize mutual information (NMI) for the clustering results.

  2. You can train multiple models in parallel by:

    $ th train.lua -dataset USPS -eta 0.9 -num_nets 5

    By this way, you weill get 5 different models, and thus 5 possible different results. Statistics such as mean and stddev can be computed on these results.

  3. You can also get the clustering performance when using raw image data and random CNN by

    $ th train.lua -dataset USPS -eta 0.9 -updateCNN 0
  4. You can also change other hyper parameters for model training, such as K_s, K_c, number of epochs in each partial unrolled period, etc.

Datasets

We upload six small datasets: COIL-20, USPS, MNIST-test, CMU-PIE, FRGC, UMist. The other large datasets, COIL-100, MNIST-full and YTF can be found in my google drive here.

Train on your own datasets

Alternatively, you can train the model on your own dataset. As preparations, you need:

  1. Create a hdf5 file with size of NxCxHxW, where N is the total number of images, C is the number of channels, H is the height of image, and W the width of image. Then move it to datasets/dataset_name/data4torch.h5

  2. Create a lua file to define the network architecture for your dataset. Put it in models_def/dataset_name.lua.

  3. Afterwards, you can run train.lua by specifying the dataset name as your own dataset. That's it!

Compared Approaches

We upload the code for the compared approaches in matlab folder. Please refer to the original paper for details and cite them properly. In this foler, we also attach the evaluation code for two metric: normalized mutual information (NMI) and clustering accuracy (AC).

Q&A

You are welcome to send message to (jw2yang at vt.edu) if you have any issue on this code.

Comments
  • Any Training  instructions?

    Any Training instructions?

    HI @jwyang Great work,and I think this is what I am looking for ! But I found that it is a little difficult for me to train custom dataset .Could your please give some help?

    My question is : How to create training dataset?What 's the format of the datas? Now i want train my custom images,I have no idea how to transform them to .h5 file could you please give a step by step training instructions()?I think this will help other people too. Thank you very much!!!!

    opened by albertyou2 10
  • Image Size Problem

    Image Size Problem

    Hi, very thankful to your released code, it helps me a lot! I have tried running on my own dataset. If the image size is 3x32x32, all things go well. However, when I set image size to 3x64x64, I got the error "/usr/local/bin/luajit: bad argument #2 to '?' (out of range at /root/torch/pkg/torch/generic/Tensor.c:913)" Could you please help me with this problem?

    opened by r05922121 8
  • parameters set

    parameters set

    hi, I have run your code successfully! now, I want to know how to set the target number of clusters, the initial image labels and CNN parameters. thank you!

    opened by frankwanL 5
  • "bad argument #2 to '?'" error on CIFAR dataset

    Hi,

    I'm trying to learn clustering on CIFAR-10 dataset. As a feature extraction model I'm using FRGC's model with minimal changes:

    • nInputLayers[0] incrreased to 4 because I have RGBA images,
    • the size of chunks have been saved in local chunkSizes = {5, 5} because I'm planning to change the size for each layer separately. So I'm using convolutional NN as following: module:add(backend.SpatialConvolution(nInputPlane, nOutputPlane, chunkSize, chunkSize, 1, 1, 0, 0))

    But I'm catching the error:

    [torch.LongStorage of size 2]
    
    sigma: 	0	
    /home/velkerr/PHd/libs/torch/install/bin/luajit: bad argument #2 to '?' (out of range at /home/velkerr/PHd/libs/torch/pkg/torch/generic/Tensor.c:704)
    stack traceback:
    	[C]: at 0x7f0a0f314580
    	[C]: in function '__newindex'
    	./affinity/affinity.lua:39: in function 'compute'
    	train.lua:201: in function 'updateLabels'
    	train.lua:284: in function 'merge_label'
    	train.lua:482: in main chunk
    	[C]: in function 'dofile'
    	...libs/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
    	[C]: at 0x00405d00
    

    As I understood, the error caused by too small dataset or too large batch size. I've experimented with different batchSizes (500, 100, 50) but I'm still seeing this error.

    What I'm doing wrong? Thank you for help in advance.

    opened by VeLKerr 4
  • class torch.LongTensor cannot be indexed, cvt2TabelLabels error

    class torch.LongTensor cannot be indexed, cvt2TabelLabels error

    I'm having trouble getting my custom data formatted into an appropriate hdf5 file, per your description. I manage to create the hdf5 file but I think I am having trouble with the 'labels' component of the hdf5 file:

    stack traceback:
    
    /home/ec2-user/distro/install/bin/luajit: train.lua:104: the class torch.LongTensor cannot be indexed
    stack traceback:
    	[C]: in function '__newindex'
    	train.lua:104: in function 'cvt2TabelLabels'
    	train.lua:162: in main chunk
    	[C]: in function 'dofile'
    	...ser/distro/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
    	[C]: at 0x004064c0
    

    In general I have a writer like this (where the labels are dummies for now for demonstration purposes):

    import glob
    import os.path
    import time
    from PIL import Image
    import tempfile
    import numpy as np
    
    ims = [np.array(Image.open(f)) for f in glob.glob('*.jpg')]
    
    def setup_hdf5_file(shape, name, **options):
        f = h5py.File(name+'.h5', 'w')
        f.create_dataset('data', shape, **options)
        f.create_dataset('labels', (len(ims),1), **options)
        return f
    
    def save_images(f):
        for i, im in enumerate(ims):
            f['data'][i,...] = im.T
            f['labels'][i,...] = int(1)
        f.close()
    
    def benchmark(name, **options):
        tstart = time.time()
        f = setup_hdf5_file((len(ims),3,256, 256), name, **options)
        save_images(f)
        tstop = time.time()
        size = os.path.getsize(name+'.h5')
        print("{0}: {1:.1f}s, {2}MB".format(name, tstop-tstart, size//1e6))
    
    
    benchmark('data4torch')
    

    it is not clear to me what format may be expected for the 'labels' (also are they even needed given the context of your use case? You don't mention this in the readme.

    opened by shaystrong 3
  • How to use cpu to run the torch code.

    How to use cpu to run the torch code.

    There is no GPU on my computer. How to use cpu to run the torch code. I have set gpu to -1, but still got error: module 'cunn' not found: No LuaRocks module found for "cunn".

    opened by xiaogangLi 3
  • It is difficult to train in Large dataset.

    It is difficult to train in Large dataset.

    I use 80000 samples to train the jointed net. But when I finished the first CNN update, it is difficult to run the next step. This code seemly have a large amount of computation in computing the 'Affinity', How can I solve this problem?

    opened by Paper99 3
  • The wrong about dependencies and the fault of running the code

    The wrong about dependencies and the fault of running the code

    Hello, I have two questions when I install the torch with ubuntu16.04.

    1. Ubuntu 16.04 don't provide the libqt4-core & libqt4-gui in the origin package, so when I run the command curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-deps | bash I failed and do not know whether it is affect the next steps.

    2. When I skip the first step, and accomplish all the install steps. I run the first test code.

    th train.lua -dataset USPS -eta 0.9

    then the fault as follows:

    /home/haichao/torch/distro-master/install/bin/luajit: ...torch/distro-master/install/share/lua/5.1/trepl/init.lua:3 89: module 'hdf5' not found:No LuaRocks module found for hdf5 no field package.preload['hdf5'] no file '/home/haichao/.luarocks/share/lua/5.1/hdf5.lua' no file '/home/haichao/.luarocks/share/lua/5.1/hdf5/init.lua' no file '/home/haichao/torch/distro-master/install/share/lua/5.1/hdf5.lua' no file '/home/haichao/torch/distro-master/install/share/lua/5.1/hdf5/init.lua' no file './hdf5.lua' no file '/home/haichao/torch/distro-master/install/share/luajit-2.1.0-beta1/hdf5.lua' no file '/usr/local/share/lua/5.1/hdf5.lua' no file '/usr/local/share/lua/5.1/hdf5/init.lua' no file '/home/haichao/.luarocks/lib/lua/5.1/hdf5.so' no file '/home/haichao/torch/distro-master/install/lib/lua/5.1/hdf5.so' no file '/home/haichao/torch/distro-master/install/lib/hdf5.so' no file './hdf5.so' no file '/usr/local/lib/lua/5.1/hdf5.so' no file '/usr/local/lib/lua/5.1/loadall.so'

    I remember I have installed the hdf5 in other folder, and I have configured the .bashrc . So are there any steps I need to do ?

    opened by zz-kuku 3
  • how to calculate diff in loss layer

    how to calculate diff in loss layer

    Hi,

    I tried to implement the loss layer using python you mentioned in the paper (eq(11)), however I have no idea how the diff should look like, because usually there will be three diffs for anchors/positives/negatives. Can you please provide the sample using only feats and labels as inputs?

    Thanks!

    opened by xiaoyuliu 2
  •  code issue

    code issue

    hello, I have readed you CVPR paper “Joint Unsupervised Learning of Deep Representations and mage cluster”,but I don't know how to run the program given. I want you help ,thank you!

    opened by alfredw2016 1
  • implementation of run_step_fast

    implementation of run_step_fast

    The code in run_step_fast(), doesn't match the formula in annotation. This part aims at updating a_us_t[idx_c_a,:]

    below is the annotation update A_t(i->idx_c_a) = r_a * A_t(i->idx_c_a) + r_b * A_t(i->idx_c_b) (fast algorithm) but code doesn't not contain r_a and r_b, which looks like: A_us_t:indexAdd(1, torch.LongTensor{idx_c_a}, A_us_t:index(1, torch.LongTensor{idx_c_b}))

    this makes me confused, should I change the code to annotation version(implementation in function merge_two_clusters())?

    Thanks for your work!

    opened by zhyx12 0
  • deep_recon function not found

    deep_recon function not found

    @jwyang hi, when i try to use this file deep_seminmf_nonlinear.m. I found it fail to run due to missing function deep_recon(). Could you help me with this problem? thx.

    opened by Psychic-DL 0
  • install lua-knn failed!

    install lua-knn failed!

    I have spent one day on configuring the environment for your torch code, when i go to the last step, i.e., install lua-knn, it failed. What's the problem? I'm so tired.

    opened by huweibo 0
  • doubt regarding the final loss

    doubt regarding the final loss

    Hi, Thank you for the code. I had a doubt regarding the Affinity function used in the reformulated loss(eq 11). From what I understand, after computing the triplets by organize_samples() function, they are fed to the criterion_triplet to get the loss. In TripletEmbedding.lua are the Affinities( A(xi,xj) and A(xi,xk) in eq(11)) represented by delta_pos and delta_net ?

    I am a bit confused because I could not see the actual Affinity function(calculated in the agg_clustering.c and as mentioned Graph Degree Linkage paper[68]) to be used while calculating the loss.

    Thanks.

    opened by ayush1997 2
  • best solution for CPU-only (knn)

    best solution for CPU-only (knn)

    If we want to use this package but not using GPU drivers, what's the best solution?

    The lua-knn package is specifically for GPU, and the only other package I found is https://github.com/rlowrance/kernel-smoothers which I have not been able to figure out how to "install" -- the makefile just looks like a test suite, and it's not included in luarocks despite being referred to by the official docs.

    Best option I can think of right now is to code the knn myself.

    opened by tjann 1
  • Affinity for clusters

    Affinity for clusters

    https://github.com/jwyang/JULE.torch/blob/d43251d93aba8c00b716c62d6d5d6c34ee29bcb9/agg_clustering_c/agg_clustering.c#L78

    In the lines 65-68 (agg_clustering.c file), you first get the length of cluster i and cluster j, which are saved in Y_i_size, Y_j_size, respectively. But I am confused that you use THTensor_(get2d)(Y, j, n) - 1 to call the element Y[j,n], in which the maximum of n is Y_j_size, but the column size of Y is Y->size[1].

    opened by eleozzr 0
Owner
Jianwei Yang
Senior Researcher @ Microsoft
Jianwei Yang
Implementation of "JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting"

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting Pytorch implementation for the paper "JOKR: Joint Keypoint Repres

null 45 Dec 25, 2022
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
Code for the paper "Unsupervised Contrastive Learning of Sound Event Representations", ICASSP 2021.

Unsupervised Contrastive Learning of Sound Event Representations This repository contains the code for the following paper. If you use this code or pa

Eduardo Fonseca 81 Dec 22, 2022
CURL: Contrastive Unsupervised Representations for Reinforcement Learning

CURL Rainbow Status: Archive (code is provided as-is, no updates expected) This is an implementation of CURL: Contrastive Unsupervised Representations

Aravind Srinivas 46 Dec 12, 2022
Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations This repo contains the Pytorch implementation of our paper: Revisit

Wouter Van Gansbeke 80 Nov 20, 2022
Framework for joint representation learning, evaluation through multimodal registration and comparison with image translation based approaches

CoMIR: Contrastive Multimodal Image Representation for Registration Framework ?? Registration of images in different modalities with Deep Learning ??

Methods for Image Data Analysis - MIDA 55 Dec 9, 2022
Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.

U2Fusion Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal (VIS-IR, medical), multi

Han Xu 129 Dec 11, 2022
A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes.

OMNI A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes. Why? When I finished my Kubernetes cluster using a few Raspber

Matias Godoy 148 Dec 29, 2022
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

TensorFlowOnSpark TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters. By combining salient features from the T

Yahoo 3.8k Jan 4, 2023
Automatic tool focused on deriving metallicities of open clusters

metalcode Automatic tool focused on deriving metallicities of open clusters. Based on the method described in Pöhnl & Paunzen (2010, https://ui.adsabs

null 2 Dec 13, 2021
The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Enformer TPU training script (wip) The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pyt

Phil Wang 10 Oct 19, 2022
Static Features Classifier - A static features classifier for Point-Could clusters using an Attention-RNN model

Static Features Classifier This is a static features classifier for Point-Could

ABDALKARIM MOHTASIB 1 Jan 25, 2022
Joint deep network for feature line detection and description

SOLD² - Self-supervised Occlusion-aware Line Description and Detection This repository contains the implementation of the paper: SOLD² : Self-supervis

Computer Vision and Geometry Lab 427 Dec 27, 2022
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

null 83 Jan 1, 2023
A Joint Video and Image Encoder for End-to-End Retrieval

Frozen️ in Time ❄️ ️️️️ ⏳ A Joint Video and Image Encoder for End-to-End Retrieval project page | arXiv | webvid-data Repository containing the code,

null 225 Dec 25, 2022
The Pytorch code of "Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification", CVPR 2022 (Oral).

DeepBDC for few-shot learning        Introduction In this repo, we provide the implementation of the following paper: "Joint Distribution Matters: Dee

FeiLong 116 Dec 19, 2022
Code for Dual Contrastive Learning for Unsupervised Image-to-Image Translation, NTIRE, CVPRW 2021.

arXiv Dual Contrastive Learning Adversarial Generative Networks (DCLGAN) We provide our PyTorch implementation of DCLGAN, which is a simple yet powerf

null 119 Dec 4, 2022
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Google Research 80 Dec 25, 2022