GoodNews Everyone! Context driven entity aware captioning for news images

Overview

This is the code for a CVPR 2019 paper, called GoodNews Everyone! Context driven entity aware captioning for news images. Enjoy!

Model preview:

GoodNews Model!

Huge Thanks goes to New York Times API for providing such a service for FREE!

Another Thanks to @ruotianluo for providing the captioning code.

Dependencies/Requirements:

pytorch==1.0.0
spacy==2.0.11
h5py==2.7.0
bs4==4.5.3
joblib==0.12.2
nltk==3.2.3
tqdm==4.19.5
urllib2==2.7
goose==1.0.25
urlparse
unidecode

Introduction

We took the first steps to move the captioning systems to interpretation (see the paper for more detail). To this end, we have used New York Times API to retrieve the articles, images and captions.

The structure of this repo is as follows:

  1. Getting the data
  2. Cleaning and formating the data
  3. How to train models

Get the data

You have 3 options to get the data.

Images only

If you want to download the images only and directly start working on the same dataset as ours, then download the cleaned version of the dataset without images: article+caption.json and put it to data/ folder and download the img_urls.json and put it in the get_data/get_images_only/ folder.

Then run

python get_images.py --num_thread 16

Then, you will get the images. After that move to Clean and Format Data section.

PS: I have recieved numerous emails regarding some of the images not present/broken in the img_urls.json. Which is why I decided to put the images on the drive to download in the name of open science. Download all images

Images + articles

If you would like the get the raw version of the article and captions to do your own cleaning and processing, no worries! First download the article_urls and go to folder get_data/with_article_urls/ and run

python get_data_with_urls.py --num_thread 16
python combine_dataset.py 

This will get you the raw version of the caption, articles and also the images. After that move to Clean and Format Data section.

I want more!

As you know, New York Times is huge. Their articles starts from 1881 (It is crazy!) until well today. So in case you want to get ALL the data or expand the data to more years, then first step is go to New York Times API and get an API key. All you have to do is just sign up for the API key.

Once you have the key go to folder get_data/with_api/ and run

python retrieve_all_urls.py --api-key XXXX --start_year XXX --end_year XXX 

This is for getting the article urls and then saving in the format of month-year. Once you have the all urls from the API, then you run

python get_data_api.py
python combine_dataset.py

get_data_api.py retrieves the articles, captions and images. combine_dataset.py combines yearly data into one file after removing data points if they have corrupt image, empty articles or empty captions. After that move to Clean and Format Data section.

Small Note

I also provide the links to images and their data splits (train, val, test). Even though I always use random seed to decide the split, just in case If the GODS meddles with the random seed, here is the link to a json where you can find each image and its split: img_splits.json

Clean and Format the Data

Now that we have the data, it is time to clean, preprocess and format the data.

Preprocess

When you reach this part, you must have captioning_dataset.json in your data/ folder.

Captions

This part is for cleaning the captions (tokenizing, removing non-ascii characters, etc.), splitting train, val, and test and creating anonymize captions.

In other words, we change the caption "Alber Einstein taught in Princeton in 1926" to "PERSON_ taught in ORGANIZATION_ in DATE_." Move to preprocess/ folder and run

python clean_captions.py

Resize Images

To resize the images to 256x256:

python resize.py --root XXXX --img_size 256

Articles

Get the article format that is needed for the encoding methods by running: create_article_set.py

python create_article_set.py

Format

Now to create H5 file for captions, images and articles, just need to go to scripts/ folder and run in order

python prepro_labels.py --max_length 31 --word_count_threshold 4
python prepro_images.py

We proposed 3 different article encoding method. You can download each of encoded article methods, articles_full_avg_, articles_full_wavg, articles_full_TBB.

Or you can use the code to obtain them:

python prepro_articles_avg.py
python prepro_articles_wavg.py
python prepro_articles_tbb.py

Train

Finally we are ready to train. Magical words are:

python train.py --cnn_weight [YOUR HOME DIRECTORY]/.torch/resnet152-b121ed2d.pth 

You can check the opt.py for changing a lot of the options such dimension size, different models, hyperparameters, etc.

Evaluate

After you train your models, you can get the score according commonly used metrics: Bleu, Cider, Spice, Rouge, Meteor. Be sure to specify model_path, cnn_model_path, infos_path and sen_embed_path when runing eval.py. eval.py is usually used in training but it is necessary to run it to get the insertion.

Insertion

Last but not least insert.py. After you run eval.py, it will produce you a json file with the ids and their template captions. To fill the correct named entity, you have to run insert.py:

python insert.py --output [XXX] --dump [True/False] --insertion_method ['ctx', 'att', 'rand']

PS: I have been requested to provide model's output, so I thought it would be best to share it with everyone. Model Output In this folder, you have:

test.json: Test set with raw and template version of the caption.

article.json: Article sentences which is needed in the insert.py.

w/o article folder: All the models output on template captions, without articles.

with article folder: Our models output in the paper with sentence attention(sen_att) and image attention(vis_att), provided in the json. Hope this is helpful to more of you.

Conclusion

Thank you and sorry for the bugs!

Comments
  • After I run clean_captions.py the news_dataset.json file is not created

    After I run clean_captions.py the news_dataset.json file is not created

    I don't get any error and I see print of captions. However, only these are created (val.json is also not created):

    Vice President Joseph R. Biden Jr. in March at a Jordanian-American military training center in Zarqa, Jordan.
    The funeral in Rimoun, Jordan, for Anwar Abu Zaid, a police captain who was killed after he attacked a police training center in November. American and Jordanian officials said they believed that the weapons he used had been meant for a program to train Syrian rebels.
    Ambulances leaving the police training center where Captain Abu Zaid gunned down five people, including two American contractors.
    Jeremy Beck and Rebecca Noelle Brinkley in the Mint Theater production of “Hindle Wakes.”
    Priscilla Chan and Mark Zuckerberg say their charity plan is to "advance human potential and promote equality in areas such as health, education, scientific research and energy."
    The official Rio 2016mona@mona:~/research/GoodNews/prepocess$ ls ../data
    total 1021M
    -rw-rw-r--  1 mona mona   71 Sep  7 12:28 .gitignore
    -rw-rw-r--  1 mona mona 985M Sep  7 12:29 captioning_dataset.json
    drwxrwxr-x 12 mona mona 4.0K Sep  7 12:47 ..
    -rw-rw-r--  1 mona mona  36M Sep  7 21:21 article_urls.json
    drwxrwxr-x  2 mona mona 4.0K Sep  7 21:22 .
    -rw-rw-r--  1 mona mona    0 Sep  9 10:23 test.json
    
    

    Do you know what's the reason and how to fix it? I have used absolute path as well but no luck!

        with open("/home/mona/research/GoodNews/data/captioning_dataset.json", "rb") as f:
            captioning_dataset = json.load(f)
    
        for k, anns in tqdm.tqdm(captioning_dataset.items()):
    
            for ix, img in anns['images'].items():
                try:
                    split = get_split()
    
                    #         import ipdb; ipdb.set_trace()
                    img = preprocess_sentence(img)
                    template, full = NER(' '.join(img))
                    if len(' '.join(template)) != 0:
                        news_data.append({'filename': k + '_' + ix + '.jpg', 'filepath': 'resized', 'cocoid': counter,
                                          'imgid': k + '_' + ix, 'sentences': [], 'sentences_full': [],
                                          #                               'sentences_article':[],
                                          'split': split})
                        news_data[counter]['sentences'].append(
                            {'imgid': counter, 'raw': ' '.join(template), 'tokens': template})
                        news_data[counter]['sentences_full'].append(
                            {'imgid': counter, 'raw': ' '.join(full), 'tokens': full})
                        counter += 1
                except:
                    print(img)
        split_to_ix = {i:n['split'] for i, n in enumerate(news_data)}
        # train = [news_data[k] for k, v in split_to_ix.items() if v =='train']
        val = [news_data[k] for k, v in split_to_ix.items() if v =='val']
        test = [news_data[k] for k, v in split_to_ix.items() if v =='test']
        with open("/home/mona/research/GoodNews/data/test.json", "wb") as f:
            json.dump(test, f)
        with open("/home/mona/research/GoodNews/data/val.json", "wb") as f:
    
    opened by monajalal 2
  • FileNotFoundError: [Errno 2] No such file or directory: 'save/show_attend_tell/model.pth'

    FileNotFoundError: [Errno 2] No such file or directory: 'save/show_attend_tell/model.pth'

    For

    (/scratch2/sjn/pyta) [jalal@goku GoodNews]$ python train.py --cnn_weight data/resnet152-b121ed2d.pth

    I get

    (/scratch2/sjn/pyta) [jalal@goku GoodNews]$ tmux attach
    
    image 127420: PERSON_ and PERSON_ in GPE_ in GPE_
    image 127431: PERSON_ left and PERSON_ in GPE_
    image 127444: PERSON_ in GPE_
    image 127447: PERSON_ in DATE_
    image 127472: PERSON_ in GPE_ on DATE_
    image 127497: PERSON_ a NORP_ NORP_ in GPE_ on DATE_
    image 127535: PERSON_ in DATE_
    image 127536: PERSON_ and PERSON_ in GPE_
    image 127546: PERSON_ in GPE_ on DATE_
    image 127584: PERSON_ a NORP_ leader of the ORG_ in GPE_
    image 127596: A UNK UNK UNK UNK UNK
    evaluating validation preformance... 5023/5000 (5.082450)
    {'testlen': 31857, 'reflen': 75785, 'guess': [31857, 26857, 22238, 17619], 'correct': [10185, 2230, 512, 159]}
    ratio: 0.42036022959688035
    Bleu_1:  0.08051972435966352
    Bleu_2:  0.04103438062512367
    Bleu_3:  0.02137332258808613
    Bleu_4:  0.012205189537708628
    ROUGE_L: 0.1975560912228822
    CIDEr:   0.1773307729244918
    Traceback (most recent call last):
      File "train.py", line 280, in <module>
        train(opt)
      File "train.py", line 238, in train
        torch.save(model.state_dict(), checkpoint_path)
      File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 361, in save
        with _open_file_like(f, 'wb') as opened_file:
      File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 229, in _open_file_like
        return _open_file(name_or_buffer, mode)
      File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 210, in __init__
        super(_open_file, self).__init__(open(name, mode))
    FileNotFoundError: [Errno 2] No such file or directory: 'save/show_attend_tell/model.pth'
    Closing remaining open files:data/data_news_label.h5...done/scratch2/goodnewsdata/data_news_image.h5...done
    

    I'll update here when I find a solution.

    opened by monajalal 1
  • Why running clean_captions.py returns empty files for val.json, test.json, and news_dataset.json?

    Why running clean_captions.py returns empty files for val.json, test.json, and news_dataset.json?

    [jalal@goku GoodNews]$ cd data/
    [jalal@goku data]$ ls
    total 9270700
          4 -rw-r--r--.  1 jalal cs-grad         71 Sep  9 11:25 .gitignore
      36744 -rw-r--r--.  1 jalal cs-grad   37622426 Sep  9 11:27 article_urls.json
    1008000 -rw-r--r--.  1 jalal cs-grad 1032190641 Sep  9 11:27 captioning_dataset.json
    8225936 -rw-r--r--.  1 jalal cs-grad 8423357695 Sep  9 11:29 resized.tar.gz
          4 drwxr-xr-x. 12 jalal cs-grad       4096 Sep  9 12:12 ..
          4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 val.json
          4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 test.json
          4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 news_dataset.json
          0 drwxr-xr-x.  2 jalal cs-grad        192 Sep  9 13:50 .
    

    The code is:

    
    [jalal@goku data]$ cd ../prepocess/
    [jalal@goku prepocess]$ cat clean_captions.py 
    import json
    import nltk
    import spacy
    import numpy as np
    import tqdm
    import unidecode
    from bs4 import BeautifulSoup
    import re
    import unicodedata
    from itertools import groupby
    
    def remove_non_ascii(words):
        """Remove non-ASCII characters from list of tokenized words"""
        new_words = []
        for word in words:
            new_word = unicodedata.normalize('NFKD', word).encode('ascii', 'ignore').decode('utf-8', 'ignore')
            new_words.append(new_word)
        return new_words
    
    def to_lowercase(words):
        """Convert all characters to lowercase from list of tokenized words"""
        new_words = []
        for word in words:
            new_word = word.lower()
            new_words.append(new_word)
        return new_words
    
    def remove_punctuation(words):
        """Remove punctuation from list of tokenized words"""
        new_words = []
        for word in words:
            new_word = re.sub(r'[^\w\s]', '', word)
            if new_word != '':
                new_words.append(new_word)
        return new_words
    
    def replace_numbers(words):
        """Replace all interger occurrences in list of tokenized words with textual representation"""
        p = inflect.engine()
        new_words = []
        for word in words:
            if word.isdigit():
                new_word = p.number_to_words(word)
                new_words.append(new_word)
            else:
                new_words.append(word)
        return new_words
    
    def remove_stopwords(words):
        """Remove stop words from list of tokenized words"""
        new_words = []
        for word in words:
            if word not in stopwords.words('english'):
                new_words.append(word)
        return new_words
    
    # def stem_words(words):
    #     """Stem words in list of tokenized words"""
    #     stemmer = LancasterStemmer()
    #     stems = []
    #     for word in words:
    #         stem = stemmer.stem(word)
    #         stems.append(stem)
    #     return stems
    #
    # def lemmatize_verbs(words):
    #     """Lemmatize verbs in list of tokenized words"""
    #     lemmatizer = WordNetLemmatizer()
    #     lemmas = []
    #     for word in words:
    #         lemma = lemmatizer.lemmatize(word, pos='v')
    #         lemmas.append(lemma)
    #     return lemmas
    
    def normalize(words):
        words = remove_non_ascii(words)
    #     words = to_lowercase(words)
        words = remove_punctuation(words)
    #     words = replace_numbers(words)
    #     words = remove_stopwords(words)
        return words
    
    def strip_html(text):
        soup = BeautifulSoup(text, "html.parser")
        return soup.get_text()
    
    def remove_between_square_brackets(text):
        return re.sub('\[[^]]*\]', '', text)
    
    def denoise_text(text):
        text = strip_html(text)
        text = remove_between_square_brackets(text)
        return text
    def replace_contractions(text):
        """Replace contractions in string of text"""
        return contractions.fix(text)
    
    def preprocess_sentence(sen):
        sen = sen.strip()
    #     sen = re.sub(uri_re, "", sen)
        sen = sen.encode('ascii',errors='ignore')
        sen = unidecode.unidecode(sen)
        sen = denoise_text(sen)
        # sen = replace_contractions(sen)
        sen = nltk.tokenize.word_tokenize(sen)
        sen = normalize([unicode(s) for s in sen])
    #     sen = normalize(unicode(sen))
    #     return sen
        return sen
    #     tokenized = nltk.tokenize.word_tokenize(temp)
    #     final = normalize(unicode(tokenized))
    #     return ''.join(final)
    
    # def NER(sen):
    #     doc = nlp(unicode(sen))
    #     return [d.ent_iob_+'-'+d.ent_type_ if d.ent_iob_ != 'O' else d.text for d in doc ], [d.text for d in doc]
    def NER(sen):
        doc = nlp(unicode(sen))
    #     text = doc.text
    #     for ent in doc.ents:
    #         text = text.replace(ent.text, ent.label_+'_')
        tokens = [d.text for d in doc]
    #     [ent.merge(ent.root.tag_, ent.text, ent.label_) for ent in doc.ents]
    #     return compact([d.ent_iob_+'-'+d.ent_type_ if d.ent_iob_ != 'O' else d.text for d in doc ]), tokens
    #     return text, tokens
        temp = [d.ent_type_+'_' if d.ent_iob_ != 'O' else d.text for d in doc]
        return [x[0] for x in groupby(temp)], tokens
    
    def get_split():
        rand = np.random.uniform()
        if rand > 0.95:
            split = 'test'
        #             test_num += 1
        elif rand > 0.91 and rand < 0.95:
            split = 'val'
        #         val_num += 1
        else:
            split = 'train'
        #         train_num += 1
        return split
    
    if __name__ == '__main__':
        np.random.seed(42)
        nlp = spacy.load('en', disable=['parser', 'tagger'])
        print('Loading spacy modules.')
        news_data = []
        counter = 0
        test_num, val_num, train_num = 0, 0, 0
    
        print('Loading the json.')
        with open("../data/captioning_dataset.json", "rb") as f:
            captioning_dataset = json.load(f)
    
        for k, anns in tqdm.tqdm(captioning_dataset.items()):
    
            for ix, img in anns['images'].items():
                try:
                    split = get_split()
    
                    #         import ipdb; ipdb.set_trace()
                    img = preprocess_sentence(img)
                    template, full = NER(' '.join(img))
                    if len(' '.join(template)) != 0:
                        news_data.append({'filename': k + '_' + ix + '.jpg', 'filepath': 'resized', 'cocoid': counter,
                                          'imgid': k + '_' + ix, 'sentences': [], 'sentences_full': [],
                                          #                               'sentences_article':[],
                                          'split': split})
                        news_data[counter]['sentences'].append(
                            {'imgid': counter, 'raw': ' '.join(template), 'tokens': template})
                        news_data[counter]['sentences_full'].append(
                            {'imgid': counter, 'raw': ' '.join(full), 'tokens': full})
                        counter += 1
                except:
                    print(img)
        split_to_ix = {i:n['split'] for i, n in enumerate(news_data)}
        # train = [news_data[k] for k, v in split_to_ix.items() if v =='train']
        val = [news_data[k] for k, v in split_to_ix.items() if v =='val']
        test = [news_data[k] for k, v in split_to_ix.items() if v =='test']
        with open("../data/test.json", "w") as f:
            json.dump(test, f)
        with open("../data/val.json", "w") as f:
            json.dump(val, f)
        with open("../data/news_dataset.json", "w") as f:
            json.dump(news_data, f)
    
    
    opened by monajalal 1
  • Where can I download news_dataset.json ?

    Where can I download news_dataset.json ?

    (base) mona@mona:~/research/GoodNews/prepocess$ python create_article_set.py
    Traceback (most recent call last):
      File "create_article_set.py", line 11, in <module>
        with open("../data/news_dataset.json", "rb") as f:
    FileNotFoundError: [Errno 2] No such file or directory: '../data/news_dataset.json'
    
    
    opened by monajalal 1
  • where should we place resized.tar.gz?

    where should we place resized.tar.gz?

    (base) mona@mona:~/research/GoodNews/images$ ls
    total 7.9G
    -rw-rw-r--  1 mona mona   71 Sep  7 12:28 .gitignore
    drwxrwxr-x 12 mona mona 4.0K Sep  7 12:28 ..
    -rw-rw-r--  1 mona mona 7.9G Sep  7 12:35 resized.tar.gz
    drwxrwxr-x  3 mona mona 4.0K Sep  7 12:37 .
    drwxrwxr-x  2 mona mona  15M Sep  7 12:41 resized
    

    I have a tree like this: tree_GoodNews

    opened by monajalal 1
  • THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error

    THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error

    Do you know how I could fix this?

    [jalal@goku GoodNews]$ python train.py --cnn_weight data/resnet152-b121ed2d.pth 
    DataLoader loading json file:  data/data_news.json
    vocab size is  37200
    DataLoader loading h5 file:  data/data_news_label.h5 /scratch2/goodnewsdata/data_news_image.h5
    read 489229 images of size 3x256x256
    max sequence length in data is 31
    assigned 445433 images to split train
    assigned 19376 images to split val
    assigned 24420 images to split test
    WARNING:tensorflow:From train.py:49: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
    
    THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error
    Traceback (most recent call last):
      File "train.py", line 280, in <module>
        train(opt)
      File "train.py", line 81, in train
        cnn_model.cuda()
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in cuda
        return self._apply(lambda t: t.cuda(device))
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 190, in _apply
        module._apply(fn)
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 196, in _apply
        param.data = fn(param.data)
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in <lambda>
        return self._apply(lambda t: t.cuda(device))
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
        torch._C._cuda_init()
    RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp:51
    Closing remaining open files:data/data_news_label.h5...done/scratch2/goodnewsdata/data_news_image.h5...done
    
    opened by monajalal 0
  • FileNotFoundError: [Errno 2] No such file or directory: '../resized/resized/51cf8b8d7e0d9c0839d2d1b1_1.jpg'

    FileNotFoundError: [Errno 2] No such file or directory: '../resized/resized/51cf8b8d7e0d9c0839d2d1b1_1.jpg'

    [jalal@goku scripts]$ python prepro_images.py 
    parsed input parameters:
    {
      "input_json": "../data/data_news.json",
      "output_h5": "../data/data_news",
      "images_root": "../resized/"
    }
    processing 0/489229 (0.00% done)
    processing 1000/489229 (0.20% done)
    processing 2000/489229 (0.41% done)
    processing 3000/489229 (0.61% done)
    processing 4000/489229 (0.82% done)
    processing 5000/489229 (1.02% done)
    processing 6000/489229 (1.23% done)
    processing 7000/489229 (1.43% done)
    processing 8000/489229 (1.64% done)
    processing 9000/489229 (1.84% done)
    processing 10000/489229 (2.04% done)
    processing 11000/489229 (2.25% done)
    processing 12000/489229 (2.45% done)
    processing 13000/489229 (2.66% done)
    processing 14000/489229 (2.86% done)
    processing 15000/489229 (3.07% done)
    processing 16000/489229 (3.27% done)
    processing 17000/489229 (3.47% done)
    processing 18000/489229 (3.68% done)
    processing 19000/489229 (3.88% done)
    processing 20000/489229 (4.09% done)
    processing 21000/489229 (4.29% done)
    Traceback (most recent call last):
      File "prepro_images.py", line 87, in <module>
        main(params)
      File "prepro_images.py", line 53, in main
        I = imread(os.path.join(params['images_root'], img['file_path']))
      File "/home/grad3/jalal/.local/lib/python3.6/site-packages/numpy/lib/utils.py", line 101, in newfunc
        return func(*args, **kwds)
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/scipy/misc/pilutil.py", line 164, in imread
        im = Image.open(name)
      File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/PIL/Image.py", line 2766, in open
        fp = builtins.open(filename, "rb")
    FileNotFoundError: [Errno 2] No such file or directory: '../resized/resized/51cf8b8d7e0d9c0839d2d1b1_1.jpg'
    

    I have the images here: /scratch3/GoodNews/resized/resized

    What am I doing wrong? The file doesn't exist in the images

    [jalal@goku resized]$ ls 51cf8b8d7e0d9c0839d2d1b1_1.jpg
    ls: cannot access 51cf8b8d7e0d9c0839d2d1b1_1.jpg: No such file or directory
    
    
    opened by monajalal 0
  • Data does not match

    Data does not match

    I'm currently trying this code but seems like provided dataset has some problems.
    The number of data which is specified in the paper does not match with what i get by following the guide.

    opened by ChanghunHyun 0
  • RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 32 and 1 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

    RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 32 and 1 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

    Have attached traceback, am not sure what is calling this error.

    Traceback (most recent call last): File "train.py", line 283, in train(opt) File "train.py", line 173, in train out = model(fc_feats, att_feats, labels, sen_embed) File "/im2p/im2penv/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/im2p/GoodNews-master/models/OldModel.py", line 100, in forward else: output, state = self.core(xt, fc_feats, att_feats, state, sen_embed) File "/im2p/im2penv/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/im2p/GoodNews-master/models/OldModel.py", line 398, in forward output, state = self.rnn(torch.cat([xt, att_res, att_res_sen.float()], 1).unsqueeze(0), state) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 32 and 1 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:689 Closing remaining open files:./data/data_news_image.h5...done./data/data_news_label.h5...done

    opened by devdev999 0
  • ValueError: u'59248f9895d0e024b5872783' is not in list when training

    ValueError: u'59248f9895d0e024b5872783' is not in list when training

    I encounter this when trying to train:

    Traceback (most recent call last): File "train.py", line 283, in train(opt) File "train.py", line 139, in train data = loader.get_batch('train') File "/im2p/GoodNews-master/dataloader.py", line 261, in get_batch sen_ixs = [self.sen_embed_keys.index(key) for key in keys] ValueError: u'59248f9895d0e024b5872783' is not in list Closing remaining open files:./data/data_news_image.h5...done./data/data_news_label.h5...done

    opened by devdev999 3
  • Picture Error

    Picture Error

    Hi and thanks for providing this repo, when I use "prepro_images.py", the results show lacking of picture 58cedcaa95d0e02489b91f23_4.jpg, the picture name is show in data_news.json, which is not in img_urls_all.json

    ,

    opened by ChenTao2017110 0
Owner
null
Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020)

Diverse Image Captioning with Context-Object Split Latent Spaces This repository is the PyTorch implementation of the paper: Diverse Image Captioning

Visual Inference Lab @TU Darmstadt 34 Nov 21, 2022
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

MuVER This repo contains the code and pre-trained model for our EMNLP 2021 paper: MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity

null 24 May 30, 2022
[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

CLNER The code is for our ACL-IJCNLP 2021 paper: Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning CLNER is a

null 71 Dec 8, 2022
Syntax-Aware Action Targeting for Video Captioning

Syntax-Aware Action Targeting for Video Captioning Code for SAAT from "Syntax-Aware Action Targeting for Video Captioning" (Accepted to CVPR 2020). Th

null 59 Oct 13, 2022
AI-based, context-driven network device ranking

Batea A batea is a large shallow pan of wood or iron traditionally used by gold prospectors for washing sand and gravel to recover gold nuggets. Batea

Secureworks Taegis VDR 269 Nov 26, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

Jia Research Lab 137 Dec 14, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

DV Lab 137 Dec 14, 2022
IAUnet: Global Context-Aware Feature Learning for Person Re-Identification

IAUnet This repository contains the code for the paper: IAUnet: Global Context-Aware Feature Learning for Person Re-Identification Ruibing Hou, Bingpe

null 30 Jul 14, 2022
Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

UACANet: Uncertainty Aware Context Attention for Polyp Segmentation Official pytorch implementation of UACANet: Uncertainty Aware Context Attention fo

Taehun Kim 85 Dec 14, 2022
CAPRI: Context-Aware Interpretable Point-of-Interest Recommendation Framework

CAPRI: Context-Aware Interpretable Point-of-Interest Recommendation Framework This repository contains a framework for Recommender Systems (RecSys), a

RecSys Lab 8 Jul 3, 2022
Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Namt0d 9 Dec 7, 2022
Fast and Context-Aware Framework for Space-Time Video Super-Resolution (VCIP 2021)

Fast and Context-Aware Framework for Space-Time Video Super-Resolution Preparation Dependencies PyTorch 1.2.0 CUDA 10.0 DCNv2 cd model/DCNv2 bash make

Xueheng Zhang 1 Mar 29, 2022
Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation by Lukas Hoyer, Dengxin Dai, and Luc Van Gool [Arxiv] [Paper] Overview Unsup

Lukas Hoyer 149 Dec 28, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

null 49 Nov 23, 2022
InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images

InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images Hong Wang, Yuexiang Li, Haimiao Zhang, Deyu Men

Hong Wang 4 Dec 27, 2022
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 170.1k Jan 4, 2023
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 170.1k Jan 5, 2023
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 153.2k Feb 13, 2021