GoodNews Everyone! Context driven entity aware captioning for news images

Last update: Dec 19, 2022

Related tags

Deep Learning GoodNews

Overview

This is the code for a CVPR 2019 paper, called GoodNews Everyone! Context driven entity aware captioning for news images. Enjoy!

Model preview:

Huge Thanks goes to New York Times API for providing such a service for FREE!

Another Thanks to @ruotianluo for providing the captioning code.

Dependencies/Requirements:

pytorch==1.0.0
spacy==2.0.11
h5py==2.7.0
bs4==4.5.3
joblib==0.12.2
nltk==3.2.3
tqdm==4.19.5
urllib2==2.7
goose==1.0.25
urlparse
unidecode

Introduction

We took the first steps to move the captioning systems to interpretation (see the paper for more detail). To this end, we have used New York Times API to retrieve the articles, images and captions.

The structure of this repo is as follows:

Getting the data
Cleaning and formating the data
How to train models

Get the data

You have 3 options to get the data.

Images only

If you want to download the images only and directly start working on the same dataset as ours, then download the cleaned version of the dataset without images: article+caption.json and put it to data/ folder and download the img_urls.json and put it in the get_data/get_images_only/ folder.

Then run

python get_images.py --num_thread 16

Then, you will get the images. After that move to Clean and Format Data section.

PS: I have recieved numerous emails regarding some of the images not present/broken in the img_urls.json. Which is why I decided to put the images on the drive to download in the name of open science. Download all images

Images + articles

If you would like the get the raw version of the article and captions to do your own cleaning and processing, no worries! First download the article_urls and go to folder get_data/with_article_urls/ and run

python get_data_with_urls.py --num_thread 16
python combine_dataset.py

This will get you the raw version of the caption, articles and also the images. After that move to Clean and Format Data section.

I want more!

As you know, New York Times is huge. Their articles starts from 1881 (It is crazy!) until well today. So in case you want to get ALL the data or expand the data to more years, then first step is go to New York Times API and get an API key. All you have to do is just sign up for the API key.

Once you have the key go to folder get_data/with_api/ and run

python retrieve_all_urls.py --api-key XXXX --start_year XXX --end_year XXX

This is for getting the article urls and then saving in the format of month-year. Once you have the all urls from the API, then you run

python get_data_api.py
python combine_dataset.py

get_data_api.py retrieves the articles, captions and images. combine_dataset.py combines yearly data into one file after removing data points if they have corrupt image, empty articles or empty captions. After that move to Clean and Format Data section.

Small Note

I also provide the links to images and their data splits (train, val, test). Even though I always use random seed to decide the split, just in case If the GODS meddles with the random seed, here is the link to a json where you can find each image and its split: img_splits.json

Clean and Format the Data

Now that we have the data, it is time to clean, preprocess and format the data.

Preprocess

When you reach this part, you must have captioning_dataset.json in your data/ folder.

Captions

This part is for cleaning the captions (tokenizing, removing non-ascii characters, etc.), splitting train, val, and test and creating anonymize captions.

In other words, we change the caption "Alber Einstein taught in Princeton in 1926" to "PERSON_ taught in ORGANIZATION_ in DATE_." Move to preprocess/ folder and run

python clean_captions.py

Resize Images

To resize the images to 256x256:

python resize.py --root XXXX --img_size 256

Articles

Get the article format that is needed for the encoding methods by running: create_article_set.py

python create_article_set.py

Format

Now to create H5 file for captions, images and articles, just need to go to scripts/ folder and run in order

python prepro_labels.py --max_length 31 --word_count_threshold 4
python prepro_images.py

We proposed 3 different article encoding method. You can download each of encoded article methods, articles_full_avg_, articles_full_wavg, articles_full_TBB.

Or you can use the code to obtain them:

python prepro_articles_avg.py
python prepro_articles_wavg.py
python prepro_articles_tbb.py

Train

Finally we are ready to train. Magical words are:

python train.py --cnn_weight [YOUR HOME DIRECTORY]/.torch/resnet152-b121ed2d.pth

You can check the opt.py for changing a lot of the options such dimension size, different models, hyperparameters, etc.

Evaluate

After you train your models, you can get the score according commonly used metrics: Bleu, Cider, Spice, Rouge, Meteor. Be sure to specify model_path, cnn_model_path, infos_path and sen_embed_path when runing eval.py. eval.py is usually used in training but it is necessary to run it to get the insertion.

Insertion

Last but not least insert.py. After you run eval.py, it will produce you a json file with the ids and their template captions. To fill the correct named entity, you have to run insert.py:

python insert.py --output [XXX] --dump [True/False] --insertion_method ['ctx', 'att', 'rand']

PS: I have been requested to provide model's output, so I thought it would be best to share it with everyone. Model Output In this folder, you have:

test.json: Test set with raw and template version of the caption.

article.json: Article sentences which is needed in the insert.py.

w/o article folder: All the models output on template captions, without articles.

with article folder: Our models output in the paper with sentence attention(sen_att) and image attention(vis_att), provided in the json. Hope this is helpful to more of you.

Conclusion

Thank you and sorry for the bugs!

Comments

After I run clean_captions.py the news_dataset.json file is not created

I don't get any error and I see print of captions. However, only these are created (val.json is also not created):

Vice President Joseph R. Biden Jr. in March at a Jordanian-American military training center in Zarqa, Jordan.
The funeral in Rimoun, Jordan, for Anwar Abu Zaid, a police captain who was killed after he attacked a police training center in November. American and Jordanian officials said they believed that the weapons he used had been meant for a program to train Syrian rebels.
Ambulances leaving the police training center where Captain Abu Zaid gunned down five people, including two American contractors.
Jeremy Beck and Rebecca Noelle Brinkley in the Mint Theater production of “Hindle Wakes.”
Priscilla Chan and Mark Zuckerberg say their charity plan is to "advance human potential and promote equality in areas such as health, education, scientific research and energy."
The official Rio 2016mona@mona:~/research/GoodNews/prepocess$ ls ../data
total 1021M
-rw-rw-r--  1 mona mona   71 Sep  7 12:28 .gitignore
-rw-rw-r--  1 mona mona 985M Sep  7 12:29 captioning_dataset.json
drwxrwxr-x 12 mona mona 4.0K Sep  7 12:47 ..
-rw-rw-r--  1 mona mona  36M Sep  7 21:21 article_urls.json
drwxrwxr-x  2 mona mona 4.0K Sep  7 21:22 .
-rw-rw-r--  1 mona mona    0 Sep  9 10:23 test.json

Do you know what's the reason and how to fix it? I have used absolute path as well but no luck!

    with open("/home/mona/research/GoodNews/data/captioning_dataset.json", "rb") as f:
        captioning_dataset = json.load(f)

    for k, anns in tqdm.tqdm(captioning_dataset.items()):

        for ix, img in anns['images'].items():
            try:
                split = get_split()

                #         import ipdb; ipdb.set_trace()
                img = preprocess_sentence(img)
                template, full = NER(' '.join(img))
                if len(' '.join(template)) != 0:
                    news_data.append({'filename': k + '_' + ix + '.jpg', 'filepath': 'resized', 'cocoid': counter,
                                      'imgid': k + '_' + ix, 'sentences': [], 'sentences_full': [],
                                      #                               'sentences_article':[],
                                      'split': split})
                    news_data[counter]['sentences'].append(
                        {'imgid': counter, 'raw': ' '.join(template), 'tokens': template})
                    news_data[counter]['sentences_full'].append(
                        {'imgid': counter, 'raw': ' '.join(full), 'tokens': full})
                    counter += 1
            except:
                print(img)
    split_to_ix = {i:n['split'] for i, n in enumerate(news_data)}
    # train = [news_data[k] for k, v in split_to_ix.items() if v =='train']
    val = [news_data[k] for k, v in split_to_ix.items() if v =='val']
    test = [news_data[k] for k, v in split_to_ix.items() if v =='test']
    with open("/home/mona/research/GoodNews/data/test.json", "wb") as f:
        json.dump(test, f)
    with open("/home/mona/research/GoodNews/data/val.json", "wb") as f:

opened by monajalal 2

FileNotFoundError: [Errno 2] No such file or directory: 'save/show_attend_tell/model.pth'

For

(/scratch2/sjn/pyta) [jalal@goku GoodNews]$ python train.py --cnn_weight data/resnet152-b121ed2d.pth

I get

(/scratch2/sjn/pyta) [jalal@goku GoodNews]$ tmux attach

image 127420: PERSON_ and PERSON_ in GPE_ in GPE_
image 127431: PERSON_ left and PERSON_ in GPE_
image 127444: PERSON_ in GPE_
image 127447: PERSON_ in DATE_
image 127472: PERSON_ in GPE_ on DATE_
image 127497: PERSON_ a NORP_ NORP_ in GPE_ on DATE_
image 127535: PERSON_ in DATE_
image 127536: PERSON_ and PERSON_ in GPE_
image 127546: PERSON_ in GPE_ on DATE_
image 127584: PERSON_ a NORP_ leader of the ORG_ in GPE_
image 127596: A UNK UNK UNK UNK UNK
evaluating validation preformance... 5023/5000 (5.082450)
{'testlen': 31857, 'reflen': 75785, 'guess': [31857, 26857, 22238, 17619], 'correct': [10185, 2230, 512, 159]}
ratio: 0.42036022959688035
Bleu_1:  0.08051972435966352
Bleu_2:  0.04103438062512367
Bleu_3:  0.02137332258808613
Bleu_4:  0.012205189537708628
ROUGE_L: 0.1975560912228822
CIDEr:   0.1773307729244918
Traceback (most recent call last):
  File "train.py", line 280, in <module>
    train(opt)
  File "train.py", line 238, in train
    torch.save(model.state_dict(), checkpoint_path)
  File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 361, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 229, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/scratch2/sjn/pyta/lib/python3.7/site-packages/torch/serialization.py", line 210, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'save/show_attend_tell/model.pth'
Closing remaining open files:data/data_news_label.h5...done/scratch2/goodnewsdata/data_news_image.h5...done

I'll update here when I find a solution.

opened by monajalal 1

Why running clean_captions.py returns empty files for val.json, test.json, and news_dataset.json?

[jalal@goku GoodNews]$ cd data/
[jalal@goku data]$ ls
total 9270700
      4 -rw-r--r--.  1 jalal cs-grad         71 Sep  9 11:25 .gitignore
  36744 -rw-r--r--.  1 jalal cs-grad   37622426 Sep  9 11:27 article_urls.json
1008000 -rw-r--r--.  1 jalal cs-grad 1032190641 Sep  9 11:27 captioning_dataset.json
8225936 -rw-r--r--.  1 jalal cs-grad 8423357695 Sep  9 11:29 resized.tar.gz
      4 drwxr-xr-x. 12 jalal cs-grad       4096 Sep  9 12:12 ..
      4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 val.json
      4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 test.json
      4 -rw-r--r--.  1 jalal cs-grad          2 Sep  9 13:50 news_dataset.json
      0 drwxr-xr-x.  2 jalal cs-grad        192 Sep  9 13:50 .

The code is:


[jalal@goku data]$ cd ../prepocess/
[jalal@goku prepocess]$ cat clean_captions.py 
import json
import nltk
import spacy
import numpy as np
import tqdm
import unidecode
from bs4 import BeautifulSoup
import re
import unicodedata
from itertools import groupby

def remove_non_ascii(words):
    """Remove non-ASCII characters from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = unicodedata.normalize('NFKD', word).encode('ascii', 'ignore').decode('utf-8', 'ignore')
        new_words.append(new_word)
    return new_words

def to_lowercase(words):
    """Convert all characters to lowercase from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = word.lower()
        new_words.append(new_word)
    return new_words

def remove_punctuation(words):
    """Remove punctuation from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = re.sub(r'[^\w\s]', '', word)
        if new_word != '':
            new_words.append(new_word)
    return new_words

def replace_numbers(words):
    """Replace all interger occurrences in list of tokenized words with textual representation"""
    p = inflect.engine()
    new_words = []
    for word in words:
        if word.isdigit():
            new_word = p.number_to_words(word)
            new_words.append(new_word)
        else:
            new_words.append(word)
    return new_words

def remove_stopwords(words):
    """Remove stop words from list of tokenized words"""
    new_words = []
    for word in words:
        if word not in stopwords.words('english'):
            new_words.append(word)
    return new_words

# def stem_words(words):
#     """Stem words in list of tokenized words"""
#     stemmer = LancasterStemmer()
#     stems = []
#     for word in words:
#         stem = stemmer.stem(word)
#         stems.append(stem)
#     return stems
#
# def lemmatize_verbs(words):
#     """Lemmatize verbs in list of tokenized words"""
#     lemmatizer = WordNetLemmatizer()
#     lemmas = []
#     for word in words:
#         lemma = lemmatizer.lemmatize(word, pos='v')
#         lemmas.append(lemma)
#     return lemmas

def normalize(words):
    words = remove_non_ascii(words)
#     words = to_lowercase(words)
    words = remove_punctuation(words)
#     words = replace_numbers(words)
#     words = remove_stopwords(words)
    return words

def strip_html(text):
    soup = BeautifulSoup(text, "html.parser")
    return soup.get_text()

def remove_between_square_brackets(text):
    return re.sub('\[[^]]*\]', '', text)

def denoise_text(text):
    text = strip_html(text)
    text = remove_between_square_brackets(text)
    return text
def replace_contractions(text):
    """Replace contractions in string of text"""
    return contractions.fix(text)

def preprocess_sentence(sen):
    sen = sen.strip()
#     sen = re.sub(uri_re, "", sen)
    sen = sen.encode('ascii',errors='ignore')
    sen = unidecode.unidecode(sen)
    sen = denoise_text(sen)
    # sen = replace_contractions(sen)
    sen = nltk.tokenize.word_tokenize(sen)
    sen = normalize([unicode(s) for s in sen])
#     sen = normalize(unicode(sen))
#     return sen
    return sen
#     tokenized = nltk.tokenize.word_tokenize(temp)
#     final = normalize(unicode(tokenized))
#     return ''.join(final)

# def NER(sen):
#     doc = nlp(unicode(sen))
#     return [d.ent_iob_+'-'+d.ent_type_ if d.ent_iob_ != 'O' else d.text for d in doc ], [d.text for d in doc]
def NER(sen):
    doc = nlp(unicode(sen))
#     text = doc.text
#     for ent in doc.ents:
#         text = text.replace(ent.text, ent.label_+'_')
    tokens = [d.text for d in doc]
#     [ent.merge(ent.root.tag_, ent.text, ent.label_) for ent in doc.ents]
#     return compact([d.ent_iob_+'-'+d.ent_type_ if d.ent_iob_ != 'O' else d.text for d in doc ]), tokens
#     return text, tokens
    temp = [d.ent_type_+'_' if d.ent_iob_ != 'O' else d.text for d in doc]
    return [x[0] for x in groupby(temp)], tokens

def get_split():
    rand = np.random.uniform()
    if rand > 0.95:
        split = 'test'
    #             test_num += 1
    elif rand > 0.91 and rand < 0.95:
        split = 'val'
    #         val_num += 1
    else:
        split = 'train'
    #         train_num += 1
    return split

if __name__ == '__main__':
    np.random.seed(42)
    nlp = spacy.load('en', disable=['parser', 'tagger'])
    print('Loading spacy modules.')
    news_data = []
    counter = 0
    test_num, val_num, train_num = 0, 0, 0

    print('Loading the json.')
    with open("../data/captioning_dataset.json", "rb") as f:
        captioning_dataset = json.load(f)

    for k, anns in tqdm.tqdm(captioning_dataset.items()):

        for ix, img in anns['images'].items():
            try:
                split = get_split()

                #         import ipdb; ipdb.set_trace()
                img = preprocess_sentence(img)
                template, full = NER(' '.join(img))
                if len(' '.join(template)) != 0:
                    news_data.append({'filename': k + '_' + ix + '.jpg', 'filepath': 'resized', 'cocoid': counter,
                                      'imgid': k + '_' + ix, 'sentences': [], 'sentences_full': [],
                                      #                               'sentences_article':[],
                                      'split': split})
                    news_data[counter]['sentences'].append(
                        {'imgid': counter, 'raw': ' '.join(template), 'tokens': template})
                    news_data[counter]['sentences_full'].append(
                        {'imgid': counter, 'raw': ' '.join(full), 'tokens': full})
                    counter += 1
            except:
                print(img)
    split_to_ix = {i:n['split'] for i, n in enumerate(news_data)}
    # train = [news_data[k] for k, v in split_to_ix.items() if v =='train']
    val = [news_data[k] for k, v in split_to_ix.items() if v =='val']
    test = [news_data[k] for k, v in split_to_ix.items() if v =='test']
    with open("../data/test.json", "w") as f:
        json.dump(test, f)
    with open("../data/val.json", "w") as f:
        json.dump(val, f)
    with open("../data/news_dataset.json", "w") as f:
        json.dump(news_data, f)

opened by monajalal 1

Where can I download news_dataset.json ?

(base) mona@mona:~/research/GoodNews/prepocess$ python create_article_set.py
Traceback (most recent call last):
  File "create_article_set.py", line 11, in <module>
    with open("../data/news_dataset.json", "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/news_dataset.json'

opened by monajalal 1

where should we place resized.tar.gz?

(base) mona@mona:~/research/GoodNews/images$ ls
total 7.9G
-rw-rw-r--  1 mona mona   71 Sep  7 12:28 .gitignore
drwxrwxr-x 12 mona mona 4.0K Sep  7 12:28 ..
-rw-rw-r--  1 mona mona 7.9G Sep  7 12:35 resized.tar.gz
drwxrwxr-x  3 mona mona 4.0K Sep  7 12:37 .
drwxrwxr-x  2 mona mona  15M Sep  7 12:41 resized

I have a tree like this: tree_GoodNews

opened by monajalal 1

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error

Do you know how I could fix this?

[jalal@goku GoodNews]$ python train.py --cnn_weight data/resnet152-b121ed2d.pth 
DataLoader loading json file:  data/data_news.json
vocab size is  37200
DataLoader loading h5 file:  data/data_news_label.h5 /scratch2/goodnewsdata/data_news_image.h5
read 489229 images of size 3x256x256
max sequence length in data is 31
assigned 445433 images to split train
assigned 19376 images to split val
assigned 24420 images to split test
WARNING:tensorflow:From train.py:49: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error
Traceback (most recent call last):
  File "train.py", line 280, in <module>
    train(opt)
  File "train.py", line 81, in train
    cnn_model.cuda()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 190, in _apply
    module._apply(fn)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 196, in _apply
    param.data = fn(param.data)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp:51
Closing remaining open files:data/data_news_label.h5...done/scratch2/goodnewsdata/data_news_image.h5...done

opened by monajalal 0

FileNotFoundError: [Errno 2] No such file or directory: '../resized/resized/51cf8b8d7e0d9c0839d2d1b1_1.jpg'

[jalal@goku scripts]$ python prepro_images.py 
parsed input parameters:
{
  "input_json": "../data/data_news.json",
  "output_h5": "../data/data_news",
  "images_root": "../resized/"
}
processing 0/489229 (0.00% done)
processing 1000/489229 (0.20% done)
processing 2000/489229 (0.41% done)
processing 3000/489229 (0.61% done)
processing 4000/489229 (0.82% done)
processing 5000/489229 (1.02% done)
processing 6000/489229 (1.23% done)
processing 7000/489229 (1.43% done)
processing 8000/489229 (1.64% done)
processing 9000/489229 (1.84% done)
processing 10000/489229 (2.04% done)
processing 11000/489229 (2.25% done)
processing 12000/489229 (2.45% done)
processing 13000/489229 (2.66% done)
processing 14000/489229 (2.86% done)
processing 15000/489229 (3.07% done)
processing 16000/489229 (3.27% done)
processing 17000/489229 (3.47% done)
processing 18000/489229 (3.68% done)
processing 19000/489229 (3.88% done)
processing 20000/489229 (4.09% done)
processing 21000/489229 (4.29% done)
Traceback (most recent call last):
  File "prepro_images.py", line 87, in <module>
    main(params)
  File "prepro_images.py", line 53, in main
    I = imread(os.path.join(params['images_root'], img['file_path']))
  File "/home/grad3/jalal/.local/lib/python3.6/site-packages/numpy/lib/utils.py", line 101, in newfunc
    return func(*args, **kwds)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/scipy/misc/pilutil.py", line 164, in imread
    im = Image.open(name)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/PIL/Image.py", line 2766, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '../resized/resized/51cf8b8d7e0d9c0839d2d1b1_1.jpg'

I have the images here: /scratch3/GoodNews/resized/resized

What am I doing wrong? The file doesn't exist in the images

[jalal@goku resized]$ ls 51cf8b8d7e0d9c0839d2d1b1_1.jpg
ls: cannot access 51cf8b8d7e0d9c0839d2d1b1_1.jpg: No such file or directory

opened by monajalal 0

Data does not match

I'm currently trying this code but seems like provided dataset has some problems.
The number of data which is specified in the paper does not match with what i get by following the guide.

opened by ChanghunHyun 0
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 32 and 1 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

Have attached traceback, am not sure what is calling this error.

Traceback (most recent call last): File "train.py", line 283, in train(opt) File "train.py", line 173, in train out = model(fc_feats, att_feats, labels, sen_embed) File "/im2p/im2penv/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/im2p/GoodNews-master/models/OldModel.py", line 100, in forward else: output, state = self.core(xt, fc_feats, att_feats, state, sen_embed) File "/im2p/im2penv/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/im2p/GoodNews-master/models/OldModel.py", line 398, in forward output, state = self.rnn(torch.cat([xt, att_res, att_res_sen.float()], 1).unsqueeze(0), state) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 32 and 1 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:689 Closing remaining open files:./data/data_news_image.h5...done./data/data_news_label.h5...done

opened by devdev999 0
ValueError: u'59248f9895d0e024b5872783' is not in list when training

I encounter this when trying to train:

Traceback (most recent call last): File "train.py", line 283, in train(opt) File "train.py", line 139, in train data = loader.get_batch('train') File "/im2p/GoodNews-master/dataloader.py", line 261, in get_batch sen_ixs = [self.sen_embed_keys.index(key) for key in keys] ValueError: u'59248f9895d0e024b5872783' is not in list Closing remaining open files:./data/data_news_image.h5...done./data/data_news_label.h5...done

opened by devdev999 3
Picture Error

Hi and thanks for providing this repo, when I use "prepro_images.py", the results show lacking of picture 58cedcaa95d0e02489b91f23_4.jpg, the picture name is show in data_news.json, which is not in img_urls_all.json

,

opened by ChenTao2017110 0

Owner

GitHub

Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020)

Diverse Image Captioning with Context-Object Split Latent Spaces This repository is the PyTorch implementation of the paper: Diverse Image Captioning

34 Nov 21, 2022

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

3 Nov 30, 2021

[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

MuVER This repo contains the code and pre-trained model for our EMNLP 2021 paper: MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity

24 May 30, 2022

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

CLNER The code is for our ACL-IJCNLP 2021 paper: Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning CLNER is a

71 Dec 8, 2022

Syntax-Aware Action Targeting for Video Captioning

Syntax-Aware Action Targeting for Video Captioning Code for SAAT from "Syntax-Aware Action Targeting for Video Captioning" (Accepted to CVPR 2020). Th

59 Oct 13, 2022

AI-based, context-driven network device ranking

Batea A batea is a large shallow pan of wood or iron traditionally used by gold prospectors for washing sand and gravel to recover gold nuggets. Batea

269 Nov 26, 2022

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

137 Dec 14, 2022

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

137 Dec 14, 2022

IAUnet: Global Context-Aware Feature Learning for Person Re-Identification

IAUnet This repository contains the code for the paper: IAUnet: Global Context-Aware Feature Learning for Person Re-Identification Ruibing Hou, Bingpe

30 Jul 14, 2022

Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

UACANet: Uncertainty Aware Context Attention for Polyp Segmentation Official pytorch implementation of UACANet: Uncertainty Aware Context Attention fo

85 Dec 14, 2022

CAPRI: Context-Aware Interpretable Point-of-Interest Recommendation Framework

CAPRI: Context-Aware Interpretable Point-of-Interest Recommendation Framework This repository contains a framework for Recommender Systems (RecSys), a

8 Jul 3, 2022

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

9 Dec 7, 2022

Fast and Context-Aware Framework for Space-Time Video Super-Resolution (VCIP 2021)

Fast and Context-Aware Framework for Space-Time Video Super-Resolution Preparation Dependencies PyTorch 1.2.0 CUDA 10.0 DCNv2 cd model/DCNv2 bash make

1 Mar 29, 2022

Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation by Lukas Hoyer, Dengxin Dai, and Luc Van Gool [Arxiv] [Paper] Overview Unsup

149 Dec 28, 2022

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

49 Nov 23, 2022

GoodNews Everyone! Context driven entity aware captioning for news images

Related tags

Overview

Introduction

Get the data

Images only

Images + articles

I want more!

Small Note

Clean and Format the Data

Preprocess

Captions

Resize Images

Articles

Format

Train

Evaluate

Insertion

Conclusion

Comments

Owner

Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020)

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

Syntax-Aware Action Targeting for Video Captioning

AI-based, context-driven network device ranking

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

IAUnet: Global Context-Aware Feature Learning for Person Re-Identification

Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

CAPRI: Context-Aware Interpretable Point-of-Interest Recommendation Framework

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Fast and Context-Aware Framework for Space-Time Video Super-Resolution (VCIP 2021)

Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images

An Open Source Machine Learning Framework for Everyone

An Open Source Machine Learning Framework for Everyone

An Open Source Machine Learning Framework for Everyone