Hello, I'm a little surprise by this code :

if self.alpha_c > 0:
            alphas = tf.transpose(tf.pack(alpha_list), (1, 0, 2))     # (N, T, L)
            alphas_all = tf.reduce_sum(alphas, 1)      # (N, L)
            alpha_reg = self.alpha_c * tf.reduce_sum((16./196 - alphas_all) ** 2)     

I dont understand the calcul to get alpha_reg. As i understand, alphas_all is the sum of alphas for every spatial location. In the paper, we want this to be close from 1, shouldnt then it be something like :

alpha_reg = self.alpha_c * tf.reduce_sum(tf.abs(1-alphas_all))    

Thanks in advance for the clarification.

Hello, yunjey, thanks a lot for your nice work! I got a problem when I run 'train.py', at the line of 'scores = evaluate(data_path='./data', split='val', get_scores=True)', raised a error like this: I'm new to this field, I'll very appreciate it if you can give some advice. Thank you!

When I run train.py,i got the problem says Traceback (most recent call last): File "train.py", line 25, in <module> main() File "train.py", line 8, in main data = load_coco_data(data_path='./data', split='train') File "/home/sxm/bishe/tensorflow-git/show-attend-and-tell-tensorflow/core/utils.py", line 13, in load_coco_data data['features'] = hickle.load(os.path.join(data_path, '%s.features.hkl' %split)) File "/home/sxm/Tensorflow/local/lib/python2.7/site-packages/hickle.py", line 616, in load h5f = file_opener(fileobj) File "/home/sxm/Tensorflow/local/lib/python2.7/site-packages/hickle.py", line 154, in file_opener h5f = h5.File(filename, mode) File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 207, in __init__ fid = make_fid(name, mode, userblock_size, fapl) File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 79, in make_fid fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl) File "h5f.pyx", line 71, in h5py.h5f.open (h5py/h5f.c:1806) IOError: unable to open file (File accessibilty: Unable to open file) Is there anybody has the same problem?I don't know how to fix it...

when i run train.py ,i can't find rnn_cell,then i use tf.contrib.rnn instead. Being given TypeError: Expected int32, got list containing Tensors of type '_Message' instead.I dont' know how to fix

It seems that you tried to load all training data into a numpy array, which causes memory error in my case. (it works on my 3* titanX 12G GPU)

what's your GPU memory?

Loaded ./data/train/train.annotations.pkl..
Traceback (most recent call last):
  File "prepro.py", line 212, in <module>
    main()
  File "prepro.py", line 195, in main
    all_feats = np.ndarray([n_examples, 196, 512], dtype=np.float32)
MemoryError

How to reduce to number of training dataset?

Maybe you can help me run your excellent project.

After following all steps of the README, an error occurs on the last step:

To generate captions, visualize attention weights and evaluate the model, please see evaluate_model.ipynb

When trying to run: solver.test(data, split='val') an ipython notebook error message:

ValueError: Variable conv_featuresbatch_norm/beta already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 208, in variable
    caching_device=caching_device)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 177, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 244, in model_variable
    caching_device=caching_device, device=device)

My guess is that something is wrong with my tensorflow installation, which is weird because I tested it out with basic usage in iPython and everything works fine there.

Do you see what is causing my problem?

When i run the evaluate_model.ipynb,it just show that: InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1500] rhs shape= [1024] [[Node: save/Assign_8 = Assign[T=DT_FLOAT, _class=["loc:@initial_lstm/b_h"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](initial_lstm/b_h, save/restore_slice_8/_33)]] [[Node: save/restore_slice_15/_4 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_7_save/restore_slice_15", _device="/job:localhost/replica:0/task:0/cpu:0"]]

hey, I encountered a problem when i run python train.py. The full error is ValueError: Variable conv_featuresbatch_norm/beta/RMSProp/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

occured in this line(87): train_op = optimizer.apply_gradients(grads_and_vars=grads_and_vars)

Firstly, i appreciate you for your code, and i am a novice. when i trained a model, i is thinking how can i use it to test a image. so i think maybe you have this code.

Can anyone help me to solve this problem? In the evaluate_model.py , when I run the solver.test(data, split='val') . I got the error like this:

INFO:tensorflow:Restoring parameters from ./model/lstm/model-19

InvalidArgumentError Traceback (most recent call last) in () ----> 1 solver.test(data, split='val')

/home/guanlida/show-attend-and-tell-tensorflow/core/solver.pyc in test(self, data, split, attention_visualization, save_sampled_captions) 195 features_batch, image_files = sample_coco_minibatch(data, self.batch_size) 196 feed_dict = { self.model.features: features_batch } --> 197 alps, bts, sam_cap = sess.run([alphas, betas, sampled_captions], feed_dict) # (N, max_len, L), (N, max_len) 198 decoded = decode_captions(sam_cap, self.model.idx_to_word) 199

/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata) 787 try: 788 result = self._run(None, fetches, feed_dict, options_ptr, --> 789 run_metadata_ptr) 790 if run_metadata: 791 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata) 995 if final_fetches or final_targets: 996 results = self._do_run(handle, final_targets, final_fetches, --> 997 feed_dict_string, options, run_metadata) 998 else: 999 results = []

/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1130 if handle is None: 1131 return self._do_call(_run_fn, self._session, feed_dict, fetch_list, -> 1132 target_list, options, run_metadata) 1133 else: 1134 return self._do_call(_prun_fn, self._session, handle, feed_dict,

/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args) 1150 except KeyError: 1151 pass -> 1152 raise type(e)(node_def, op, message) 1153 1154 def _extend_graph(self):

InvalidArgumentError: transpose expects a vector of size 1. But input(1) is a vector of size 2 [[Node: transpose_1 = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](Squeeze, transpose_1/perm)]]

Caused by op u'transpose_1', defined at: File "/home/guanlida/anaconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/home/guanlida/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/guanlida/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start ioloop.IOLoop.instance().start() File "/home/guanlida/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start super(ZMQIOLoop, self).start() File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start handler_func(fd_obj, events) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events self._handle_recv() File "/home/guanlida/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv self._run_callback(callback, msg) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback callback(*args, **kwargs) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher return self.dispatch_shell(stream, msg) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell handler(stream, idents, msg) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request user_expressions, allow_stdin) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell interactivity=interactivity, compiler=compiler, result=result) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2827, in run_ast_nodes if self.run_code(code, result): File "/home/guanlida/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in solver.test(data, split='val') File "core/solver.py", line 188, in test alphas, betas, sampled_captions = self.model.build_sampler(max_len=20) # (N, max_len, L), (N, max_len) File "core/model.py", line 216, in build_sampler betas = tf.transpose(tf.squeeze(beta_list), (1, 0)) # (N, T) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1285, in transpose ret = gen_array_ops.transpose(a, perm, name=name) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3658, in transpose result = _op_def_lib.apply_op("Transpose", x=x, perm=perm, name=name) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/guanlida/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): transpose expects a vector of size 1. But input(1) is a vector of size 2 [[Node: transpose_1 = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](Squeeze, transpose_1/perm)]]

In this program, when I try using the whole dataset of MS COCO, I found that I got memory error on my GPU (NVIDIA TITAN XP), so I only use 10% of the train data to train the model and set the batch_size from 128 to 1 in the train.py. I completed the train.py and got the model in the lstm file. But when I want to run the evaluate_model finally, I change the dim_hidden from 1500 to 1024 and in the In[4] solver = CaptioningSolver(model, data, data, n_epochs=15, batch_size=1, update_rule='adam', learning_rate=0.0025, print_every=2000, save_every=1, image_path='./image/val2014_resized', pretrained_model=None, model_path='./model/lstm', test_model='./model/lstm/model-19', print_bleu=False, log_path='./log/') I changed the batch to 1 and the test_model from lstm3 to lstm(because there is not a file named lstm3). Those are my all change and I think when I changed the parameter, the shape or the dimension of the tensor has changed, so I got this error.

Error which I get when I run evaluate.ipynb code is, there is no "./data/val/val.features.hkl" file available.

Provided argument 'file_obj' does not appear to be a valid hickle file! (Unable to open file (unable to open file: name = './data/train/train.features.hkl', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0))

What should I do to solve this problem?

while working on this project I had experienced an IOError even after executing the prepro.py before train.py. Here's a google collaborator link of the project

https://colab.research.google.com/drive/1Zi_KsAmDp9UjB1I-qkWFry91IHwv7R_y?usp=sharing

Excuse me. When I try to run train.py i get the InvalidArgumentError

InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: attention_layer/w_1 [[{{node attention_layer/w_1}}]] (1) Invalid argument: Nan in summary histogram for: attention_layer/w_1 [[{{node attention_layer/w_1}}]] [[_arg_Placeholder_1_0_1/_161]] 0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_call(self, fn, *args) 1382 '\nsession_config.graph_options.rewrite_options.' 1383 'disable_meta_optimizer = True') -> 1384 raise type(e)(node_def, op, message) 1385 1386 def _extend_graph(self):

InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: attention_layer/w_1 [[node attention_layer/w_1 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] (1) Invalid argument: Nan in summary histogram for: attention_layer/w_1 [[node attention_layer/w_1 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[_arg_Placeholder_1_0_1/_161]] 0 successful operations. 0 derived errors ignored.

Original stack trace for 'attention_layer/w_1': File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 664, in launch_instance app.start() File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelapp.py", line 477, in start ioloop.IOLoop.instance().start() File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 888, in start handler_func(fd_obj, events) File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events self._handle_recv() File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv self._run_callback(callback, msg) File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 432, in run_callback callback(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher return self.dispatch_shell(stream, msg) File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell handler(stream, idents, msg) File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request user_expressions, allow_stdin) File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 196, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/usr/local/lib/python3.6/dist-packages/ipykernel/zmqshell.py", line 533, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell interactivity=interactivity, compiler=compiler, result=result) File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2828, in run_ast_nodes if self.run_code(code, result): File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 2, in solver.train() File "/content/drive/My Drive/Colab Notebooks/show-attend-and-tell-master/core/solver.py", line 97, in train tf.compat.v1.summary.histogram(var.op.name, var) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/summary/summary.py", line 179, in histogram tag=tag, values=values, name=scope) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_logging_ops.py", line 329, in histogram_summary "HistogramSummary", tag=tag, values=values, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

how to solve this problem ?

Although the code looks nice, I found this code isn't intact. If someone is trying to use it, please be careful.

1. maxlen: it shouldn't be the fixed-length as it defined as 15. In the paper, section 4.3 Training Proceduce, author uses the other method which is corresponding the homogeneous_data.py in the paper's code.
2. Beam search: this code doesn't implement beam search.

So far, I only found out these things and I decide to use the paper's original code instead of this. Anyway, I still appreciate the tensorflow attempt. The code is clear and beautiful. Thanks for sharing.