After I finished training, I ran the test.py file and reported the following error. I have not solved it, I hope I can get your help. @Kyubyong

` INFO:tensorflow:Restoring parameters from log/1/model_E04-L4.76-49080 INFO:root:# get hypotheses

Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[125,100] = 100 is not in [0, 100) [[{{node encoder/positional_encoding/embedding_lookup}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test.py", line 54, in hypotheses = get_hypotheses(num_test_batches, num_test_samples, sess, y_hat, m.idx2token) File "/mnt/data/xxxx/transformer-master/utils.py", line 144, in get_hypotheses h = sess.run(tensor) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[125,100] = 100 is not in [0, 100) [[node encoder/positional_encoding/embedding_lookup (defined at /mnt/data/xxxx/transformer-master/modules.py:295) ]]

Caused by op 'encoder/positional_encoding/embedding_lookup', defined at: File "test.py", line 41, in y_hat, _ = m.eval(xs, ys) File "/mnt/data/xxxx/transformer-master/model.py", line 163, in eval memory, sents1 = self.encode(xs, False) File "/mnt/data/xxxx/transformer-master/model.py", line 50, in encode enc += positional_encoding(enc, self.hp.maxlen1) File "/mnt/data/xxxx/transformer-master/modules.py", line 295, in positional_encoding outputs = tf.nn.embedding_lookup(position_enc, position_ind) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 316, in embedding_lookup transform_fn=None) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 133, in _embedding_lookup_and_transform result = _clip(array_ops.gather(params[0], ids, name=name), File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 3273, in gather return gen_array_ops.gather_v2(params, indices, axis, name=name) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3748, in gather_v2 "GatherV2", params=params, indices=indices, axis=axis, name=name) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): indices[125,100] = 100 is not in [0, 100) [[node encoder/positional_encoding/embedding_lookup (defined at /mnt/data/xxxx /transformer-master/modules.py:295) ]] `

Hi @Kyubyong as you can see the key masking code as following:

# Key Masking key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis=-1))) # (N, T_k) key_masks = tf.tile(key_masks, [num_heads, 1]) # (h*N, T_k) key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1]) # (h*N, T_q, T_k)

the params keys，is the sum of word_embedding and position_embedding. it means that even the word in a sentence is padding 0, as add postion_embedding to the word_embedding, there's no 0 vector for the final word_embedding. therefore, the key_masks must all be one, no zero! so I'm confused if the code works?

in train.py line 77 _, _eval_summaries = sess.run([eval_init_op, eval_summaries]) this line will execute eval_summaries with train data, do:

sess.run(eval_init_op)       
_eval_summaries = sess.run(eval_summaries)

instead

as the paper said or in some other implementation: self.w_qs = nn.Linear(d_model, n_head * d_k)
the data size is larger. but in this project, it is

       Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # (h*N, T_q, C/h) 
       K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # (h*N, T_k, C/h) 
       V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # (h*N, T_k, C/h)

it's like using partial of Q/K/V to form one head. Can anyone help to explain why it uses "split" and "concat" to get multi-head?

Thanks!

Hi, I have a question about the codes.

        # Linear projections
        Q = tf.layers.dense(queries, num_units, activation=tf.nn.relu) # (N, T_q, C)
        K = tf.layers.dense(keys, num_units, activation=tf.nn.relu) # (N, T_k, C)
        V = tf.layers.dense(keys, num_units, activation=tf.nn.relu) # (N, T_k, C)

Is there a mechanism that tied these three layers between multiple blocks? It seems their parameters are not shared between different blocks. What should i do to tie them?

Thanks!

Bumps tensorflow from 1.12.0 to 2.5.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I’m learning your code recently, but here I don’t know where to feed the data and train it.

I sincerely hope that you can reply！

`

summary_writer = tf.summary.FileWriter(hp.logdir, sess.graph)
sess.run(train_init_op)
total_steps = hp.num_epochs * num_train_batches
_gs = sess.run(global_step)
for i in tqdm(range(_gs, total_steps+1)):
    _, _gs, _summary = sess.run([train_op, global_step, train_summaries])
    epoch = math.ceil(_gs / num_train_batches)
    summary_writer.add_summary(_summary, _gs)

    if _gs and _gs % num_train_batches == 0:
        logging.info("epoch {} is done".format(epoch))
        _loss = sess.run(loss) # train loss

        logging.info("# test evaluation")
        _, _eval_summaries = sess.run([eval_init_op, eval_summaries])
        summary_writer.add_summary(_eval_summaries, _gs)

        logging.info("# get hypotheses")
        hypotheses = get_hypotheses(num_eval_batches, num_eval_samples, sess, y_hat, m.idx2token)

        logging.info("# write results")
        model_output = "iwslt2016_E%02dL%.2f" % (epoch, _loss)
        if not os.path.exists(hp.evaldir): os.makedirs(hp.evaldir)
        translation = os.path.join(hp.evaldir, model_output)
        with open(translation, 'w') as fout:
            fout.write("\n".join(hypotheses))

        logging.info("# calc bleu score and append it to translation")
        calc_bleu(hp.eval3, translation)

        logging.info("# save models")
        ckpt_name = os.path.join(hp.logdir, model_output)
        saver.save(sess, ckpt_name, global_step=_gs)
        logging.info("after training of {} epochs, {} has been saved.".format(epoch, ckpt_name))

        logging.info("# fall back to train mode")
        sess.run(train_init_op)
summary_writer.close()`

In function normalize(in modules.py), beta & gamma are set as variables. I don't know why should they be trainable. Couldn't I just use 0. & 1.?

def normalize(inputs,
              epsilon=1e-8,
              scope="ln",
              reuse=None):
    """Applies layer normalization.

    Args:
      inputs: A tensor with 2 or more dimensions, where the first dimension has
        `batch_size`.
      epsilon: A floating number. A very small number for preventing ZeroDivision Error.
      scope: Optional scope for `variable_scope`.
      reuse: Boolean, whether to reuse the weights of a previous layer
        by the same name.

    Returns:
      A tensor with the same shape and data dtype as `inputs`.
    """
    with tf.variable_scope(scope, reuse=reuse):
        inputs_shape = inputs.get_shape()
        params_shape = inputs_shape[-1:]

        mean, variance = tf.nn.moments(inputs, [-1], keep_dims=True)
        beta = tf.Variable(tf.zeros(params_shape))
        gamma = tf.Variable(tf.ones(params_shape))
        normalized = (inputs - mean) / ((variance + epsilon) ** (.5))
        outputs = gamma * normalized + beta

    return outputs

AttributeError: module 'tensorflow.contrib.linalg' has no attribute 'LinearOperatorTriL'

new tensorflow api

https://www.tensorflow.org/api_docs/python/tf/linalg/LinearOperatorLowerTriangular

Hi, I was just looking through the positional encoding code, and I see this line: https://github.com/Kyubyong/transformer/blob/37febcef69f6e4757a853df342c5c5fd299039b5/modules.py#L148

It looks wrong to me. Shouldn't it be something like the following?

 rad_block = tf.div(position_block, tf.pow(10000, tf.div(unit_block, num_units // 2))) 

I am trying to run for sinusoid PE, but it throws the following error.

File "train.py", line 51, in __init__ scope="enc_pe") File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 885, in binary_op_wrapper y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y") File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 836, in convert_to_tensor as_ref=False) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 926, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 774, in _TensorTensorConversionFunction (dtype.name, t.dtype.name, str(t))) ValueError: Tensor conversion requested dtype float32 for Tensor with dtype float64: 'Tensor("encoder/enc_pe/embedding_lookup:0", shape=(32, 49, 512), dtype=float64)'

Bumps tensorflow from 1.12.0 to 2.9.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

when I running the prepro.py, it is terminating at spm.SentencePieceTrainer.Train(train).And I get no any information. Has anyone encountered this problem? how to deal with it?

INFO:root:# Train a joint BPE model with sentencepiece

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Source Code:

    padding_num = -2 ** 32 + 1
    if type in ("k", "key", "keys"):
        key_masks = tf.to_float(key_masks)
        key_masks = tf.tile(key_masks, [tf.shape(inputs)[0] // tf.shape(key_masks)[0], 1]) # (h*N, seqlen)
        key_masks = tf.expand_dims(key_masks, 1)  # (h*N, 1, seqlen)
        outputs = inputs + key_masks * padding_num

I think the outputs should be:

    padding_num = -2 ** 32 + 1
    if type in ("k", "key", "keys"):
        key_masks = tf.to_float(key_masks) # (N, T_k)
        key_masks = tf.tile(key_masks, [tf.shape(inputs)[0] // tf.shape(key_masks)[0], 1]) # (h*N, seqlen)
        key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(key_masks)[1], 1]) # (h*N, T_q, seqlen)
        paddings = tf.ones_like(key_masks) * padding_num
        outputs = tf.where(tf.equal(key_masks, 0), paddings, inputs)

When I run train.py, the code shows xx running to a 5% break and shows the following problem: tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[27,100] = 100 is not in [0, 100)

Does anyone know how I can solve this problem?

1、为啥分成多头后又进行的concat？

Split and concat

    Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0)  # (h*N, T_q, d_model/h)
    K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0)  # (h*N, T_k, d_model/h)
    V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0)  # (h*N, T_k, d_model/h)

2、为啥下面的这个操作等价于多头的拼接 outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=2) # (N, T_q, d_model)