This is my model:

mid = (td.Map(td.Tensor((24,10)) >>
                td.ScopedLayer(convLayer) >>
                td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >>
         td.Fold(td.Concat() >>
                 td.Function(td.FC(128)), td.FromTensor(tf.zeros(128))))
model = ( mid  >>  td.Function(td.FC(1)))

(The ScopedLayer(convLayer) part is related to solved issue #14 )

I trained it using model: compiler = td.Compiler.create((model, td.Scalar())), and it works. However, now I want the result of mid for a set of values. I can do encod.eval(X[i]) for every piece of data, but that is painfully slow. However, if I try to create a new compiler:

newComp=td.Compiler.create((mid,))
Y=sess.run(mid,newComp.build_feed_dict(X))

I get: TypeError: block <td.Pipe> is already a child of <td.Pipe>

How can I solve this? eval on each entry will take a bit over 4 hours ...

Hello all,

I try to do this:

import tensorflow as tf
import tensorflow_fold as td

# Create RNN cells using the TensorFlow RNN library
char_cell = td.ScopedLayer(tf.contrib.rnn.BasicLSTMCell(num_units=16), 'char_cell')
word_cell = td.ScopedLayer(tf.contrib.rnn.BasicLSTMCell(num_units=32), 'word_cell')

# character LSTM converts a string to a word vector
char_lstm = (td.InputTransform(lambda s: [ord(c) for c in s]) >>
             td.Map(td.Scalar('int32') >>
                    td.Function(td.Embedding(128, 8))) >>
             td.RNN(char_cell))
# word LSTM converts a sequence of word vectors to a sentence vector.
# word_lstm = td.Map(char_lstm >> td.GetItem(1)) >> td.RNN(word_cell)

# ---------------------------------------------------------------------

sess = tf.InteractiveSession()

(
    td.Map(char_lstm >> td.GetItem(1)) >>
    td.RNN(word_cell)
).eval(["bon","ben"])

and i get:

(...)
/opt/conda/envs/python2/lib/python2.7/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.pyc in _linear(args, output_size, bias, bias_start, scope)
    729   # Calculate the total size of arguments on dimension 1.
    730   total_arg_size = 0
--> 731   shapes = [a.get_shape() for a in args]
    732   for shape in shapes:
    733     if shape.ndims != 2:

AttributeError: 'tuple' object has no attribute 'get_shape'

Hello, I am looking to use tf-fold for a seq2seq VAE.

I can see how the encoder might be implemented (with RNN), but I am struggling to find how you might go about producing a sequence from an RNN decoder, i.e. computing a value, then using the new hidden state to compute the next etc..

Could broadcast help here? Something like a scan function seems more appropriate.

I'm trying to figure out how to use td.ScopedLayer() with tf.conv1d(). What am I doing wrong?

def conv1d_layer(x):
  return td.ScopedLayer(tf.nn.conv1d(x, tf.zeros([3, 1, 1]), stride=2, padding="VALID"))

test_seq = [[x for x in xrange(0,randint(1,10))] for v in xrange(randint(3,5))]

print((td.Map(td.Map(td.Scalar())) >> conv1d_layer).eval(test_seq))

Getting the following error:

Traceback (most recent call last):
  File "test.py", line 109, in 
    print((td.Map(td.Map(td.Scalar())) >> conv1d_layer).eval(test_seq))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_fold/blocks/blocks.py", line 156, in __rshift__
    return Pipe(self, rhs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_fold/blocks/blocks.py", line 997, in Pipe
    return _pipe([convert_to_block(b) for b in blocks],
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_fold/blocks/blocks.py", line 2036, in convert_to_block
    raise TypeError('%s cannot be converted to a block' % str(block_like))
TypeError: < function conv1d_layer at 0x7fc5e6c28320 > cannot be converted to a block

Hello, When I executed the same code posted on the The sentiment notebook, following error message is displayed.

InvalidArgumentError: Cannot assign a device to node 'Adagrad/update_word_embedding/weights/Unique': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/GPU:0' Colocation Debug Info: Colocation group had the following types and devices: SparseApplyAdagrad: CPU UnsortedSegmentSum: GPU CPU StridedSlice: GPU CPU Shape: GPU CPU Unique: GPU CPU VariableV2: GPU CPU Const: GPU CPU [[Node: Adagrad/update_word_embedding/weights/Unique = Unique[T=DT_INT32, _class=["loc:@word_embedding/weights"], out_idx=DT_INT32](gradients/while/Function_19/word_embedding_1/Gather/Enter_grad/b_acc_6)]]

If I change sess = tf.InteractiveSession() to following code,

config = tf.ConfigProto(allow_soft_placement = True) sess = tf.InteractiveSession(config = config)

then, following error occurs

InvalidArgumentError: AttrValue must not have reference type value of float_ref for attr 'tensor_type' ; NodeDef: word_embedding/weights/Adagrad/_47 = _Recv[_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2263_word_embedding/weights/Adagrad", tensor_type=DT_FLOAT_REF, _device="/job:localhost/replica:0/task:0/cpu:0"](^Adagrad/learning_rate/_49, ^Adagrad/update_word_embedding/weights/UnsortedSegmentSum, ^Adagrad/update_word_embedding/weights/Unique); Op<name=_Recv; signature= -> tensor:tensor_type; attr=tensor_type:type; attr=tensor_name:string; attr=send_device:string; attr=send_device_incarnation:int; attr=recv_device:string; attr=client_terminated:bool,default=false; is_stateful=true> [[Node: word_embedding/weights/Adagrad/_47 = _Recv[_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2263_word_embedding/weights/Adagrad", tensor_type=DT_FLOAT_REF, _device="/job:localhost/replica:0/task:0/cpu:0"](^Adagrad/learning_rate/_49, ^Adagrad/update_word_embedding/weights/UnsortedSegmentSum, ^Adagrad/update_word_embedding/weights/Unique)]]

I found that when I change the adagrad to GradientDescentOptimizer or AdamOptimizer, the code works, but the accuracy of the model is inferior than the adagrad version. I found that Adagrad is not working on gpu <AdaGradOptimizer on GPU (Making word2vec tutorial runnable on GPU)> any help, please?

Generic tensorflow code do not suffer this problem... so I guess this is related to the fold library?

Hello,

I am trying to do some recursive computation using TF Fold, and there is an error which I want to debug.

Here is some code:

expr_decl_1 = td.ForwardDeclaration(td.PyObjectType(), td.TensorType([128]))
expr_decl_2 = td.ForwardDeclaration(td.PyObjectType(), td.TensorType([50]))

case_module1 = td.Record([('time_idx', td.Scalar('int32')),
                                            ('batch_idx', td.Scalar('int32'))])
case_module1 = case_module1 >> td.Function(Module1)
case_module2 = td.Record([('input_0', expr_decl_2()),
                                  ('time_idx', td.Scalar('int32')),
                                  ('batch_idx', td.Scalar('int32'))])
case_module2 = case_module2 >> td.Function(Module2)
case_module3 = td.Record([('input_0', expr_decl_1()),
                                  ('time_idx', td.Scalar('int32')),
                                  ('batch_idx', td.Scalar('int32'))])
case_module3 = case_module3 >> td.Function(Module3)
case_output = td.Record([('input_0', expr_decl_1()),
                                  ('time_idx', td.Scalar('int32')),
                                  ('batch_idx', td.Scalar('int32'))])
case_output = case_output >> td.Function(OutputModule)

recursion_case_1 = td.OneOf(td.GetItem('module'), { '_module1': case_module1})
expr_decl2().resolve_to(recursion_case_1)
recursion_case2 = td.OneOf(td.GetItem('module'), {'_module2': case_module2, '_module3': case_module3})
expr_decl_1().resolve_to(recursion_case_2)

output_scores = td.OneOf(td.GetItem('module'), {
                '_Output': case_output,
                INVALID_EXPR: dummy_scores})
self.compiler = td.Compiler.create(output_scores)
self.output = self.compiler.output_tensors[0]

I was wondering how I could print the intermediate outputs, which come from Module1, Module2, Module3, and OutputModule? This way I can see which module (series of NN layers) is causing the network to predict 0.

Here is an explanation of what this code is doing: Basically, what I have is 3 separate modules (referred to as Module1, Module2, Module3). The first one takes no input, and outputs a size [batch_size, 50] tensor. This is given as input to module 2, which produces a size [batch_size, 128] tensor. That is given as input to module3. These modules are to be assembled using a dictionary that is passed in.

Does anyone have an idea how to print these #values?

Edit: For more context, I am trying to build a Neural Modular Network. Most of the code is similar to that described here: https://github.com/ronghanghu/n2nmn/blob/master/models_clevr/nmn3_model.py. However, the key difference in my case is that I am trying to use different shapes of inputs.

In the code below, I'm taking in a list, converting it to an ndarray, and then feeding it to a td.Tensor block. When I do this, Fold tells me that the variable x I'm passing to the td.Tensor is a Fold TensorType object, but x is really a numpy ndarray. TypeError: Type mismatch between input type TensorType((1, 5, 1), 'float32') and expected input type PyObjectType() in <td.Tensor dtype='float32' shape=(1, 5, 1)>.

def list_to_tensor(x, name='input_layer'):
	list_to_tensor = td.Composition()
	with list_to_tensor.scope():
		sh  = [1,len(x),1]
		x   = np.array(x, dtype=np.float32).reshape(sh)
		tn  = td.Tensor(sh).reads(x)
	return tn
input_layer = td.InputTransform(list_to_tensor)
features3 = [x for x in range(5)]
p = input_layer.eval(features3)
print p
print type(p)

When I only return x without passing it to td.Tensor, I can clearly see that it's a numpy ndarray:

def list_to_tensor(x, name='input_layer'):
	list_to_tensor = td.Composition()
	with list_to_tensor.scope():
		sh  = [1,len(x),1]
		x   = np.array(x, dtype=np.float32).reshape(sh)
		print ("type(x): ",type(x))
		tn  = td.Tensor(sh)
	return (x, tn)

This outputs fine:

(array([[[ 0.],
        [ 1.],
        [ 2.],
        [ 3.],
        [ 4.]]], dtype=float32), <td.Tensor dtype='float32' shape=(1, 5, 1)>)

Why is it telling me that in the first scenario that x is a Fold TensorType?

Since the fold library receives a tree structured data as an input(of RNN), I find it hard to feed another input when I tried to use two inputs for two models.

What do I want to do?

I want to train two models, say Recursive NN and CNN, to classify the sentiment of a sentence.

What is the input format?

First input format for Recursive NN is a binary parsed tree, made from the stanford coreNLP, labels suggest (1 to 5) sentiment. I made up this example to classify the sentiment of the sentence.

ex1)

(4 (3 You) (3 (3 (3 can) (2 (5 trust) (4 stackoverflow))) (3 .)))

Second input would be just a sentence(same sentence with no parsing), for CNN.

ex2) You can trust stackoverflow.

Why is it challenging?

Sentiment example classify sentiment using recursive NN. And it seems that using CNN alone in the fold library is plausible.

def tokenize(s):
  label, phrase = s[1:-1].split(None, 1)
  return label, sexpr.sexpr_tokenize(phrase)

In the linked example, tokenize function(above) is the first function to receive inputs after feed process. I think I can concatenate the two inputs and pass it to the feed, then separate them later.

The problem is, the input for recursive NN model is tied to the model, which is computed node by node. A parent node computes its state vector using the state vectors of its two children(as the input is the binary tree). Following code snippet is taken from the linked example.

embed_subtree = td.ForwardDeclaration(name='embed_subtree')
...
def logits_and_state():
  ...
  pair2vec = (embed_subtree(), embed_subtree())
...
embed_subtree.resolve_to(embed_tree(logits_and_state(), is_root=False))
...

Since CNN requires the full sentence as its input, how can I feed the full sentence to the CNN module?

Actually, I posted this question to the stackoverflow first. I thought the stackoverflow was a better place to post application questions. Well.. I hope that the contributors of the fold library feel interested to this question.

Hi suppose I have an RNN where at each time step I have labels for the output, in this case how might I use fold to set up the computation graph. I am aware that we can get the last output state of the RNN using td.RNN, but is there a way to accomplish what I described?

Replace each input word with a different image on that cute animation on the first page. Replace 'embed' with a tf.nn.conv2d. That's what I want to try. Can it be done?

I tried to td.Function(tf.nn.conv2d) but it expects more arguments...

Hello, I build fold and tensorflow from source by instructions in "https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/sources.md". However, when I impement "import tensorflow_fold as td" I get the error:

tensorflow.python.framework.errors_impl.NotFoundError: /home/xxx/.local/lib/python3.5/site-packages/tensorflow_fold/loom/_deserializing_weaver_op.so: undefined symbol: _ZN10tensorflow7strings6

Also I have tried install tensorflow-1.0 and 1.4 directly from network by pip , but the error still exist.

Implementing "ldd _deserializing_weaver_op.so" output the following information: linux-vdso.so.1 => (0x00007fff972d2000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f76c2ab6000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f76c27ad000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f76c242a000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f76c2214000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76c1e4a000) /lib64/ld-linux-x86-64.so.2 (0x0000562a389c0000)

Hi, It looks like the data/information regarding the usage of the Fold library is outdated now. Links are broken. I am trying to install GPU version of the Fold in the conda environment for TF 2.3.0/Python 3.7 but the information link is broken now. So I am not getting any help regarding the usage of Fold.

I am interested in using Tensorflow Fold, but I would like for my application to be extensible and reproducible. Are there plans to upgrade this repo to more recent versions of TF? If not, how extensive is the task of doing the upgrading myself? I am willing to work on a PR for that.

I've written custom parsers and emitters for everything from docstrings to classes and functions. However, I recently came across an issue with the TensorFlow codebase: inconsistent use of Args: and Arguments: in its docstrings. It is easy enough to extend my parsers to support both variants, however it looks like Arguments: is wrong anyway, as per:

• https://google.github.io/styleguide/pyguide.html#doc-function-args @ ddccc0f

• https://chromium.googlesource.com/chromiumos/docs/+/master/styleguide/python.md#describing-arguments-in-docstrings @ 9fc0fc0

• https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html @ c0ae8e3

Therefore, only Args: is valid. This PR replaces them throughout the codebase.

PS: For related PRs, see tensorflow/tensorflow/pull/45420

I perform sentiment analysis and I want to get the root embeddings of all trees after I train the whole training dataset of trees. I will use them all in a novel algorithm I've been developing.

What is the Python code for it? I want to obtain those root vectors (tensors) only, not the output scores for each sentiment label. I tried printing the names of all tensors through the following line of code:

[print(n.name) for n in tf.get_default_graph().as_graph_def().node]

However, I am not sure whether this code snippet prints the name of the root embedding tensor let alone exactly which one of them represents the root embedding.

Would it be possible to write an algorithm like NEAT which evolves the topology of a neural network in Fold? From my understanding, Fold does not support dynamic computation graphs in this sense, but these dynamic graphs are rather input dependent.