• Issue: For some reason both pip install tensor-sensor pip install tensor-sensor[torch] attempt to install Tensorflow too.

• Environment:

  • win10 latest (10.10.2020)
  • conda 4.8.3 virtual env
  • pytorch 1.6.0 installed via conda (the official way)
  • no tensorflow

• Workaround: pip install tensor-sensor --no-deps pip install graphviz

Hi,

Thanks for the awesome library! This has really made my debugging life much easier.

Just a question. Is there any plan to support JAX? I think this can be similarly supported since the API of JAX almost looks identical to NumPy.

Currently, in the analysis.py _shape method it always try to check if torch.Size exist. So for a keras user, if they don't have torch install, it will throw an error since analysis.py is importing it.

  File "/home/shawley/Downloads/tensor-sensor/tsensor/analysis.py", line 27, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

Related #8

Hi! I stumbled across this library and noticed I could help. I've written a couple of libraries that are great for this stuff:

• https://github.com/alexmojaki/executing
• https://github.com/alexmojaki/pure_eval

Here is a demo of how you could use it for this kind of project:

import ast

import executing
import pure_eval
import sys


def explain_error():
    ex = executing.Source.executing(sys.exc_info()[2])
    if not (ex.node and isinstance(ex.node, ast.BinOp)):
        return

    evaluator = pure_eval.Evaluator.from_frame(ex.frame)
    atok = ex.source.asttokens()

    try:
        print(f"Cannot add "
              f"{atok.get_text(ex.node.left)} = {evaluator[ex.node.left]!r} and "
              f"{atok.get_text(ex.node.right)} = {evaluator[ex.node.right]!r}")
    except pure_eval.CannotEval:
        print(f"Cannot safely evaluate operands of {ex.text()}. Extract them into variables.")


a = ["abc", 3]

try:
    print(a[0] + a[1])
except:
    explain_error()

try:
    print("only print once") + 3
except:
    explain_error()

To run this you will need to pip install executing pure_eval asttokens.

This should improve the parsing and such significantly. For example this will handle line continuations just fine. pure_eval will only evaluate simple expressions to avoid accidentally triggering side effects.

This uses the ast module from the standard library. Is there a reason you wrote your own parser? The best place to learn about ast is here: https://greentreesnakes.readthedocs.io/en/latest/

I'll let you integrate it into your code yourself, but let me know if you have questions.

Very often, people already have the tensor library they are using, keras, torch, tensorflow if they are coming for this library. Currently, the package tries to install all dependencies. For example, if I am using PyTorch, I don't really need to install the big TensorFlow library in the environment.

Added options

 pip install tensor-sensor[all]
 pip install tensor-sensor[torch]
 pip install tensor-sensor[tensorflow]

Enable PDF (and other supported formats) savefig in explainer by removing hardcoded extension

import torch
W = torch.rand(d,n_neurons)
b = torch.rand(n_neurons,1)
X = torch.rand(n,d)
with ts.explain(savefig="my_inspection.pdf"):
    Y = W @ X.T + b

From @sbrugman:

Our concrete issue was with Pytorch (unexpectedly) converting tensors with only integers to float, which later in the program resulted in an error because it could not be used as an index. Another issue was changing size from 32 to 64 bit floats.

It's indeed the element type of the matrix.

There are multiple somewhat related issues: https://discuss.pytorch.org/t/problems-with-target-arrays-of-int-int32-types-in-loss-functions/140/2 https://discuss.pytorch.org/t/why-pytorch-is-giving-me-hard-time-with-float-long-double-tensor/14678/6

The common denominator between dimensionality debugging is that both type and dimensionality are practically hidden from the user:

import numpy as np
import tsensor as ts

x = np.arange(6, dtype=np.float32)

with ts.explain(savefig="types.pdf"):
    print(x.dtype)
    print((x*x).dtype)
    print((np.sin(x)).dtype)
    print((x + np.arange(6)).dtype)
    print((np.multiply.outer(x, np.arange(2.0))).dtype)
    print((np.outer(x, np.arange(2.0))).dtype)

Hello.

I am using tensor-sensor to visualise DAGs of the domain specific language Slate, which is used in a code generation framework for Finite element methods called Firedrake. Slate expresses linear algebra operations on tensors. I am using tensor-sensor to visualise the DAG before and after an optimisation pass. An example would be the following:

Before optimisation: triplemul_beforeopt.pdf After optimisation: tripleopt_afteropt.pdf

While the visualisation of the tree is correct in both cases, I would quite like to suppress the node for the brackets (i.e. for the SubExpr node) to avoid confusion about the amount of temporaries generated. Is there already a way of controlling this as a user and if not would there be interest in supporting it?

Best wishes and thanks in advance, Sophia

Hi! I really thank u for your brilliant work of tsensor which help me to debug more effectively.

But recently, when I run this code in Jupyter or Pycharm, it always leads to a KeyError:

code: with ts.explain(): a = np.ones(3)

KeyError report: KeyError Traceback (most recent call last) in () 1 with ts.explain(): ----> 2 a = np.ones(3) 3

F:\anaconda_file2\envs\test\lib\site-packages\numpy\core\numeric.py in ones(shape, dtype, order) 206 """ 207 a = empty(shape, dtype, order) --> 208 multiarray.copyto(a, 1, casting='unsafe') 209 return a 210

<array_function internals> in copyto(*args, **kwargs)

F:\anaconda_file2\envs\test\lib\site-packages\tsensor\analysis.py in listener(self, frame, event, arg) 266 267 def listener(self, frame, event, arg): --> 268 module = frame.f_globals['name'] 269 info = inspect.getframeinfo(frame) 270 filename, line = info.filename, info.lineno

KeyError: 'name'

Is there anything I can do to fix this problem? Grateful to gain any feedback!

Hello,

I created a decorator to call clarify around the forward function of my custom Pytorch models (derived from torch.nn.Module).

Said decorator looks like this:

def clarify(function: callable) -> callable:
    """ Clarify decorator."""

    def call_clarify(*args, **kwargs):
        with tsensor.clarify(fontname="DejaVu Sans"):
            return function(*args, **kwargs)

    return call_clarify

When doing machine learning using Pytorch, models (derived from torch.nn.Module) can sometimes be "stacked". In a translation task, an EncoderDecoder's forward will call its Decoder's forward, itself calling the forward of an Attention module, for example.

In such a case, this results in nested clarify calls, which raise a succession of Exceptions, because some of the topmost clarify function do not exit correctly. To be more specific, l.124 of analysis.py, self.view can be None, which then raises an Exception on self.view.show().

A quick fix (that I did in local) was adding a check line 131:

                if self.view:
                    if self.show=='viz':
                        self.view.show()
                    augment_exception(exc_value, self.view.offending_expr)

However, I am not sure this would be the best fix possible, as I am not sure whether that is a common problem or not and how/if this is intended to be fixed. What do you think?

I've a feeling that at some point many(including me) would like to contribute to this library and it would be great if it had some contribution guidelines.

There is some overlap with these boxes:

import torch
import tsensor

n = 200         # number of instances
d = 764         # number of instance features
nhidden = 256

Whh = torch.eye(nhidden, nhidden)  # Identity matrix
Uxh = torch.randn(nhidden, d)
bh  = torch.zeros(nhidden, 1)
h = torch.randn(nhidden, 1)         # fake previous hidden state h
r = torch.randn(nhidden, 3)         # fake this computation
X = torch.rand(n,d)                 # fake input

with tsensor.explain(savefig):
    r*h

The following code generates an exception but instead of showing the result of the operand subexpressions, it shows all bits of it:

import torch
import tsensor

n = 200         # number of instances
d = 764         # number of instance features
nhidden = 256

Whh = torch.eye(nhidden, nhidden)  # Identity matrix
Uxh = torch.randn(nhidden, d)
bh  = torch.zeros(nhidden, 1)
h = torch.randn(nhidden, 1)         # fake previous hidden state h
# r = torch.randn(nhidden, 1)         # fake this computation
r = torch.randn(nhidden, 3)         # fake this computation
X = torch.rand(n,d)                 # fake input

# Following code raises an exception
with tsensor.clarify():
    h = torch.tanh(Whh @ (r*h) + Uxh @ X.T + bh)  # state vector update equation

The following exception not only generates a huge stack trace but also TensorSensor gives an error message augmentation indicating that Y = model(X) is the issue because it does not descend into tensor library code. It would be better to allow it to see inside the model pipeline. so that it can notice that the error is actually here:

nn.Linear(10, n_neurons)

which should be

nn.Linear(n_neurons, 10)

Here's the full example:

from torch import nn
n = 20
n_neurons = 50
model = nn.Sequential(
    nn.Linear(784, n_neurons), # 28x28 flattened image
    nn.ReLU(),
    nn.Linear(10, n_neurons),  # 10 output classes (0-9) <---- ooops! reverse those
    nn.Softmax(dim=1)
)
X = torch.rand(n,784) # n instances of feature vectors with 784 pixels
with tsensor.clarify():
    Y = model(X)

The error message we get is here:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-32-203c7ad8d609> in <module>
      1 with tsensor.clarify():
----> 2     Y = model(X)

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    137     def forward(self, input):
    138         for module in self:
--> 139             input = module(input)
    140         return input
    141 

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     94 
     95     def forward(self, input: Tensor) -> Tensor:
---> 96         return F.linear(input, self.weight, self.bias)
     97 
     98     def extra_repr(self) -> str:

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1845     if has_torch_function_variadic(input, weight):
   1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1847     return torch._C._nn.linear(input, weight, bias)
   1848 
   1849 

RuntimeError: mat1 and mat2 shapes cannot be multiplied (20x50 and 10x50)
Cause: model(X) tensor arg X w/shape [20, 784]

great works!

debug these math expression code is a very big problem

not only scalar vector matrix , high rank tensor support

Visualizing 3D tensors and beyond:

TensorDim is contain : left to right [N,C,H,W] N->C->H->W TensorDim like flatten Onions,cabbage box in box represent dim TensorDim has real represention information (N,C,H,W etc.)

namedtensor support

plaidml dsl Types Tensor TensorDim TensorIndex

represent actual data: audio, 1D plot, text : vector image ,2D plot : matrix etc.

expression graph:

different color input variable ,leaf parameters/variable with temporal variable / activation : "Road width" elementwise leaf / elementwise or slice or other connection(edge) print AST forward & backward mode

animation:

expression step computation mode (debug) slice ,reshape, .T() other manipulate N-d array TensorDim/TensorIndex operators example : matmul = m@v m@v = (v.squeeze(0).expand_as(w) * w).sum(1,keepdim=True).unsqueeze(1) more ... conv2d etc

interactive:

interactive build block(visual programing) reverse interactive(debug) : selected tensor elements and follow it in expression graph selected element : muti view , like convolution-visualizer Input(Inputgrad) Weight(Weightgrad) Output(Outputgrad) views

NN support:

NN module visualization (conv2d) bigger computation graph : pytorchrec multi computation graph visualization and live debug

other useful link

SimpNet memory map Visualize the virtual address space of a Windows process on a Hilbert curve.

nvidia ppt

tensor network

einops