This is the converter/parser I committed here, adapted to the current IR in #3.

Heavily WIP, just posting for early feedback.

• [x] Add documentation
• [x] Lots of tests
• [x] Get rid of BLOCKS/RETURNS hack and actually return the generated functions
• [x] Consider if an alternative for the CONSTANTS dictionary is needed

Still WIP, but I made a start with the IR. Some points:

• The nodes have an object hierarchy so that you can use isinstance.
• I imagine that we might have situations in which we want to serialize the IR (e.g. create protobufs) and such, so trying to keep it pretty minimal.
• Edges are one-directional right now, since bidirectionality is only needed in some passes (and requires more bookkeeping when mutating the graph) I think it's best to try and keep def-use edges separate somehow
• I was thinking we can hide all the debug information in a single dictionary. Debug information would include things such as human readable names of the variable and the Python AST node that created the node from the parser.

This adds the myia.cconv.NestingAnalyzer class, used as follows:

def f(x):
    def g():
        return x
    return g

f_graph = parse(f)

na = NestingAnalyzer().run(f_graph)
assert na.parents == {f_graph: None, g_graph: f_graph}
assert na.fvs == {f_graph: set(), g_graph: {x_node}}

(This won't run because g_graph etc. are not set to the corresponding graphs and nodes, but it should illustrate the point nonetheless.)

All dependencies are now defined in pyproject.toml and most are installed with poetry install. Conda dependencies are put in custom sections in the file and you have to use scripts/make_deps.py to generate a requirements.conda file. scripts/install-deps-[g/c]pu.sh will do it for you.

This also moves some scripts into scripts/.

Oh, and PyTorch is upgraded to 1.3.0. Unfortunately there are a few errors with 1.4.0 (to investigate).

Hi @abergeron @breuleux

I just make this PR to ease any discussion around the RNG implementation.

Currently, I have implemented the RNG in pure Python, based on Theano code. For reference:

• Theano code is here: https://github.com/Theano/Theano/blob/master/theano/sandbox/rng_mrg.py
• Original Java implementation by Pierre Lecuyer is here: https://github.com/umontreal-simul/ssj/blob/master/src/main/java/umontreal/ssj/rng/MRG31k3p.java

Implementation currently seems to work. I also added a test to compare output with the Theano output, and it currently produces same results.

Then, my next step was to try to compile the python implementation into myia, but I am facing many issues:

• Python implementation checks input seeds, as all values are not allowed, and generate some exceptions if bad seeds are given. It seems I can not currently compile exceptions raising with Myia, though I saw a raise primitive in operations. I don't yet know why compilation fails.
• I need to generate the output array and populate it with random values, but I don't yet have a way to create new data. @breuleux told me about scalar_to_array + distribute op, but, from what I saw, it seems these operations are currently used in zero_like op (isn't it?), which is not suited here. So I guess I need to create a zeros operation that can take just shape and dtype as argument, and I guess I should use np.zeros as entry point (ie. value to be replaced by the op later in compilation).
• I also need to create the initial random state from given seed, which can be either an integer or a ready-to-use integer vector with 6 values. I guess I should create something like np.asarray op here to ease this step.
• I simplified the random generator so that it uses only one vector as state to update, instead of using a matrix as in Theano (in Theano, this should be equivalent to get random numbers with parameter nstreams=1). Thus, in intermediate functions, I could just return the new updated state vector (it contains only 6 values) instead of trying to update a potential huge matrix inplace. So, I removed a place where inplace update was used, but I will still need to make inplace updates when generating random numbers, as I will need to a) create the output array, and b) update each array value with a generated random scalar.

This is where I am, currently. What do you think?

In the spirit of early PRs, here is my code for the VM.

What I'm trying to do here is to execute the graph directly to make easier to convert into a debugger. I'm still handling tail calls correctly so that we don't need an infinite amount of memory.

This is currently very inefficient for temporary storage of the computed values (as in all values are kept until the function returns, with the exception that tail calls are not considered returns). There can also be situations where we will recompute values more than once when they are used across internal functions. The results should always be good though.

Missing operations in relay backend to make all tests pass:

• array_setitem
• conv_transpose2d
• gather
• scatter
• scatter_add

I am currently working on conv_transpose2d. Issues:

• Seems limited to only groups=1 and dilation=(1, 1): https://github.com/apache/incubator-tvm/blob/master/python/tvm/relay/op/strategy/x86.py#L169
• Persistent error when running pytest -xvvs tests/frontends/test_pytorch_ops.py::test_torch_conv2d on this branch:

(myia) notoraptor@notoraptor-linux:~/mila/dev/git/myia$ pytest -xvvs tests/frontends/test_pytorch_ops.py::test_torch_conv2d
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/notoraptor/anaconda3/envs/myia/bin/python
cachedir: .pytest_cache
rootdir: /media/win/Users/notoraptor/mila/dev/git/myia, inifile: pytest.ini
plugins: cov-2.8.1
collected 6 items                                                                                                                                                                                          

tests/frontends/test_pytorch_ops.py::test_torch_conv2d[relay-cpu-grad0] <- tests/multitest.py ANTLR runtime and generated code versions disagree: 4.8!=4.7.2
ANTLR runtime and generated code versions disagree: 4.8!=4.7.2
Cannot find config for target=llvm, workload=('conv2d_NCHWc.x86', ('TENSOR', (2, 6, 4, 5), 'float32'), ('TENSOR', (3, 6, 3, 3), 'float32'), (2, 3), (3, 2, 3, 2), (1, 1), 'NCHW', 'NCHW', 'float32'). A fallback configuration is used, which may bring great performance regression.
FAILED

================================================================================================= FAILURES =================================================================================================
____________________________________________________________________________________ test_torch_conv2d[relay-cpu-grad0] ____________________________________________________________________________________

test = <tests.multitest.MyiaFunctionTest object at 0x7f7570a72650>

    def runtest(test):
>       test.run(fn)

tests/multitest.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/multitest.py:166: in run
    return self.runtest(self, fn, **self.spec)
tests/frontends/test_pytorch_ops.py:147: in _fwd_and_bwd
    res = gpipeline.run(input=fn, argspec=[*argspec, sens_type])
myia/pipeline/pipeline.py:144: in run
    return self.make()(**args)
myia/pipeline/pipeline.py:201: in __call__
    return self[:](**args)
myia/pipeline/pipeline.py:245: in __call__
    raise results['error']
myia/pipeline/pipeline.py:229: in run_and_catch
    results = fn(**valid_args)
myia/pipeline/steps.py:402: in step_compile
    out = resources.backend.compile(graph, argspec, outspec)
myia/pipeline/resources.py:201: in compile
    return self.backend.compile(graph, argspec, outspec)
myia/compile/backends/__init__.py:277: in compile
    return self.proc.call_method('compile', graph, argspec, outspec)
myia/compile/channel/__init__.py:131: in call_method
    return self._read_msg()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <myia.compile.channel.RPCProcess object at 0x7f756e1fb5d0>

    def _read_msg(self):
        RemoteHandle.current_channel = self
        try:
            res = self.loader.get_data()
        finally:
            RemoteHandle.current_channel = None
        if isinstance(res, LoadedError):
>           raise res
E           myia.utils.serialize.LoadedError: Traceback (most recent call last):
E           
E             File "/media/win/Users/notoraptor/mila/dev/git/myia/myia/compile/channel/__main__.py", line 38, in _rpc_server
E               res = meth(*args, **kwargs)
E           
E             File "/media/win/Users/notoraptor/mila/dev/git/myia/myia/compile/backends/__init__.py", line 297, in compile
E               return handle(self.real.compile(graph, argspec, outspec))
E           
E             File "/media/win/Users/notoraptor/mila/dev/git/myia/myia/compile/backends/relay.py", line 884, in compile
E               self.exec_kind)
E           
E             File "/media/win/Users/notoraptor/mila/dev/git/myia/myia/compile/backends/relay.py", line 675, in run
E               add_functions(self.module, function_map, self.types)
E           
E             File "/media/win/Users/notoraptor/mila/dev/git/myia/myia/compile/backends/relay_helpers.py", line 275, in add_functions
E               mod[gv] = funcs[gv]
E           
E             File "/home/notoraptor/anaconda3/envs/myia/lib/python3.7/site-packages/tvm/ir/module.py", line 75, in __setitem__
E               return self._add(var, val)
E           
E             File "/home/notoraptor/anaconda3/envs/myia/lib/python3.7/site-packages/tvm/ir/module.py", line 84, in _add
E               _ffi_api.Module_Add(self, var, val, update)
E           
E             File "tvm/_ffi/_cython/./packed_func.pxi", line 308, in tvm._ffi._cy3.core.PackedFuncBase.__call__
E           
E             File "tvm/_ffi/_cython/./packed_func.pxi", line 253, in tvm._ffi._cy3.core.FuncCall
E           
E             File "tvm/_ffi/_cython/./base.pxi", line 159, in tvm._ffi._cy3.core.CALL
E           
E           tvm._ffi.base.TVMError: Traceback (most recent call last):
E             [bt] (8) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(TVMFuncCall+0x65) [0x7f8ea4b23a25]
E             [bt] (7) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0x539dc4) [0x7f8ea440fdc4]
E             [bt] (6) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0x539a24) [0x7f8ea440fa24]
E             [bt] (5) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(tvm::IRModuleNode::Add(tvm::GlobalVar const&, tvm::BaseFunc const&, bool)+0x425) [0x7f8ea440d875]
E             [bt] (4) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0x5322d4) [0x7f8ea44082d4]
E             [bt] (3) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(tvm::relay::InferType(tvm::relay::Function const&, tvm::IRModule const&, tvm::GlobalVar const&)+0x1db) [0x7f8ea49c545b]
E             [bt] (2) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0xaeec28) [0x7f8ea49c4c28]
E             [bt] (1) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0x5245fa) [0x7f8ea43fa5fa]
E             [bt] (0) /home/notoraptor/anaconda3/envs/myia/lib/libtvm.so(+0x423d01) [0x7f8ea42f9d01]
E             File "/home/user/conda-bld/tvm-libs_1584032126820/work/src/ir/error.cc", line 133
E           TVMError: 
E           Error(s) have occurred. The program has been annotated with them:
E           
E           In `main`: 
E           v0.0.4
E           fn (%v_parameter25: Tensor[(2, 6, 4, 5), float32], %v_parameter26: Tensor[(3, 6, 3, 3), float32], %v_parameter27: Tensor[(3), float32], %v_parameter28: float32) -> (Tensor[(2, 6, 4, 5), float32], Tensor[(3, 6, 3, 3), float32], Tensor[(3), float32]) {
E             let %seq.0 = (meta[relay.Constant][0],);
E             let %seq.1 = (meta[relay.Constant][1], meta[relay.Constant][2], meta[relay.Constant][3], meta[relay.Constant][4]);
E             let %seq.2 = (meta[relay.Constant][5], meta[relay.Constant][6], meta[relay.Constant][7], meta[relay.Constant][8]);
E             let %seq.3 = broadcast_to(%v_parameter28, meta[relay.attrs.InitOpAttrs][0]);
E             let %seq.4 = sum(%seq.3, axis=[0, 2, 3], keepdims=True);
E             let %seq.5 = reshape(%seq.4, newshape=[3]);
E             let %seq.6 = meta[relay.Constant][9];
E             let %seq.7 = (meta[relay.Constant][10], meta[relay.Constant][11]);
E             let %seq.8 = (meta[relay.Constant][12], meta[relay.Constant][13]);
E             let %seq.9 = (meta[relay.Constant][14], meta[relay.Constant][15]);
E             let %seq.10 = (meta[relay.Constant][16], meta[relay.Constant][17], meta[relay.Constant][18], meta[relay.Constant][19]);
E             %0 = reshape(%v_parameter25, newshape=[1, -1, 0, 0]);
E             %1 = tile(%seq.3, reps=[1, 6, 1, 1]);
E             %2 = reshape(%1, newshape=[-1, 1, 0, 0]);
E             %3 = nn.conv2d(%0, %2, padding=[3, 2, 3, 2], dilation=[2, 3], groups=12);
E             %4 = reshape(%3, newshape=[2, 6, 3, 4, 3]);
E             %5 = sum(%4, axis=[0]);
E             %6 = transpose(%5, axes=[1, 0, 2, 3]);
E             let %seq.11 = strided_slice(%6, begin=[0, 0, 0, 0], end=[None, None, 3, 3]);
E             let %seq.12 = (1, 0);
E             let %seq.13 = ();
E             let %seq.14 = nn.conv2d_transpose(%seq.3, %v_parameter26, channels=3, kernel_size=[3, 3], strides=[2, 3], output_padding=[1, 0], padding=[3, 2, 3, 2]) in particular dimension 1 conflicts 3 does not match 6; unable to unify: `Tensor[(3, 3, 3, 3), float32]` and `Tensor[(3, 6, 3, 3), float32]`; in particular dimension 1 conflicts 3 does not match 6; unable to unify: `Tensor[(2, 3, 4, 5), float32]` and `Tensor[(2, 6, 4, 5), float32]`; ;
E             let %seq.15 = (%seq.14, %seq.11, %seq.5);
E             %seq.15
E           }
E           // meta data omitted. you can use show_meta_data=True to include meta data

myia/compile/channel/__init__.py:148: LoadedError
========================================================================================= short test summary info ==========================================================================================
FAILED tests/frontends/test_pytorch_ops.py::test_torch_conv2d[relay-cpu-grad0] - myia.utils.serialize.LoadedError: Traceback (most recent call last):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================ 1 failed in 17.73s ============================================================================================

The main error is in this line: let %seq.14 = nn.conv2d_transpose(%seq.3, %v_parameter26, channels=3, kernel_size=[3, 3], strides=[2, 3], output_padding=[1, 0], padding=[3, 2, 3, 2]) in particular dimension 1 conflicts 3 does not match 6; unable to unify:Tensor[(3, 3, 3, 3), float32]andTensor[(3, 6, 3, 3), float32]; in particular dimension 1 conflicts 3 does not match 6; unable to unify:Tensor[(2, 3, 4, 5), float32]andTensor[(2, 6, 4, 5), float32]; ;. I still don't know what causes this.

This adds gradients for various primitives, and gradient generation. The tests pass and there is 100% coverage, but it's worth mentioning the whole thing is slightly rushed. A few comments:

• There are two xfails which should be fixed at some point.
• Support for array operations has not been added yet.
• The default pipeline doesn't support gradients yet. The one that does is in test_grad, but it's a bit wonky (likely responsible for one of the xfails).
• The grad transform isn't very well documented.
• J/Jinv won't work with the debug VM right now.

Factored this out of #4 so that it is a bit easier to review. This fixes a bunch of issues with the node naming. The current naming:

>>> def f(x):
...     return x
>>> parse(f)
<f = Graph(parameters=[x = Parameter(<f = Graph(parameters=[...])>)])>
>>> parse(f).return_
Apply(Inputs([Constant(<myia.primops.Return object at 0x106271630>), x = Parameter(<f = Graph(parameters=[x = Parameter(<f = Graph(parameters=[...])>)])>)]), <f = Graph(parameters=[x = Parameter(<f = Graph(parameters=[...])>)])>)

With this PR:

>>> parse(f)
<myia.anf_ir.Graph(name=f, parameters=[x], return_=_apply1) object at 0x105c72da0>
>>> parse(f).return_
<myia.anf_ir.Apply(name=_apply2, inputs=[_constant3, x], graph=f) object at 0x105c82eb8>

Main changes:

• Make generated names valid variable names
• Base generated names on type of object we are naming
• Use str representation where needed to avoid recursively printing the entire graph
• Ensure uniqueness of repr by printing object ID like CPython does
• Only represent constant with value if its a literal (to avoid printing really long strings when its value is e.g. a graph)

@abergeron @breuleux

This PR adds tan to relay backend.

NB:

TVM API changed again, and now it seems tvm.relay.Module does not exist anymore. So, I guess we would need to update whole relay backend again to support new TVM API before merging this pull request.

On my computer, I use this code to compile relay with current TVM code:

import tvm
from tvm import relay
from tvm.contrib import graph_runtime


# Old way, does not work anymore currently.
def compile_function_old(inputs, output):
    f = relay.Function(list(inputs), output)
    m = relay.Module({})
    fn = relay.GlobalVar('f')
    m[fn] = f
    e = relay.create_executor(mod=m)
    c = e.evaluate(m[fn])
    return c


# New way. I don't know if another way exists, but this works now.
def compile_function(inputs, output):
    ir = relay.Function(list(inputs), output)
    mod = tvm.IRModule.from_expr(ir)
    target = tvm.target.create('llvm')
    graph, lib, params = relay.build(mod, target=target)
    ctx = tvm.cpu()
    module = graph_runtime.create(graph, lib, ctx)

    def fn(*args):
        for i in range(len(inputs)):
            module.set_input(i, args[i])
        module.set_input(**params)
        module.run()
        out = module.get_output(0)
        return out

    return fn

Going through parse.py in master I noticed that the parser is responsible for checking whether we support the given AST, and if not, raising an informative syntax error. In Tangent we actually do this in a separate pass, which I think might be a good idea. It keeps the parser a bit cleaner (no need for checks) and separates two logical processes.

I also found a comment that says "maybe inherit from SyntaxError? Investigate." so I had a look, and it is a good idea. If you set the fields appropriately, it will automatically pretty print the error message with the filename, a little arrow, etc.

bartvm-macbookpro3:myia bartvm$ python myia/fence.py
Traceback (most recent call last):
  File "myia/fence.py", line 120, in <module>
    fence.visit(ast.parse(textwrap.dedent(inspect.getsource(f))))
  File "myia/fence.py", line 88, in visit
    self.generic_visit(node)
  File "/Users/bartvm/anaconda3/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "myia/fence.py", line 88, in visit
    self.generic_visit(node)
  File "/Users/bartvm/anaconda3/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "myia/fence.py", line 93, in visit
    return visitor(node)
  File "myia/fence.py", line 98, in visit_Pass
    self.raise_("you shall not pass")
  File "myia/fence.py", line 109, in raise_
    self.lines[self.lineno - 1], message)
  File "myia/fence.py", line 113
    pass
    ^
__main__.MyiaSyntaxError: invalid syntax: you shall not pass

I attached a minimal proof of concept of this approach. The syntax checker is conservative i.e. it rejects everything unless it's been explicitly marked as supported.

Bumps pillow from 6.2.2 to 9.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps pillow from 6.2.2 to 9.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps pillow from 6.2.2 to 9.0.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I reopend my PR here, as @breuleux PR is now merged.

• Still WIP, I will fix more tests today.
• Numpy dependencies removed.
• Basic support for dict added.
• currently 187 failed, 170 passed, 3 xfailed among added tests

It currently is not possible to array_getitem or array_setitem with single item in each dim non-statically; because current interface always assumes slices (even if slice is interval of size 1).