You might already have these changes locally, but I figured I'd upload this just in case, since people that watched the YouTube video probably expect these functions to be implemented.

Thank you @evcu for raising, my little 2D toy problem converged and instead of going on to proper tests and double checking through the recursion I got all trigger-happy and amused with puppies. The core issue is that if variables are re-used then their gradient will be accumulated for each path. Do you think this simpler reference counting idea will work as a potential simpler solution? The idea is to suppress backward() calls until the very last one.

(Love your Stylized puppy in your branch btw! :D)

class Value:
    """ stores a single scalar value and its gradient """

    def __init__(self, data):
        self.data = data
        self.grad = 0
        self.backward = lambda: None
        self.refs = 0

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data)
        self.refs += 1
        other.refs += 1
        
        def backward():
            if out.refs > 1:
                out.refs -= 1
                return
            self.grad += out.grad
            other.grad += out.grad
            self.backward()
            other.backward()
        out.backward = backward

        return out

    def __radd__(self, other):
        return self.__add__(other)

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data)
        self.refs += 1
        other.refs += 1
        
        def backward():
            if out.refs > 1:
                out.refs -= 1
                return
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
            self.backward()
            other.backward()
        out.backward = backward

        return out

    def __rmul__(self, other):
        return self.__mul__(other)

    def relu(self):
        out = Value(0 if self.data < 0 else self.data)
        self.refs += 1
        def backward():
            if out.refs > 1:
                out.refs -= 1
                return
            self.grad += (out.data > 0) * out.grad
            self.backward()
        out.backward = backward
        return out

    def __repr__(self):
        return f"Value(data={self.data}, grad={self.grad})"

Hey Andrej -- just want to say thanks so much for your YouTube video on micrograd. The video has been absolutely enlightening.

Quick question -- while re-implementing micrograd on my end, I noticed that __pow__ (in Value) was missing a back-propagation definition for other. Is this expected? https://github.com/karpathy/micrograd/blob/c911406e5ace8742e5841a7e0df113ecb5d54685/micrograd/engine.py#L39-L40

Feature

• Convert Micrograd into a PyPI package.

Need

• Would be easier for institutions or bootcamps to help adopt this for their students.
• As an organizer of the Data Science Club at SJSU, I would love to introduce the fundamentals with this library.

…he label was positive and visa versa; probably to avoid implementing sub. Changed the loss and demo still avoiding implementing sub but sign of score now matches label

Hi, unless I'm misunderstanding something, zero_grad in nn.py is zeroing out the gradients on the parameter nodes, but shouldn't it do it on all the nodes in the graph? Otherwise the inner nodes will keep accumulating them.

Each node points to 0 or 1 other node, never to more. There's only one unique path by which each node can be reached by going backwards from the output node. This means that in the backward() method v is never already in visited. The algorithm thus simplifies to a depth-first traversal.

Side-note: if a node could be reached by more than 1 path from the output node, the node would have a different gradient for each of the paths. So, while you could reuse nodes for the forward calculation and get correct results, there's no facility to store several gradients per node. Thus it's unnecessary to account for this possibility here.

The bug only pops up if a crazy person like me accidentally or intentionally re-initiates a value.

before the fix: a = Value(2.0) b = Value(a) print(b) output: Value(data=Value(data=2.0, grad=0), grad=0)

after the fix: a = Value(2.0) b = Value(a) print(b) output: Value(data=2.0, grad=0)

it's very minor and can be ignored but would mean a lot for my very first open-source contribution

Hello, I came across this from your YT video tutorials, thank you for making these!

In engine.py, you implement back propagation using explicit topological order computation. Are there any reasons why we would not recursively call _backward for every child ? e.g. implement backward function in Value as such:

    def backward(self):
        self._backward()
        for v in self._prev:
            v.backward()

Does it have something to do with how backprop is implemented in actual NN libraries? Is recursion harder to parallelise in practice compared to using topological ordering?

Thank you

Hi @karpathy I was solving the assignment as mentioned in the YouTube video. In the Softmax function, I was getting the following error TypeError: unsupported operand type(s) for +: 'int' and 'Value'

This is the line where I am getting the error

def softmax(logits):
  counts = [logit.exp() for logit in logits]
  denominator = sum(counts) #Here I am getting the Typeerror
  out = [c / denominator for c in counts]
  return out

And, my add function in Value Class is the following

def __add__(self, other): # exactly as in the video
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    
    def _backward():
      self.grad += 1.0 * out.grad
      other.grad += 1.0 * out.grad
    out._backward = _backward
    
    return out

So my query is on the sum of list function. It is probably similar to counts[i].add(counts[i+1]) and then we keep on adding to the result till the end of the list. So this add function should work well. But I am not sure why it is not working, am I missing something? Thanks in advance

Hi, Andrej, Thanks for this excellent library! It may be useful not only for Python developers, but for C#, F#, Pascal etc. developers too, so I wrote a C# port for .NET ecosystem. The basic info about this is here: https://github.com/ColorfulSoft/System.AI/blob/master/Docs/micrograd.NET.md Best, Gleb S. Brykin

Added note about a .NET C# reimplementation.

I know that you don't accept PRs and I don't even expect you to endorse or comment on this work but I just wanted you to know that micrograd is now living in other languages as well.

I know you don't accept pull requests, but you may want to make the change where the log shows accuracy at the point of logging. At the moment it shows the "old" accuracy.

The Layer class special cases single output neurons:

return out[0] if len(out) == 1 else out

This is nice for loss functions but currently breaks for intermediate layers with a single neuron.

from micrograd.nn import MLP
from micrograd.engine import Value

# works
model = MLP(2, [16, 16, 1])
x = [Value(1.0), Value(-2.0)]
print(model(x))

# fails: intermediate layer with 1 neuron
model = MLP(2, [16, 1, 1])
x = [Value(1.0), Value(-2.0)]
print(model(x))

This PR fixes this by correctly handling a single input Value.

Note that I currently only check for Value instances explicitly, which is good enough for intermediate layers.

To handle single network input neurons, we should also check for float and int scalars. I kept it as minimal as possible for now, but I can add that if you want. (Technically, it would probably be best to check if the input is iterable or not, but doing that right nowadays requires a try statement.)