Learn · 2

Autograd

Mark a tensor with requiresGrad, do arithmetic, call backward(), read .grad. Backward is synchronous — only reading the number back is awaited.

The smallest possible gradient

Pull w toward 5 and check the derivative by hand: d/dw (w − 5)² = 2(w − 5), which at w = 3 is −4.

Gradients accumulate

Calling backward() twice adds into .grad rather than replacing it — the same as torch, and the reason every training loop starts with zeroGrad(). Try running this block twice in a row.

Turning it off

noGrad(fn) runs a function with the tape switched off — evaluation, and anything an optimizer does to parameters in place. There is also gradMode for callers that need to hold the switch themselves; a binding in another language cannot pass a callback across the boundary.

Chains, not just single steps

The graph is built as you go — eager, like torch. Every operation that has a backward rule records its parents, and backward() walks them.

The gradient, drawn

autograd does not give one slope, it gives the slope at every input. Here is d/dx x² plotted across x — the straight line 2x, read straight from .grad.

What to remember

• Mark a leaf with requiresGrad, build an expression, call backward(), read .grad.
• Gradients accumulate — zero them (zeroGrad) between steps or they add up.
• noGrad turns tracking off for inference.
• Backward is synchronous; only reading .grad back is awaited.