Autograd is PyTorch's automatic differentiation engine. It records the operations on tensors that require a gradient and, with backward(), computes the derivatives of the loss with respect to each parameter. These gradients feed the optimizer to update the model.
Example
w = torch.tensor(2., requires_grad=True)
(w ** 2).backward() # w.grad = 4