The problem is from my training loop: it doesn’t detach or repackage the hidden state in between batches? If so, then loss.backward() is trying to back-propagate all the way through to the start of time, which works for the first batch but not for the second because the graph for the first batch has been … Read more
TL;DR: Use both. They do different things, and have different scopes. with torch.no_grad – disables tracking of gradients in autograd. model.eval() changes the forward() behaviour of the module it is called upon eg, it disables dropout and has batch norm use the entire population statistics with torch.no_grad The torch.autograd.no_grad documentation says: Context-manager that disabled [sic] … Read more
tensor.detach() creates a tensor that shares storage with tensor that does not require grad. It detaches the output from the computational graph. So no gradient will be backpropagated along this variable. The wrapper with torch.no_grad() temporarily set all the requires_grad flag to false. torch.no_grad says that no operation should build the graph. The difference is … Read more