@cleros is pretty on the point about the use of retain_graph=True. In essence, it will retain any necessary information to calculate a certain variable, so that we can do backward pass on it. An illustrative example Suppose that we have a computation graph shown above. The variable d and e is the output, and a … Read more
Do you know about “vanishing” and “exploding” gradients in backpropagation? I’m not too familiar with Haskell so I can’t easily see what exactly your backprop is doing, but it does look like you are using a logistic curve as your activation function. If you look at the plot of this function you’ll see that the … Read more
I think that biases are almost always helpful. In effect, a bias value allows you to shift the activation function to the left or right, which may be critical for successful learning. It might help to look at a simple example. Consider this 1-input, 1-output network that has no bias: The output of the network … Read more