Do you know about “vanishing” and “exploding” gradients in backpropagation? I’m not too familiar with Haskell so I can’t easily see what exactly your backprop is doing, but it does look like you are using a logistic curve as your activation function. If you look at the plot of this function you’ll see that the … Read more
Units: The amount of “neurons”, or “cells”, or whatever the layer has inside it. It’s a property of each layer, and yes, it’s related to the output shape (as we will see later). In your picture, except for the input layer, which is conceptually different from other layers, you have: Hidden layer 1: 4 units … Read more
Judging from the examples you provide, I’m assuming that by ANNs, you mean multilayer feed-forward networks (FF nets for short), such as multilayer perceptrons, because those are in direct competition with SVMs. One specific benefit that these models have over SVMs is that their size is fixed: they are parametric models, while SVMs are non-parametric. … Read more
Logits is an overloaded term which can mean many different things: In Math, Logit is a function that maps probabilities ([0, 1]) to R ((-inf, inf)) Probability of 0.5 corresponds to a logit of 0. Negative logit correspond to probabilities less than 0.5, positive to > 0.5. In ML, it can be the vector of … Read more
In the neural network terminology: one epoch = one forward pass and one backward pass of all the training examples batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you’ll need. number of iterations = number of passes, each pass using [batch size] … Read more
I think that biases are almost always helpful. In effect, a bias value allows you to shift the activation function to the left or right, which may be critical for successful learning. It might help to look at a simple example. Consider this 1-input, 1-output network that has no bias: The output of the network … Read more