SparseCategoricalCrossentropy and CategoricalCrossentropy both compute categorical cross-entropy. The only difference is in how the targets/labels should be encoded. When using SparseCategoricalCrossentropy the targets are represented by the index of the category (starting from 0). Your outputs have shape 4×2, which means you have two categories. Therefore, the targets should be a 4 dimensional vector with … Read more
If there’s a mask in your model, it’ll be propagated layer-by-layer and eventually applied to the loss. So if you’re padding and masking the sequences in a correct way, the loss on the padding placeholders would be ignored. Some Details: It’s a bit involved to explain the whole process, so I’ll just break it down … Read more
There is just one cross (Shannon) entropy defined as: H(P||Q) = – SUM_i P(X=i) log Q(X=i) In machine learning usage, P is the actual (ground truth) distribution, and Q is the predicted distribution. All the functions you listed are just helper functions which accepts different ways to represent P and Q. There are basically 3 … Read more
When you use a custom loss, you need to put it without quotes, as you pass the function object, not a string: def root_mean_squared_error(y_true, y_pred): return K.sqrt(K.mean(K.square(y_pred – y_true))) model.compile(optimizer = “rmsprop”, loss = root_mean_squared_error, metrics =[“accuracy”])
From model documentation: loss: String (name of objective function) or objective function. See losses. If the model has multiple outputs, you can use a different loss on each output by passing a dictionary or a list of losses. The loss value that will be minimized by the model will then be the sum of all … Read more
For me, the issue occurred when upgrading from numpy 1.19 to 1.20 and using ray‘s RLlib, which uses tensorflow 2.2 internally. Simply downgrading with pip install numpy==1.19.5 solved the problem; the error did not occur anymore. Update (comment by @codeananda): You can also update to a newer TensorFlow (2.6+) version now that resolves the problem … Read more
Use weight_decay > 0 for L2 regularization: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)
Regression with neural networks is hard to get working because the output is unbounded, so you are especially prone to the exploding gradients problem (the likely cause of the nans). Historically, one key solution to exploding gradients was to reduce the learning rate, but with the advent of per-parameter adaptive learning rate algorithms like Adam, … Read more