regularized – Tarik Billa

RNN Regularization: Which Component to Regularize?

December 20, 2023 by Tarik

Regularizers that’ll work best will depend on your specific architecture, data, and problem; as usual, there isn’t a single cut to rule all, but there are do’s and (especially) don’t’s, as well as systematic means of determining what’ll work best – via careful introspection and evaluation. How does RNN regularization work? Perhaps the best approach … Read more

TensorFlow – regularization with L2 loss, how to apply to all weights, not just last one?

March 4, 2023 by Tarik

A shorter and scalable way of doing this would be ; vars = tf.trainable_variables() lossL2 = tf.add_n([ tf.nn.l2_loss(v) for v in vars ]) * 0.001 This basically sums the l2_loss of all your trainable variables. You could also make a dictionary where you specify only the variables you want to add to your cost and … Read more

L1/L2 regularization in PyTorch

January 17, 2023 by Tarik

Use weight_decay > 0 for L2 regularization: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)