pytorch

How do I initialize weights in PyTorch?

by

Single layer To initialize the weights of a single layer, use a function from torch.nn.init. For instance: conv1 = torch.nn.Conv2d(…) torch.nn.init.xavier_uniform(conv1.weight) Alternatively, you can modify the parameters by writing to conv1.weight.data (which is a torch.Tensor). Example: conv1.weight.data.fill_(0.01) The same applies for biases: conv1.bias.data.fill_(0.01) nn.Sequential or custom nn.Module Pass an initialization function to torch.nn.Module.apply. It will … Read more

Check the total number of parameters in a PyTorch model

by

PyTorch doesn’t have a function to calculate the total number of parameters as Keras does, but it’s possible to sum the number of elements for every parameter group: pytorch_total_params = sum(p.numel() for p in model.parameters()) If you want to calculate only the trainable parameters: pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad) Answer inspired … Read more

Why do we “pack” the sequences in PyTorch?

by

I have stumbled upon this problem too and below is what I figured out. When training RNN (LSTM or GRU or vanilla-RNN), it is difficult to batch the variable length sequences. For example: if the length of sequences in a size 8 batch is [4,6,8,5,4,3,7,8], you will pad all the sequences and that will result … Read more

What does model.train() do in PyTorch?

by

model.train() tells your model that you are training the model. This helps inform layers such as Dropout and BatchNorm, which are designed to behave differently during training and evaluation. For instance, in training mode, BatchNorm updates a moving average on each new batch; whereas, for evaluation mode, these updates are frozen. More details: model.train() sets … Read more

What does model.eval() do in pytorch?

by

model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn off them during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() … Read more

How do I print the model summary in PyTorch?

by

Yes, you can get exact Keras representation, using the pytorch-summary package. Example for VGG16: from torchvision import models from torchsummary import summary vgg = models.vgg16() summary(vgg, (3, 224, 224)) —————————————————————- Layer (type) Output Shape Param # ================================================================ Conv2d-1 [-1, 64, 224, 224] 1,792 ReLU-2 [-1, 64, 224, 224] 0 Conv2d-3 [-1, 64, 224, 224] 36,928 … Read more

Why do we need to call zero_grad() in PyTorch?

by

In PyTorch, for every mini-batch during the training phase, we typically want to explicitly set the gradients to zero before starting to do backpropragation (i.e., updating the Weights and biases) because PyTorch accumulates the gradients on subsequent backward passes. This accumulating behaviour is convenient while training RNNs or when we want to compute the gradient … Read more

How do I save a trained model in PyTorch?

by

Found this page on their github repo: Recommended approach for saving a model There are two main approaches for serializing and restoring a model. The first (recommended) saves and loads only the model parameters: torch.save(the_model.state_dict(), PATH) Then later: the_model = TheModelClass(*args, **kwargs) the_model.load_state_dict(torch.load(PATH)) The second saves and loads the entire model: torch.save(the_model, PATH) Then later: … Read more