You can get the output shape of a layer by layer.output_shape. for layer in model.layers: print(layer.output_shape) Gives you: (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, 64, 64, 40) (None, None, … Read more
What is exactly attention? To be able to understand this question, we need to dive a little into certain problems which attention seeks to solve. I think one of the seminal papers on hard attention is Recurrent Models of Visual Attention and I would encourage the reader to go through that paper, even if it … Read more
TensorFlow stores all operations on an operational graph. This graph defines what functions output to where, and it links it all together so that it can follow the steps you have set up in the graph to produce your final output. If you try to input a Tensor or operation on one graph into a … Read more
In practice even in NLP you see that RNNs and CNNs are often competitive. Here’s a 2017 review paper that shows this in more detail. In theory it might be the case that RNNs can handle the full complexity and sequential nature of language better but in practice the bigger obstacle is usually properly training … Read more
Yes, you can emulate one by another, the reason for having them separate is efficiency. LSTMCell is a cell that takes arguments: Input of shape batch × input dimension; A tuple of LSTM hidden states of shape batch x hidden dimensions. It is a straightforward implementation of the equations. LSTM is a layer applying an … Read more
From RNNs in Tensorflow, a Practical Guide and Undocumented Features by Denny Britz, published in August 21, 2016. tf.nn.rnn creates an unrolled graph for a fixed RNN length. That means, if you call tf.nn.rnn with inputs having 200 time steps you are creating a static graph with 200 RNN steps. First, graph creation is slow. … Read more
Regularizers that’ll work best will depend on your specific architecture, data, and problem; as usual, there isn’t a single cut to rule all, but there are do’s and (especially) don’t’s, as well as systematic means of determining what’ll work best – via careful introspection and evaluation. How does RNN regularization work? Perhaps the best approach … Read more
The output for the LSTM is the output for all the hidden nodes on the final layer. hidden_size – the number of LSTM blocks per layer. input_size – the number of input features per time-step. num_layers – the number of hidden layers. In total there are hidden_size * num_layers LSTM blocks. The input dimensions are … Read more
TimeDistributedDense applies a same dense to every time step during GRU/LSTM Cell unrolling. So the error function will be between predicted label sequence and the actual label sequence. (Which is normally the requirement for sequence to sequence labeling problems). However, with return_sequences=False, Dense layer is applied only once at the last cell. This is normally … Read more
Setting timesteps = 1 (since, I want one timestep for each instance) and reshaping the X_train and X_test as: import numpy as np X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1])) X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1])) This worked!