in tensorflow, default initialization used is glorot normal initialization which is also known as xavier normal initialization. To use the same setting in pytorch, the following practice should be done. 2d convolution module example linear module example pytorch supports other initialization functions, and one can use those initialization functions in Read more…
Problem an LSTM module will be used as an example. Assume a simple net that includes an LSTM module. According to the docs, the weight and bias can be accessed by weight_ih_l[k], weight_hh_l[k], bias_ih_l[k], bias_hh_l[k]. For this example, I want to set the b_if to one. So, I could do Read more…
https://github.com/pytorch/pytorch/issues/20102 also mentioned in tensorflow docs: https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM
paper link: https://arxiv.org/pdf/2002.07141v1.pdf What is PNNL? Progressive Neural Network Learning(PNNL) is a class of algorithms that incrementally construct the network’s topology and optimize its parameters based on the training data. At each incremental training step, a PNNL algorithm adds a new set of neurons to the existing network topology and Read more…
when adding samples, I modified the reference code to exclude terminating status samples in hope that this would less complicate the batch creation process. the original code: the modified code: However, this small change made a huge difference in training convergence. The modification failed to ever get the total reward Read more…
paper link: https://arxiv.org/abs/1412.6572 This paper introduces Fast Gradient Signed Method(FGSM) adversarial attack along with some useful insights on why linearity of deep learning networks would allow such attacks. The FGSM method is regarded as the method introduced after using L-BGFS method to generate adversarial samples. These two share similar ideas Read more…
paper link: https://arxiv.org/pdf/1312.6199.pdf It is not a single node or unit of a layer that describes one kind of feature but it is the space, or all the nodes in a layer that defines some feature. In other words, it is not a single node’s value that measure a strength Read more…
While training CRNN for text prediction, I found that best path decoding predicts more properly and clearly compared to beam width. Beam width decoding results tended to be excessively messy.
paper link Key Points set forget bias to 1 when training LSTM layers to get GRU comparable results in language models, lstm is better than gru
Code: using tensorflow 1.14 The tk.keras.backend.ctc_batch_cost uses tensorflow.python.ops.ctc_ops.ctc_loss functions which has preprocess_collapse_repeated parameter. In some threads, it comments that this parameters should be set to True when the tf.keras.backend.ctc_batch_cost function does not seem to work, such as inconverging loss. However, my experience is that although setting this parameter to True Read more…