https://arxiv.org/ftp/arxiv/papers/1912/1912.07016.pdf TLDR: key points this paper proposes to use densent+CTC for text recognition, and in that process propose some modifications to original densenet which are 1) new block, called Fast Dense Block(FDB) 2) FDenseNet-U: fast densenet + upsampling block 3) use convolution layer with stride 2 instead of maxpooling 4) Read more…
https://arxiv.org/pdf/1708.07120.pdf key idea in one sentence get min/max learning rate from LR range test, and run one cycle of cyclical learning rate at the start of training to efficiently train networks to find super convergence. Notes This paper uses a simplification of the second order Hessian-Free optimization to estimate optimal Read more…
paper link: https://arxiv.org/pdf/1911.09070.pdf BiFPN multiple bifpn layers for scaling use depth-wise convolution layers bidirectional cross-scale connections + weighted feature fusion. weighted feature fusion different weight for each resolution features learnable weights summarize that there are three different approaches for doing weighted feature fusion unbounded fusion: because it is unbounded, can Read more…
pdf link: https://arxiv.org/pdf/1910.09792.pdf Key Idea deep neural networks learn to generalize but also memorize noisy cases, which is bad. if the DNN is generalized to clean samples, then adding some perturbation to these samples would not increase loss significantly. But for memorized noisy cases, adding some perturbation would increase loss Read more…
in tensorflow, default initialization used is glorot normal initialization which is also known as xavier normal initialization. To use the same setting in pytorch, the following practice should be done. 2d convolution module example linear module example
Problem an LSTM module will be used as an example. Assume a simple net that includes an LSTM module. According to the docs, the weight and bias can be accessed by weight_ih_l[k], weight_hh_l[k], bias_ih_l[k], bias_hh_l[k]. For this example, I want to set the b_if to one. So, I could do Read more…
https://github.com/pytorch/pytorch/issues/20102 also mentioned in tensorflow docs: https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM
paper link: https://arxiv.org/pdf/2002.07141v1.pdf What is PNNL? Progressive Neural Network Learning(PNNL) is a class of algorithms that incrementally construct the network’s topology and optimize its parameters based on the training data. At each incremental training step, a PNNL algorithm adds a new set of neurons to the existing network topology and Read more…
when adding samples, I modified the reference code to exclude terminating status samples in hope that this would less complicate the batch creation process. the original code: the modified code: However, this small change made a huge difference in training convergence. The modification failed to ever get the total reward Read more…
paper link: https://arxiv.org/abs/1412.6572 This paper introduces Fast Gradient Signed Method(FGSM) adversarial attack along with some useful insights on why linearity of deep learning networks would allow such attacks. The FGSM method is regarded as the method introduced after using L-BGFS method to generate adversarial samples. These two share similar ideas Read more…