Uncategorized
add tokens to huggingface tokenizer
Here’s a sample code of how to load huggingface tokenizer, add some custom tokens and save it.
deep learning
pytorch implementation of sinusoidal position encoding
There are existing sinusoidal position encoding modules out there, but the ones that I confronted were mostly assuming the position to be incrementing from 0 to the size of sequence. For example, when a token embedding sequence with shape of (B, L, D_token) is given then the sinusoidal position encoding module will take this tensor as input and manually create a tensor (B,L) where the values for each row is (0,1,2,3, …., L-1) and then apply sinusoidal encoding on this.
(more…)deep learning
relu, gelu , swish, mish activation function comparison
RELU(2018) arxiv: https://arxiv.org/abs/1803.08375 f(x) = max(0,x) GELU(2016) despite introduced earlier than relu, in DL literature its popularity came after relu due to its characteristics that compensate for the drawbacks of relu. Like relu, gelu as no upper bound and bounded Read more…
python
web scraping many pdf files from websites for dataset preparation using python
There are good websites which store a lot of pdf files that can be used for data. However, it can be quite difficult to automate the downloading process with a simple web scraping tool because the data download url that the user wants resides behind another webpage. I will use IMF website for example.
(more…)pytorch
creating your own pytorch scheduler
here is an example of a scheduler that I subclassed in pytorch.
deep learning
paper review: “Donut : Document Understanding Transformer without OCR”
arxiv: https://arxiv.org/abs/2111.15664 Key Points visual document understanding model which does OCR + downstream task in one step with a single end-to-end model outputs are generative, and formatted to be convertible to JSON, which makes this architecture highly compatible to various downstream Read more…
deep learning
paper review: “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”
arxiv: https://arxiv.org/abs/1910.13461 key points propose autoregressive model named BART, which is architecturally similar to standard transformer encoder + decoder Check out 5 pretraining tasks, and experiment which pretraining task is most helpful test BART performance with large scale pretraining on downstream Read more…
deep learning
paper summary: “DocFormer: End-to-End Transformer for Document Understanding”
arxiv: https://arxiv.org/abs/2106.11539 this work proposes a backbone for visual document understanding domain. It uses text, visual, spatial features. Key points use text, visual, spatial features at each encoding layer, keep feeding in visual and spatial features on the input side. Read more…
deep learning
paper review: “LayoutLMV2: Multi-Modal Pre-training for Visually-Rich Document Understanding”
arxiv: https://arxiv.org/abs/2012.14740 LayoutLMv2 is a work improved from LayoutLM. This line of work focues on creating a model for document understanding. The main task of this genre is to extract useful information when a document is given. In this case, Read more…