Uncategorized
add tokens to huggingface tokenizer
Here’s a sample code of how to load huggingface tokenizer, add some custom tokens and save it.
deep learning
pytorch implementation of sinusoidal position encoding
There are existing sinusoidal position encoding modules out there, but the ones that I confronted were mostly assuming the position to be incrementing from 0 to the size of sequence. For example, when a token Read more…
deep learning
relu, gelu , swish, mish activation function comparison
RELU(2018) arxiv: https://arxiv.org/abs/1803.08375 f(x) = max(0,x) GELU(2016) despite introduced earlier than relu, in DL literature its popularity came after relu due to its characteristics that compensate for the drawbacks of relu. Like relu, gelu as Read more…