Chadrick's Blog

  • Posts

tokens

Uncategorized

add tokens to huggingface tokenizer

Here’s a sample code of how to load huggingface tokenizer, add some custom tokens and save it.

By admin, 4 weeks ago
Recent Posts
  • add tokens to huggingface tokenizer
  • pytorch implementation of sinusoidal position encoding
  • relu, gelu , swish, mish activation function comparison
  • web scraping many pdf files from websites for dataset preparation using python
  • creating your own pytorch scheduler
Categories
  • algorithm
  • c/c++
  • cuda
  • data science
  • deep learning
  • error fix
  • kicad
  • kotlin
  • linux
  • postgres
  • python
  • pytorch
  • tensorflow
  • Uncategorized
  • web
Recent Comments
  • S on error fix: onnxruntime “Type Error: Type ‘tensor(int64)’ of input parameter of operator(Min) in node is invalid’
  • Berowne on error fix: onnxruntime “Type Error: Type ‘tensor(int64)’ of input parameter of operator(Min) in node is invalid’
  • RaphaĆ«l on error fix: onnxruntime “Type Error: Type ‘tensor(int64)’ of input parameter of operator(Min) in node is invalid’
  • Ahmed on fixing return data array from apollo client is duplicate of first item in array
  • Anonymous on error fix: onnxruntime “Type Error: Type ‘tensor(int64)’ of input parameter of operator(Min) in node is invalid’
Archives
  • June 2022
  • May 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021
  • May 2021
  • April 2021
  • March 2021
  • February 2021
  • January 2021
  • December 2020
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • February 2020
  • January 2020
  • December 2019
  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
Meta
  • Register
  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org
  • Posts
Hestia | Developed by ThemeIsle