The main difference between special tokens and normal tokens is their function. Normal tokens, such as words or parts of words, make up the actual content of text.
They operate over structured conversations and tool interactions. Special tokens are how that structure is encoded. Role Tokens and Chat Templates.
OpenAI uses two main BPE encodings: cl100k_base (GPT-3.5, GPT-4) and o200k_base (GPT-4o and later). Each has its own set of special tokens with different IDs
As we can see from the illustration, And Special Tokens has many fascinating aspects to explore.
"programmatically filter out the special tokens during token predictions" - could you clarify what you mean by special tokens then (examples maybe)?
Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from...
To enable effective learning and prediction, the implementation uses special tokens to demarcate history, prediction intervals, and temporal alignment
TeX creates and uses those special token values to help with processing your macro call, as well explore and explain below. A brief word on how token lists are stored: nodes.
![Image showcasing The architecture of our model. [CLS] is a special token placed at the ...](https://www.researchgate.net/publication/365781166/figure/fig1/AS:11431281103090395@1669607466200/The-architecture-of-our-model-CLS-is-a-special-token-placed-at-the-beginning-of-each_Q640.jpg "The architecture of our model. [CLS] is a special token placed at the ...")
