Warning: Some posts on this platform may contain adult material intended for mature audiences only. Viewer discretion is advised. By clicking ‘Continue’, you confirm that you are 18 years or older and consent to viewing explicit content.
Absolutely. They are sort of a compression scheme so the tokens contain different numbers of characters based on how frequent that string is. So common words like “the” will typically be one token, or maybe even common phrases like “I am”. On the other hand rare punctuation such as “~” may be its own token. There will also be tokens for many common prefixes and suffixes such as “non” and “n’t”. The tokens of each model are different but they definitely vary in length.
Absolutely. They are sort of a compression scheme so the tokens contain different numbers of characters based on how frequent that string is. So common words like “the” will typically be one token, or maybe even common phrases like “I am”. On the other hand rare punctuation such as “~” may be its own token. There will also be tokens for many common prefixes and suffixes such as “non” and “n’t”. The tokens of each model are different but they definitely vary in length.