tokenizer
Breaks words apart
A tokenizer is software that breaks text into smaller units called tokens—words, subwords, or characters—for processing by a computer program.
In natural language processing and large language models, tokenization is the essential first step: The tokenizer split "unhappiness" into three tokens.
Different tokenizers use different strategies, affecting how efficiently a model handles various languages and vocabulary.
Grammatical structure extraction
- segmentationDivision into sections
- natural language processingComputational analysis of speech
- NLPMachine language comprehension
- text analysisStudy of written material
lexical analysis · semantic analysis · pattern recognition · preprocessing · subword · vocabulary · embedding · transformer
Character-to-token conversion process
- parsingGrammatical structure extraction
- abstract syntax treeHierarchical code representation
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
lexeme · scanner · source code · bytecode · delimiter · terminal · grammar · context-free
Smallest meaningful language unit
- lexemeFundamental dictionary unit
- lemmaBase form, or helper theorem
- morphologyStudy of structural forms
- segmentationDivision into sections
word boundary · affix · stem · root · inflection · agglutination · orthography · grapheme
Confirming genuine identity
- OAuthToken-based authorization standard
- credentialProof of qualification
- validationConfirming truth or accuracy
- expirationEnd of validity
parsing
Grammatical structure extraction
- segmentationDivision into sections
- natural language processingComputational analysis of speech
- NLPMachine language comprehension
text analysis · lexical analysis · semantic analysis · pattern recognition · preprocessing · subword · vocabulary · embedding · transformer
lexical analysis
Character-to-token conversion process
- parsingGrammatical structure extraction
- abstract syntax treeHierarchical code representation
- compilerSource-to-machine-code translator
interpreter · lexeme · scanner · source code · bytecode · delimiter · terminal · grammar · context-free · BNF
morpheme
Smallest meaningful language unit
- lexemeFundamental dictionary unit
- lemmaBase form, or helper theorem
- morphologyStudy of structural forms
segmentation · word boundary · affix · stem · root · inflection · agglutination · orthography · grapheme · phoneme
authentication
Confirming genuine identity
- OAuthToken-based authorization standard
- credentialProof of qualification
- validationConfirming truth or accuracy
tokenizerassembler
Converts assembly to executable
aggregator · joiner · serializer · merger · composer
tokenizercode generator
Automated programming tool
compiler · interpreter · renderer · synthesizer · transpiler
tokenizerserializer
Formats data for storage
renderer · generator · synthesizer · encoder · assembler