tokenized
Broken into pieces
Tokenized describes something that has been broken into smaller units called tokens.
In computing and language processing, text gets tokenized into individual words or subword pieces for analysis: The model tokenized the sentence before processing it.
In finance, tokenized assets are represented as digital tokens on a blockchain: tokenized real estate, tokenized securities.
The underlying idea is conversion into discrete, manageable units.
Fundamental dictionary unit
- morphemeSmallest meaningful language unit
- lemmaBase form, or helper theorem
- phonemeMinimal distinctive sound unit
- graphemeMinimal written symbol unit
polysemy · homonymy · synonymy · antonymy · collocation · etymology · discourse · phraseology
Dividing into manageable parts
- parsingGrammatical structure extraction
- delimitingFixing outer limits
- splittingDividing into parts
- partitioningDividing into sections
breaking · separating · subdividing · fragment · span · slice · segment
Syntax-interpreting program
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
- scannerDevice that reads data
- lexical analysisCharacter-to-token conversion process
abstract syntax tree · delimiter · string · regex · pattern matching · source code · preprocessor · API
Manipulating written digital content
- vocabularyOne's word collection
- embeddingPlacing within surrounding mass
- transformerVoltage-altering electrical device
- BERTPretrained language representation model
GPT · neural network · preprocessing · corpus · training data · inference · sequence · padding
lexeme
Fundamental dictionary unit
- morphemeSmallest meaningful language unit
- lemmaBase form, or helper theorem
- phonemeMinimal distinctive sound unit
grapheme · polysemy · homonymy · synonymy · antonymy · collocation · etymology · discourse · phraseology · corpus
chunking
Dividing into manageable parts
- parsingGrammatical structure extraction
- delimitingFixing outer limits
- splittingDividing into parts
partitioning · breaking · separating · subdividing · fragment · span · slice · segment
parser
Syntax-interpreting program
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
- scannerDevice that reads data
lexical analysis · abstract syntax tree · delimiter · string · regex · pattern matching · source code · preprocessor · API · JSON
text processing
Manipulating written digital content
- vocabularyOne's word collection
- embeddingPlacing within surrounding mass
- transformerVoltage-altering electrical device
BERT · GPT · neural network · preprocessing · corpus · training data · inference · sequence · padding · truncation
- wrongedabused, mistreated, victimized, exploited, oppressed
- anonymizedanonymized, pseudonymized, redacted, masked, censored
tokenizedcontinuous
Without pause or gap
unsegmented · whole · unprocessed · raw · undivided
tokenizedphysical
Of body or matter
non-digital · traditional · conventional · analog · paper-based