tokenizes
Breaks into pieces
Tokenizes means to break something into smaller units called tokens.
In computing, a program tokenizes text by splitting it into individual words, symbols, or meaningful pieces for processing: The parser tokenizes the input before analyzing it.
In finance, to tokenize an asset is to represent it as digital tokens on a blockchain: The company tokenizes real estate investments.
Syntax-interpreting program
- delimiterBoundary-marking character
- source codeProgramming instructions before compilation
- compilerSource-to-machine-code translator
- abstract syntax treeHierarchical code representation
lexical analysis · regular expression · regex · interpreter · preprocessor · parsing · syntax · lexeme
Smallest meaningful language units
- wordsUnits of language
- sentenceComplete thought, grammatically
- utteranceSpoken expression aloud
- segmentationDivision into sections
parsing · chunking · parts of speech · tagging · annotation · constituents · inflection · derivation
Scheduled time for activity
- credentialProof of qualification
- authenticationConfirming genuine identity
- authorizationFormal permission granted
- OAuthToken-based authorization standard
verification · digital signature · encryption · login · identity · multifactor · passcode
parser
Syntax-interpreting program
- delimiterBoundary-marking character
- source codeProgramming instructions before compilation
- compilerSource-to-machine-code translator
abstract syntax tree · lexical analysis · regular expression · regex · interpreter · preprocessor · parsing · syntax · lexeme · scanner
morphemes
Smallest meaningful language units
segmentation · parsing · chunking · parts of speech · tagging · annotation · constituents · inflection · derivation · affixes
session
Scheduled time for activity
- credentialProof of qualification
- authenticationConfirming genuine identity
- authorizationFormal permission granted
OAuth · verification · digital signature · encryption · login · identity · multifactor · passcode
tokenizesconcatenates
Joins in series
joins · merges · combines · reassembles · aggregates