tokenize
Break into pieces
Tokenize means to break something into smaller units called tokens.
In computing and linguistics, text is tokenized into words, phrases, or symbols for processing: The parser tokenizes the input before analysis.
In finance, to tokenize an asset is to represent it as digital tokens on a blockchain: The company tokenized its real estate holdings.
Credit card systems tokenize sensitive data by replacing card numbers with unique identifiers for security.
Examine syntax closely
- scannerDevice that reads data
- delimiterBoundary-marking character
- compilerSource-to-machine-code translator
- preprocessorPre-compilation directive handler
Separation into parts
- segmentDistinct part of whole
- separateTo set apart
- partitionBarrier separating spaces
- chunkThick solid piece
divide · break-up · decompose · parse · unit · piece · fragment · portion
Assembled written works
- preprocessingInitial data preparation steps
- stemmingRoot-form reduction process
- chunkingDividing into manageable parts
- textWritten or printed words
whitespace · punctuation · subword · vocabulary · embedding · BERT · transformer · segmentation
Smallest meaningful language unit
- lexemeFundamental dictionary unit
- phonemeMinimal distinctive sound unit
- syllableUnit of pronunciation
- wordLinguistic meaning unit
affix · prefix · suffix · root · stem · inflection · grapheme · orthography
Cryptographic linked transaction record
- transactionExchange of goods/money
- walletSmall case for currency
- cryptocurrencyEncrypted decentralized money
- NFTBlockchain-verified digital ownership
minting · tokenomics · fungible · non-fungible · staking · airdrop · burn · supply
Secure storage chamber
- secureProtected from harm
- maskingConcealing by covering
- merchantOne who sells wares
- transactionExchange of goods/money
cardholder · PAN · encryption · checkout · Stripe
Authentication string
- credentialProof of qualification
- sessionScheduled time for activity
- OAuthToken-based authorization standard
- authenticationConfirming genuine identity
authorization · identity · verification · secure · encrypted · hash · salt · expiration
parse
Examine syntax closely
- scannerDevice that reads data
- delimiterBoundary-marking character
- compilerSource-to-machine-code translator
preprocessor · grammar · terminal · lookahead · tokenizer
split
Separation into parts
chunk · divide · break-up · decompose · parse · unit · piece · fragment · portion · element
corpus
Assembled written works
- preprocessingInitial data preparation steps
- stemmingRoot-form reduction process
- chunkingDividing into manageable parts
text · whitespace · punctuation · subword · vocabulary · embedding · BERT · transformer · segmentation
morpheme
Smallest meaningful language unit
word · affix · prefix · suffix · root · stem · inflection · grapheme · orthography · segmentation
blockchain
Cryptographic linked transaction record
- transactionExchange of goods/money
- walletSmall case for currency
- cryptocurrencyEncrypted decentralized money
NFT · minting · tokenomics · fungible · non-fungible · staking · airdrop · burn · supply · decentralization
vault
Secure storage chamber
transaction · cardholder · PAN · encryption · checkout · Stripe
password
Authentication string
- credentialProof of qualification
- sessionScheduled time for activity
- OAuthToken-based authorization standard
authentication · authorization · identity · verification · secure · encrypted · hash · salt · expiration · validation
- computing verbsrun, execute, process, compute, calculate
- encodingcipher, UTF-8, encryption, binary, ASCII
- string functionsreplace, substring, concatenate, trim, strip
- degradebelittle, demean, humiliate, degrade, debase
- security actionsauthenticate, authorize, detect, defend, encrypt
- data operationsedit, backup, modify, fetch, read
tokenizereconstruct
Build again from remains
synthesize · generate · render · compose · serialize