tokenization
Breaking into pieces
Tokenization is the process of breaking something into smaller units called tokens.
In computing and natural language processing, text is tokenized into words, subwords, or characters so software can analyze it: The model's tokenization split "unbelievable" into three pieces.
In finance, tokenization converts assets into digital tokens on a blockchain, or replaces sensitive data like credit card numbers with non-sensitive placeholders for security.
Grammatical structure extraction
- segmentationDivision into sections
- stemmingRoot-form reduction process
- subwordText chunk below word-level
- morphemeSmallest meaningful language unit
lexeme · corpus · vocabulary · bigram · trigram · whitespace · delimiter · boundary
Device that reads data
- parserSyntax-interpreting program
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
- syntaxRules for arranging words
Fulfilling a financial obligation
- vaultSecure storage chamber
- PANIndian tax identifier
- maskingConcealing by covering
- encryptionConverting data into code
compliance · merchant · transaction · surrogate · privacy · obfuscation · pseudonymization
parsing
Grammatical structure extraction
- segmentationDivision into sections
- stemmingRoot-form reduction process
- subwordText chunk below word-level
morpheme · lexeme · corpus · vocabulary · bigram · trigram · whitespace · delimiter · boundary · preprocessing
scanner
Device that reads data
- parserSyntax-interpreting program
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
payment
Fulfilling a financial obligation
encryption · compliance · merchant · transaction · surrogate · privacy · obfuscation · pseudonymization
splitting
Dividing into parts
- chunkingDividing into manageable parts
- partitioningDividing into sections
- unitsStandard measures of quantity
fragments · pieces · subunits · decomposition · breaking · dividing · span · offset
- privacyprivacy, private, seclusion, confidentiality, solitude
- text editingcopy, editing, paste, undo, delete
- data privacydata privacy, GDPR, personal data, data protection, de-identification
- data sciencealgorithm, data analytics, data, machine learning, dataset
- fintechfintech, mobile banking, digital wallet, mobile payment, digital wallets
- data processingnormalization, data entry, standardization, encoding, scaling
tokenizationconcatenation
Joining items end-to-end
joining · merging · reassembly · reconstruction · synthesis
token, tokens, tokenism, tokenistic, tokenize, tokenizing, tokenized