tokenizing
Splitting into pieces
Tokenizing is the process of breaking something into smaller units called tokens.
In computing and linguistics, it means dividing text into individual words, phrases, or meaningful pieces for analysis: The program tokenizes each sentence before processing.
In finance and blockchain, tokenizing converts real-world assets into digital tokens that can be traded or tracked: They're tokenizing real estate to allow fractional ownership.
Grammatical structure extraction
- segmentationDivision into sections
- preprocessingInitial data preparation steps
- text miningComputational document analysis
- n-gramsContiguous linguistic subsequences
stemming · vocabulary · corpus · bigrams · trigrams · subword · whitespace · delimiters
Smallest meaningful language units
- phonemesMinimal sound distinctions
- parts of speechCategories of grammatical function
- taggingAttaching identifying markers
- morphologyStudy of structural forms
affixes · compounds · syllables · graphemes · orthography · multilingual · agglutinative · logographic
Systematic grouping by traits
- pattern recognitionIdentifying regularities in inputs
- machine learningAlgorithms that improve from data
- neural networksLayered computational learning systems
- data scienceExtracting meaning from information
preprocessing · cleaning · normalization · vectorization · embedding · batch processing · padding · truncation
Confirming genuine identity
- sessionScheduled time for activity
- OAuthToken-based authorization standard
- credentialsEvidence of qualification or identity
- authorizationFormal permission granted
encryption · validation · expiration · single sign-on · identity
parsing
Grammatical structure extraction
- segmentationDivision into sections
- preprocessingInitial data preparation steps
- text miningComputational document analysis
n-grams · stemming · vocabulary · corpus · bigrams · trigrams · subword · whitespace · delimiters · lexical analysis
morphemes
Smallest meaningful language units
- phonemesMinimal sound distinctions
- parts of speechCategories of grammatical function
- taggingAttaching identifying markers
morphology · affixes · compounds · syllables · graphemes · orthography · multilingual · agglutinative · logographic
classification
Systematic grouping by traits
- pattern recognitionIdentifying regularities in inputs
- machine learningAlgorithms that improve from data
- neural networksLayered computational learning systems
data science · preprocessing · cleaning · normalization · vectorization · embedding · batch processing · padding · truncation · encoding
authentication
Confirming genuine identity
- sessionScheduled time for activity
- OAuthToken-based authorization standard
- credentialsEvidence of qualification or identity
authorization · encryption · validation · expiration · single sign-on · identity
- organizingorganizing, categorizing, grouping, structuring, classifying
- discriminationracism, systemic discrimination, Jim Crow laws, sexism, racial harassment
- parsingparser, parsing, context-free grammar, BNF, terminals
tokenizingmerging
Blending into one
combining · concatenating · joining · aggregating · fusing
tokenizingsynthesizing
Combining elements into unity
reconstructing · reassembling · composing · generating · rendering