tokenisation
Breaking into units
Tokenisation is the process of breaking text into smaller units called tokens—typically words, subwords, or characters.
In natural language processing, tokenisation is a fundamental step before any text can be analysed or processed by algorithms: The model's tokenisation splits "unhappiness" into three pieces.
In finance and security, tokenisation replaces sensitive data with non-sensitive placeholders: Credit card tokenisation protects customer information.
Also spelled tokenization.
Currency substitutes
- preprocessingInitial data preparation steps
- textWritten or printed words
- parsingGrammatical structure extraction
- segmentationDivision into sections
subword · vocabulary · embedding · encoder · decoder · transformer · BERT · GPT
Smallest meaningful language units
- phonemesMinimal sound distinctions
- graphemesUnits of writing systems
- lexiconVocabulary of a language
- morphologyStudy of structural forms
inflection · derivation · syntax · semantics · segmentation · orthography · syllables · compounds
Converting data into code
- sensitive dataInformation needing protection from exposure
- vaultSecure storage chamber
- fraud preventionStopping dishonest financial activity
- authenticationConfirming genuine identity
digital payments · secure storage · credentials · obfuscation · data protection
Character-to-token conversion process
- scannerDevice that reads data
- parserSyntax-interpreting program
- compilerSource-to-machine-code translator
- interpreterOne who conveys meaning between languages
tokens · regular expressions · abstract syntax tree · source code · keywords · identifiers · literals · operators
tokens
Currency substitutes
- preprocessingInitial data preparation steps
- textWritten or printed words
- parsingGrammatical structure extraction
segmentation · subword · vocabulary · embedding · encoder · decoder · transformer · BERT · GPT · corpus
morphemes
Smallest meaningful language units
morphology · inflection · derivation · syntax · semantics · segmentation · orthography · syllables · compounds · affixes
encryption
Converting data into code
- sensitive dataInformation needing protection from exposure
- vaultSecure storage chamber
- fraud preventionStopping dishonest financial activity
authentication · digital payments · secure storage · credentials · obfuscation · data protection
lexical analysis
Character-to-token conversion process
- scannerDevice that reads data
- parserSyntax-interpreting program
- compilerSource-to-machine-code translator
interpreter · tokens · regular expressions · abstract syntax tree · source code · keywords · identifiers · literals · operators · delimiters
tokenisationconcatenation
Joining items end-to-end
joining · merging · aggregation · synthesis · combination