lexical analysis
Breaking into tokens
Lexical analysis is the first stage of processing text in computing, where a stream of characters is broken into meaningful units called tokens.
A compiler performs lexical analysis to identify keywords, operators, and identifiers before parsing code.
The term also applies to linguistic study of vocabulary and word formation.
The lexical analysis phase flagged an unexpected character in line 47.
- tokenizerText segmentation tool
- lexemeFundamental dictionary unit
- compilingGathering into unified whole
- literalMeaning exactly what's said
- break pointMoment before potential shift
keyword · orthography · word boundary · syntax · computer science · etymology · identifier · text mining · automata · look ahead · punctuation · character set · whitespace · segmentation · programming · symbols · strings · word frequency
- NLPtokenization, natural language processing, NLP, parsing, stemming
- code analysiscode review, code smell, linting, lint, parsing
- corpus linguisticscorpus, word counts, word frequency, concordance, n-grams
- parsingparser, parsing, context-free grammar, BNF, terminals
- text algorithmspattern matching, text search, regex, BOM, grep
- text analysisbigram, trigram, tokenization, writing style, word frequency
lexical analysiscode generation
Automated software creation
semantic analysis · parsing · optimization · linking · assembly