pretraining
Base model education
Pretraining is the initial phase of training a machine learning model on large amounts of general data before fine-tuning it for specific tasks.
A language model might be pretrained on billions of words from the internet, learning patterns of language broadly, then later refined for particular applications like translation or summarization.
The model's pretraining took weeks on specialized hardware.
Small precise optimization adjustments
- embeddingsLearned numerical encodings
- transformerVoltage-altering electrical device
- BERTPretrained language representation model
- GPTPre-trained text generator
unsupervised · corpus · dataset · parameters · weights · representation · encoder · decoder
Foundational preparation for something
- foundationUnderlying support structure
- preliminaryBefore the main part
- orientationPosition relative to surroundings
- briefingConcise information session
onboarding · primer · familiarization · warm-up · baseline · scaffolding · setup · prerequisite
Pre-activity preparation routine
- stretchingExtending muscles or material
- drillsRepetitive training routines
- base-buildingEstablishing core structures
- periodizationDivision into distinct periods
off-season · camp · regimen · endurance · strength · aerobic · conditioning · fitness
Brain's self-rewiring ability
- memoryMind's retention capacity
- acquisitionAct of obtaining something
- retentionContinued holding or possession
- synapsesNeuron-to-neuron transmission points
cognition · learning · comprehension · priming · encoding · schema · conceptualization · mental
Diminished reaction over time
- stimulusWhat prompts a reaction
- responseAnswer to stimulus
- operantProducing an effect
- classicalOf ancient Greece/Rome
behaviorism · reinforcement · desensitization · adaptation · shaping · extinction · generalization · acclimation
fine-tuning
Small precise optimization adjustments
- embeddingsLearned numerical encodings
- transformerVoltage-altering electrical device
- BERTPretrained language representation model
GPT · unsupervised · corpus · dataset · parameters · weights · representation · encoder · decoder · tokenization
groundwork
Foundational preparation for something
- foundationUnderlying support structure
- preliminaryBefore the main part
- orientationPosition relative to surroundings
briefing · onboarding · primer · familiarization · warm-up · baseline · scaffolding · setup · prerequisite · introductory
warm-up
Pre-activity preparation routine
- stretchingExtending muscles or material
- drillsRepetitive training routines
- base-buildingEstablishing core structures
periodization · off-season · camp · regimen · endurance · strength · aerobic · conditioning · fitness · preparation
neuroplasticity
Brain's self-rewiring ability
- memoryMind's retention capacity
- acquisitionAct of obtaining something
- retentionContinued holding or possession
synapses · cognition · learning · comprehension · priming · encoding · schema · conceptualization · mental · readiness
habituation
Diminished reaction over time
classical · behaviorism · reinforcement · desensitization · adaptation · shaping · extinction · generalization · acclimation · sensitization
- machine learningneural networks, deep learning, backpropagation, underfitting, CNNs
- NLPtokenization, natural language processing, NLP, parsing, stemming
- AI conceptsneural network, artificial general intelligence, neural net, superintelligence, CNN
pretrainingfine-tuning
Small precise optimization adjustments
finetuning · adaptation · inference · deployment · evaluation
train, training, trainer, trainee, untrained, retrain, retraining, retrains, entrain