hyperparameter
Learning rate, perhaps
A hyperparameter is a setting chosen before training a machine learning model, as opposed to parameters the model learns during training.
Learning rate, batch size, and number of hidden layers are hyperparameters.
Tuning them—often through trial and error—significantly affects model performance: We spent weeks optimizing hyperparameters before the model converged properly.
Distinct historical stretches
- dropoutOne who quits early
- momentumProduct of mass, velocity
- regularizationPenalty preventing model complexity
- sweepBrush clean with broom
trial · sensitivity · ablation · scheduler · pruning
Process of approaching sameness
- overfittingExcess adaptation to samples
- underfittingModel too simple
- generalizationDeriving principles from examples
- plateauFlat highland area
Stacked levels or coatings
- neuronsSignal-transmitting brain cells
- strideExtended single footstep
- poolingMerging shared contributions
- depthDistance below surface
width · bottleneck · dilation · channels
Eleventh Greek letter
- penaltyPunishment for wrongdoing
- mixupConfusing error or misunderstanding
- cutoutShape removed from material
- sparsityLack of abundance
L1 · constraint · shrinkage · ridge · lasso
Spanish missionary name
- pretrainedPreviously taught skills
- checkpointInspection station on route
- baselineStandard for comparison
- priorExisting before another
epochs
Distinct historical stretches
- dropoutOne who quits early
- momentumProduct of mass, velocity
- regularizationPenalty preventing model complexity
sweep · trial · sensitivity · ablation · scheduler · pruning
convergence
Process of approaching sameness
- overfittingExcess adaptation to samples
- underfittingModel too simple
- generalizationDeriving principles from examples
layers
Stacked levels or coatings
- neuronsSignal-transmitting brain cells
- strideExtended single footstep
- poolingMerging shared contributions
depth · width · bottleneck · dilation · channels
lambda
Eleventh Greek letter
Xavier
Spanish missionary name
- pretrainedPreviously taught skills
- checkpointInspection station on route
- baselineStandard for comparison
- Bayesian statisticslikelihood, prior, posterior, burn-in, convergence
- hyper-hyperactive, hyperlink, hyperbole, hypertension, hyperspace
- AI conceptsneural network, artificial general intelligence, neural net, superintelligence, CNN
- Regularizationdropout, ridge, lasso, shrinkage, L1
hyperparameterparameter
Boundary-setting factor
weight · coefficient · bias · embedding · prediction