vanishing gradient
Backpropagation's fading signal
Vanishing gradient is a problem in training deep neural networks where gradients become extremely small as they propagate backward through many layers, effectively preventing earlier layers from learning.
The network's weights stop updating meaningfully, stalling the training process.
The opposite problem, exploding gradient, occurs when gradients grow uncontrollably large.
Various architectural innovations—residual connections, different activation functions, normalization techniques—were developed specifically to address vanishing gradients.
- backpropagationReverse gradient computation method
- explodingBursting outward violently
- deep learningLayered neural network methods
- sigmoidCurving in two directions
- deepFar from surface
hidden layers · layers · neural · network · saturation · activation · weights · training · convergence · skip · connections · dying · neurons · loss · chain rule · chain · rule · propagation
- machine learningneural networks, deep learning, backpropagation, underfitting, CNNs
- neural networksneural network, neural networks, hidden layers, backpropagation, layers