mutual information
Dependence between variables
Mutual information is a measure from information theory quantifying how much knowing one variable tells you about another.
If two variables are independent, their mutual information is zero; if one perfectly predicts the other, it's maximal.
The concept appears in statistics, machine learning, and cryptography wherever relationships between variables matter: We calculated the mutual information between gene expression and disease outcome.
Algorithms that improve from data
- pattern recognitionIdentifying regularities in inputs
- data miningExtracting insights from datasets
- natural language processingComputational analysis of speech
- word associationLinking concepts spontaneously
neural networks · clustering · gene expression · bioinformatics · signal processing · speech recognition · text mining · decision trees
Irish river or name
- entropyQuantified chaos in systems
- bitsSmall portions or pieces
- redundancyNeedless duplication
- data compressionReducing file size digitally
divergence · uncertainty · random variables · expected value · noise
Statistical relationship measure
- covarianceJoint deviation measure
- dependenceState of needing another
- independenceFreedom from external control
- chi-squareStatistic for categorical data
association · bivariate · multivariate · statistical significance · hypothesis testing · stochastic
machine learning
Algorithms that improve from data
- pattern recognitionIdentifying regularities in inputs
- data miningExtracting insights from datasets
- natural language processingComputational analysis of speech
word association · neural networks · clustering · gene expression · bioinformatics · signal processing · speech recognition · text mining · decision trees
Shannon
Irish river or name
- entropyQuantified chaos in systems
- bitsSmall portions or pieces
- redundancyNeedless duplication
data compression · divergence · uncertainty · random variables · expected value · noise
correlation
Statistical relationship measure
- covarianceJoint deviation measure
- dependenceState of needing another
- independenceFreedom from external control
chi-square · association · bivariate · multivariate · statistical significance · hypothesis testing · stochastic
- entropydisorder, randomness, dissipation, irreversibility, thermodynamics
- information theoryentropy, bit, bits, binary digit, redundancy
- corpus linguisticscorpus, word counts, word frequency, concordance, n-grams
- signalssignal, signals, signaling, sign, waved
- distance measureshamming, cosine, jaccard, geodesic, great circle
- feature engineeringencoding, scaling, normalization, standardization, imputation
mutual informationorthogonality
State of being independent
noise · randomness · entropy · independence
information, informed, mutual, misinformation, uninformed, informative, informing, informs, mutually