Semi-Supervised Learning
Hybrid labeled/unlabeled learning technique
Semi-supervised learning is a machine learning approach that trains models using a combination of a small amount of labeled data and a much larger amount of unlabeled data, aiming to achieve better performance than using the labeled data…
Definition
Semi-supervised learning is a machine learning approach that trains models using a combination of a small amount of labeled data and a much larger amount of unlabeled data, aiming to achieve better performance than using the labeled data alone.
Overview
Semi-supervised learning sits between supervised learning, which relies entirely on labeled data, and unsupervised learning, which uses none. It exploits the common real-world situation where unlabeled data is abundant and cheap to collect, while labeling it (through manual annotation, expert review, or costly experiments) is expensive and slow. By incorporating the additional structure present in the unlabeled data — such as the natural clustering or manifold shape of the data distribution — semi-supervised methods can achieve accuracy closer to fully supervised approaches while using only a fraction of the labeling effort. Common techniques include self-training (also called pseudo-labeling), where a model trained on the small labeled set generates predicted labels for the unlabeled data, and the most confident predictions are added back into the training set for further rounds of training; consistency regularization, which encourages a model to produce similar outputs for an unlabeled example under different small perturbations or augmentations of it, on the assumption that a good decision boundary should not run through a densely populated cluster of similar points; and graph-based methods, which propagate labels through a graph built from similarities between data points. Modern deep semi-supervised methods like FixMatch and MixMatch combine consistency regularization with pseudo-labeling and have achieved strong results, particularly in image classification, approaching fully supervised accuracy with a small fraction of the labels. Semi-supervised learning is widely used in domains where labeling is a genuine bottleneck: medical imaging, where expert radiologist annotation is expensive and slow; speech recognition, where transcribed audio is far scarcer than raw audio; web-scale text and image classification, where labeled examples are a tiny fraction of available data; and fraud or anomaly detection, where confirmed labeled cases of fraud are rare relative to the volume of unlabeled transactions. It is closely related to but distinct from self-supervised learning, which constructs its own supervisory labels directly from unlabeled data's inherent structure rather than combining separately labeled and unlabeled sets.
Key Concepts
- Combines a small labeled dataset with a much larger unlabeled dataset
- Aims to approach fully supervised accuracy with far less labeling effort
- Common techniques include self-training/pseudo-labeling and consistency regularization
- Graph-based methods propagate labels via similarity between data points
- Modern methods like FixMatch and MixMatch combine multiple techniques
- Distinct from but related to self-supervised learning
- Exploits natural clustering/manifold structure of unlabeled data
- Particularly effective in image classification and speech recognition