ResNet
By Microsoft Research
ResNet (Residual Network) is a convolutional neural network architecture introduced by Microsoft Research in 2015 that uses "skip connections" to let very deep networks be trained effectively, solving a degradation problem that had…
Definition
ResNet (Residual Network) is a convolutional neural network architecture introduced by Microsoft Research in 2015 that uses "skip connections" to let very deep networks be trained effectively, solving a degradation problem that had previously made adding more layers to a network hurt, rather than help, accuracy.
Overview
Before ResNet, researchers observed a counterintuitive problem when training very deep convolutional neural networks: beyond a certain depth, adding more layers actually made training accuracy worse, not better — not due to overfitting, but because deep networks became harder to optimize, as gradients had to propagate through many layers during training. ResNet's key contribution was the residual, or "skip," connection: instead of forcing each stacked layer to learn a completely new transformation of its input, each block learns a residual — the difference between its input and desired output — and adds that residual back to the original input via a shortcut connection that bypasses the layers. This simple architectural change made it dramatically easier to train very deep networks; ResNet variants were successfully trained with over 100 layers, and a 152-layer version won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2015, achieving a top-5 error rate lower than human-level performance benchmarks used in the competition at the time. This was a substantial improvement over prior architectures, which had struggled to train effectively much beyond roughly 20-30 layers. ResNet's residual connection concept proved to be one of the most broadly influential ideas in deep learning, extending well beyond computer vision: variants of skip connections are now standard components in many modern architectures, including the Transformer blocks that underlie large language models, where residual connections help stabilize training of very deep stacks of attention and feed-forward layers. As a computer vision backbone, ResNet (in variants like ResNet-18, ResNet-50, and ResNet-152) remains widely used today as a pretrained feature extractor for transfer learning in image classification, object detection (including as a backbone in some YOLO variants), and segmentation tasks, valued for its strong balance of accuracy, training stability, and relative simplicity compared to newer architectures like Vision Transformers.
Key Concepts
- Introduces residual (skip) connections that bypass stacked layers
- Enables training of very deep networks (100+ layers) without degrading accuracy
- Won the 2015 ImageNet Large Scale Visual Recognition Challenge
- Available in multiple depths — ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152
- Widely used as a pretrained backbone for transfer learning in computer vision
- Residual connection concept influenced later architectures, including Transformers