Autoencoder
An autoencoder is a type of neural network trained to reconstruct its own input by first compressing it into a smaller latent representation (encoding) and then reconstructing the original data from that representation (decoding).
Definition
An autoencoder is a type of neural network trained to reconstruct its own input by first compressing it into a smaller latent representation (encoding) and then reconstructing the original data from that representation (decoding).
Overview
An autoencoder consists of two connected networks: an encoder that maps the input to a lower-dimensional latent space (often called a bottleneck), and a decoder that reconstructs the original input from that compressed representation. The network is trained end-to-end by minimizing a reconstruction loss — typically mean squared error for continuous data — that measures how closely the decoder's output matches the original input. Because the bottleneck is smaller than the input dimensionality, the network is forced to learn a compressed, information-dense encoding that captures the most important structure in the data rather than simply memorizing it, functioning as a nonlinear, learned analog to PCA. Several autoencoder variants address specific goals. Denoising autoencoders are trained to reconstruct clean input from a corrupted or noisy version, forcing the model to learn robust features rather than trivial identity mappings. Sparse autoencoders add a regularization penalty encouraging most latent units to be inactive, promoting more interpretable, disentangled representations. Variational autoencoders (VAEs) extend the basic architecture with a probabilistic latent space, enabling them to generate new, plausible samples rather than just reconstructing inputs, which basic autoencoders cannot reliably do since their latent space isn't structured for smooth sampling. Autoencoders are used for anomaly detection, since data that reconstructs poorly is likely different from the training distribution; for dimensionality reduction and feature learning, as an alternative to PCA when data has nonlinear structure; for image denoising and inpainting; and as a pretraining or representation-learning step before fine-tuning on a downstream supervised task. While plain autoencoders have been partly superseded by other self-supervised pretraining techniques for large-scale representation learning, the encoder-decoder architecture remains foundational, underpinning both VAEs and, more distantly, the broader class of generative models that build on learned latent representations.
Key Concepts
- Encoder-decoder architecture trained to reconstruct its own input
- Compresses data into a smaller latent bottleneck representation
- Trained by minimizing reconstruction loss, typically without labeled data
- Denoising variants learn robust features by reconstructing from corrupted input
- Sparse variants encourage interpretable, disentangled latent representations
- Functions as a nonlinear, learned analog to PCA for dimensionality reduction
- Used for anomaly detection based on reconstruction error
- Foundational architecture underlying variational autoencoders