Sparse Mixture Model
A sparse mixture model is a neural network design in which only a small subset of the model's parameters are activated for any given input, rather than the entire network running on every token — the term overlaps closely with Mixture of Experts (MoE), the dominant technique for achieving this sparsity in modern large…
2 resources across 1 library
Glossary Terms(2)
Mixtral
Mixtral is a family of open-weight large language models from Mistral AI built on a sparse mixture-of-experts (MoE) architecture, which activates only a subset…
Sparse Mixture Model
A sparse mixture model is a neural network design in which only a small subset of the model's parameters are activated for any given input, rather than the ent…