ICLR 2017 · 2017
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean
Google Brain
What this paper is
Outrageously Large Neural Networks (Shazeer et al., ICLR 2017) introduced the sparsely-gated mixture-of-experts layer: a trainable gating network routes each input to a small subset of expert subnetworks, achieving over 1,000x more model capacity with minimal added computation. Sparse MoE later became a standard frontier-LLM efficiency technique.
Abstract summary
Introduces a sparsely-gated mixture-of-experts (MoE) layer for deep neural networks that achieves over 1,000x improvement in model capacity with minimal computational overhead. A trainable gating network routes each input to a small subset of expert sub-networks, enabling models with billions of parameters while keeping inference compute tractable. The architecture achieved state-of-the-art results on language modeling and machine translation benchmarks.
Our summary in our own words, see the canonical source links below for the original abstract.
Why we cite this paper
Mixture of Experts is the architectural foundation behind the most efficient frontier AI models (Mixtral 8x7B, DeepSeek MoE, GShard). For venture analysis, MoE adoption signals a startup's engineering sophistication: teams deploying MoE architectures achieve better cost-per-token economics, which directly impacts gross margin and unit economics. We track MoE adoption patterns in open-source AI startups as a momentum signal.
Where this matters for deal flow
Key findings
- 1Sparsely-gated MoE layers achieve 1,000x+ capacity increase with only 6-7% compute overhead by activating only a subset of experts per input.
- 2A trainable gating network learns to route inputs to the most relevant experts, balancing load across the network automatically.
- 3Models with up to 137 billion parameters were trained efficiently, establishing the viability of trillion-parameter architectures.
- 4MoE enables conditional computation: model capacity scales with parameters but inference cost scales only with active experts.
- 5The batch-domain penalty for load balancing ensures no single expert becomes a bottleneck, a technique still used in modern MoE implementations.
Canonical sources
Frequently Asked Questions
What is Mixture of Experts (MoE)?▾
A neural network architecture where multiple expert sub-networks specialize in different parts of the input space. A gating network routes each input to only the most relevant experts, allowing the model to have many parameters (high capacity) while keeping inference compute low (only a few experts activate per input).
Who wrote the original Mixture of Experts paper?▾
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean at Google Brain, published at ICLR 2017 (arXiv:1701.06538).
How does MoE reduce inference cost?▾
In a sparsely-gated MoE layer, only 1-2 experts (out of potentially hundreds) are activated for any single input. This means model capacity can be 137 billion parameters while inference compute resembles a much smaller dense model, because only the activated experts process the input.
Which modern models use Mixture of Experts?▾
Mixtral 8x7B (Mistral AI), DeepSeek MoE, GShard, Switch Transformer, and reportedly GPT-4 all use MoE architectures. The technique enables frontier-scale models at a fraction of dense-model compute cost.
Why does MoE matter for AI startups?▾
MoE architecture directly improves the cost-per-token ratio, which is the dominant cost driver for LLM-based products. Startups that deploy MoE efficiently can achieve 3-10x better gross margins compared to dense-model approaches, making them more attractive from a unit economics standpoint.
See who is building on this, before the round prices it in
We track engineering acceleration across the AI & Machine Learning and AI Infrastructure sectors this paper informs: commit velocity, contributor influx, and repo-creation pulse, surfacing breakout teams 21 to 47 days before the fundraise is public.
Five breakout startups, every Sunday, before the round gets crowded
The free Acceleration Watch: five venture-backed teams accelerating on the engineering signal, translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.
Other research papers
NeurIPS 2017 · 2017
Attention Is All You Need
NeurIPS 2020 · 2020
Language Models are Few-Shot Learners
NeurIPS 2022 · 2022
Training language models to follow instructions with human feedback
NeurIPS 2020 · 2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
ICLR 2022 · 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv preprint · 2022
Constitutional AI: Harmlessness from AI Feedback
Read our own methodology paper
Code-Side Sourcing methodology, replicable on the open dataset.
Read /methodologyRelated papers
More in Research Paper
Related topics