ICLR 2022 · 2021
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang
Microsoft
What this paper is
LoRA (Hu et al., 2021) freezes a pretrained model and adds small trainable low-rank matrices to attention weights, matching full fine-tuning on benchmarks while updating only 0.1% to 1% of parameters. GPU memory and storage drop by orders of magnitude, so per-task fine-tunes ship as small shareable files.
Abstract summary
Introduces Low-Rank Adaptation (LoRA): a parameter-efficient fine-tuning technique that adds small low-rank matrices to a frozen base model. Demonstrates that LoRA matches full fine-tuning performance on multiple benchmarks while updating only 0.1%-1% of parameters. Reduces GPU memory requirements and storage footprint by orders of magnitude.
Our summary in our own words, see the canonical source links below for the original abstract.
Why we cite this paper
LoRA is the standard parameter-efficient fine-tuning method in 2026, deployed across Hugging Face's PEFT ecosystem and integrated into every major open-weight LLM serving stack. Engineering signals around LoRA adapter ecosystems are one of the cleanest measures of practical AI-application velocity in our /trend/ai-coding-tools-2026 leaderboard.
Where this matters for deal flow
Key findings
- 1LoRA matches full fine-tuning quality on benchmarks while updating only 0.1%-1% of base-model parameters.
- 2Adapters can be mixed and matched at inference time, enabling multi-tenant LLM serving.
- 3Memory and storage requirements drop by 3-10× compared to full fine-tuning.
- 4Established as the default PEFT method for open-weight models (Llama, Mistral, Qwen, Gemma).
Canonical sources
Related glossary terms
Frequently Asked Questions
What is LoRA?▾
Low-Rank Adaptation, a parameter-efficient fine-tuning method that adds small low-rank matrices to a frozen base model. See /define/lora for the full term definition.
Who created LoRA?▾
Edward J. Hu and colleagues at Microsoft, published at ICLR 2022 (arXiv:2106.09685).
How many parameters does LoRA update?▾
Only 0.1%-1% of the base model's parameters, while matching full fine-tuning quality on the paper's benchmarks and cutting GPU memory and storage requirements by roughly 3-10×.
Why is LoRA the default fine-tuning method?▾
Its parameter efficiency makes specialization cheap, and adapters can be mixed and matched at inference time to enable multi-tenant serving. It is the standard PEFT method for open-weight models like Llama, Mistral, Qwen, and Gemma.
See who is building on this, before the round prices it in
We track engineering acceleration across the AI & Machine Learning and AI Infrastructure sectors this paper informs: commit velocity, contributor influx, and repo-creation pulse, surfacing breakout teams 21 to 47 days before the fundraise is public.
Five breakout startups, every Sunday, before the round gets crowded
The free Acceleration Watch: five venture-backed teams accelerating on the engineering signal, translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.
Other research papers
NeurIPS 2017 · 2017
Attention Is All You Need
NeurIPS 2020 · 2020
Language Models are Few-Shot Learners
NeurIPS 2022 · 2022
Training language models to follow instructions with human feedback
NeurIPS 2020 · 2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
arXiv preprint · 2022
Constitutional AI: Harmlessness from AI Feedback
NeurIPS 2022 · 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Read our own methodology paper
Code-Side Sourcing methodology, replicable on the open dataset.
Read /methodologyRelated papers
More in Research Paper
Related topics