AI & Machine Learning · sub-niche
On-device LLM runtimes.
Privacy, latency, cost, three reasons every app eventually wants a 3-8B model running on the user's machine.
Reading the two labels: team-sized build build cost means only makes sense as a team bet, multiple quarters of salary before any revenue, the kind of project incumbents are better positioned to start. Steady, one deal per month deal velocity means a round closes somewhere in this category most quarters, neither hot nor dead.
Quick take: On-device LLM runtimes is a team-sized build-cost, steady, one deal per month-velocity opportunity inside AI & Machine Learning, with 3 public reference points. Heavy lift to build, but the moat compounds, every supported architecture and platform combo adds defensibility. Fund teams with prior systems experience (compiler, kernel, graphics). Don't fund teams whose only background is fine-tuning notebooks.
Why now
Apple Silicon and consumer GPUs are now fast enough. The runtime that ships the cleanest mobile + desktop + browser experience wins the long tail of privacy-bound apps.
What the signal looks like
Repos with C++/Rust core, contributor list of WebGPU/Metal/CUDA specialists, and benchmarks against llama.cpp in the README.
Public examples
We name publicprojects + categories only, never founders we track inside the paid product. The buyer’s edge stays inside the product.
- llama.cpp forks with WebGPU bindings
- MLX-based mobile runtimes
- ONNX Runtime extensions for new architectures
What this displaces
Cloud-hosted inference for tasks that don't need it (autocomplete, redaction, transcription).
How to validate it in an afternoon
Before committing build time or a thesis memo to on-device llm runtimes, run three cheap checks against public engineering activity. Each takes minutes and none require access to private data.
- Count active builders. Search GitHub for repositories matching this category, then check how many accepted commits in the last 14 days. More than a handful of active teams means the category has energy, not just mentions.
- Look for the steady, one deal per month pattern in funding. If funded companies keep appearing here, a round closes somewhere in this category most quarters, neither hot nor dead. Cross-check the ai & machine learning leaderboard to see whether any of the accelerators sit adjacent to this niche.
- Test the team-sized build cost assumption honestly: only makes sense as a team bet, multiple quarters of salary before any revenue, the kind of project incumbents are better positioned to start. If your calendar cannot absorb that, the opportunity is real but not yours yet.
The weekly signal feed tracks 10 AI & Machine Learning sub-niches including this one, so the cohort side of this check can run continuously instead of manually.
Our build-vs-invest call
Heavy lift to build, but the moat compounds, every supported architecture and platform combo adds defensibility. Fund teams with prior systems experience (compiler, kernel, graphics). Don't fund teams whose only background is fine-tuning notebooks.
Common questions about this niche
- Isn't llama.cpp already winning?
- For desktop, mostly. Mobile, browser, and embedded are still being decided. There's room for a portable runtime above llama.cpp.
- Is this a feature of the OS?
- Apple and Google will ship their own. But the cross-platform runtime, Mac + Windows + Linux + iOS + Android + WebGPU, is a third-party slot.
- What's the wedge product?
- Usually a developer SDK first, then a consumer app that uses it as proof.
Five breakout startups, every Sunday, before the round gets crowded
The free Acceleration Watch: five venture-backed teams accelerating on the engineering signal, translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.
More inside AI & Machine Learning
- LLM eval harnesses Reproducible eval suites that an AI-native team can drop into CI and trust by lunchtime.
- Agent orchestration frameworks The 'LangChain for X' slot is still wide open, pick a vertical, ship the runtime, win the wedge.
- Retrieval-augmented search libraries RAG-as-a-library, bring-your-own embedding, bring-your-own vector store, win on developer ergonomics.
- Fine-tuning tools for non-ML teams Take fine-tuning out of the notebook. Product teams want to point at JSONL and get a deployable adapter.