AI & Machine Learning · sub-niche
LLM eval harnesses.
Reproducible eval suites that an AI-native team can drop into CI and trust by lunchtime.
Reading the two labels: month-long build build cost means one focused builder needs roughly a month of full-time work before the tool is usable by a stranger. Hot, multiple deals per month deal velocity means multiple funded companies are landing in this category per quarter right now.
Quick take: LLM eval harnesses is a month-long build-cost, hot, multiple deals per month-velocity opportunity inside AI & Machine Learning, with 3 public reference points. Build it as a vertical eval (legal, medical, code review) rather than a general harness, the general slot is crowded. The signal that something is breaking out: a single vertical's eval repo getting starred by three or more competing product teams in the same week.
Why now
Every model swap (GPT → Claude → Gemini → Llama variant) breaks the prompt graph. Teams need an eval layer that survives provider churn.
What the signal looks like
Repos crossing 1k stars inside a quarter, with the contributor list dominated by ML platform engineers from infra-heavy companies, not researchers.
Public examples
We name publicprojects + categories only, never founders we track inside the paid product. The buyer’s edge stays inside the product.
- Promptfoo-style YAML eval harnesses
- DeepEval-style pytest plugins
- OpenAI Evals forks tuned to a single vertical
What this displaces
Hand-rolled notebook eval scripts and the prompt engineer's weekly Excel sheet.
How to validate it in an afternoon
Before committing build time or a thesis memo to llm eval harnesses, run three cheap checks against public engineering activity. Each takes minutes and none require access to private data.
- Count active builders. Search GitHub for repositories matching this category, then check how many accepted commits in the last 14 days. More than a handful of active teams means the category has energy, not just mentions.
- Look for the hot, multiple deals per month pattern in funding. If funded companies keep appearing here, multiple funded companies are landing in this category per quarter right now. Cross-check the ai & machine learning leaderboard to see whether any of the accelerators sit adjacent to this niche.
- Test the month-long build cost assumption honestly: one focused builder needs roughly a month of full-time work before the tool is usable by a stranger. If your calendar cannot absorb that, the opportunity is real but not yours yet.
The weekly signal feed tracks 10 AI & Machine Learning sub-niches including this one, so the cohort side of this check can run continuously instead of manually.
Our build-vs-invest call
Build it as a vertical eval (legal, medical, code review) rather than a general harness, the general slot is crowded. The signal that something is breaking out: a single vertical's eval repo getting starred by three or more competing product teams in the same week.
Common questions about this niche
- Why is an eval harness a niche, not a feature of every LLM tool?
- Because evals are model-agnostic and product-agnostic, they belong in a separate layer that survives provider swaps. Teams that bury evals inside a single product end up with brittle CI.
- Should I build or fund?
- Build if you already have the vertical's golden dataset. Fund if you don't, the moat is data, not framework.
- What's the GitHub signal that an eval repo is going to raise?
- Star velocity is a weak signal; what matters is whether engineers from three or more named product companies are filing issues in the same month.
Five breakout startups, every Sunday, before the round gets crowded
The free Acceleration Watch: five venture-backed teams accelerating on the engineering signal, translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.
More inside AI & Machine Learning
- Agent orchestration frameworks The 'LangChain for X' slot is still wide open, pick a vertical, ship the runtime, win the wedge.
- Retrieval-augmented search libraries RAG-as-a-library, bring-your-own embedding, bring-your-own vector store, win on developer ergonomics.
- Fine-tuning tools for non-ML teams Take fine-tuning out of the notebook. Product teams want to point at JSONL and get a deployable adapter.
- On-device LLM runtimes Privacy, latency, cost, three reasons every app eventually wants a 3-8B model running on the user's machine.