Evaluating AI features you actually ship.
Why offline benchmarks aren't enough, and how we build evaluation harnesses that live in the same repo as the product.
Public benchmarks tell you very little about whether your AI feature is going to work for your users on your data. The gap between benchmark score and shipped product quality is where most AI initiatives quietly die.
Evals as first-class artifacts
We build evaluation harnesses the same way we build test suites — versioned, runnable in CI, and tied to a specific version of the product. When a prompt changes, the eval runs. When the model changes, the eval runs. Regression is caught before users see it.
The dataset is the moat
The interesting asset is rarely the model. It's the evaluation dataset — the carefully curated examples that encode what 'good' actually means for this product. That dataset compounds. Everything else is replaceable.