Research
Production AI evaluation harness
A concept for repeatable testing of AI models and agents against agreed quality and risk criteria, before release and after changes.
- Updated:
- What it is
- A concept for a reusable evaluation approach: representative test cases and agreed criteria that an AI capability is measured against before release and re-measured when anything material changes.
- What it is not
- Not a product, not a benchmark and not evidence that any particular AI system is safe or effective.
The question
How do you know an AI capability is good enough for live use, and that it is still good enough after the model, prompt or data changes?
What the harness would check
- Task success against a clearly defined metric.
- Behaviour on representative test cases, including difficult and harmful inputs.
- Grounding and retrieval quality where the system uses retrieval.
- Failure and fallback behaviour, and when a person is brought in.
- Cost and latency against agreed budgets.
- Regression when the model, provider, prompt or data source changes.
Why it matters
Prototype evidence does not automatically mean production readiness. Repeatable evaluation turns that judgement into evidence, and keeps it current as models and context change.
Known limitations
This page describes a concept drafted from Vitalos's own delivery methods. It makes no claim about any implementation, results or client use; none are published here. No test suites, datasets or results exist on this page, and how it would integrate with specific model providers has not been designed.