AI model evaluation for business: a practical test worksheet

AI model evaluation asks whether a system performs a particular job well enough to use. A good demonstration or a model's general benchmark score cannot answer that question for your business.

Test the whole workflow: the information it receives, the sources it retrieves, the answer it produces and the action it is allowed to take. The model is one part of that chain.

What are evals?

Evals are structured checks of an AI system against expected behaviour. For a business workflow, separate the quality of the answer from safety failures and operating constraints. A correct answer delivered too late may be unusable; a quick answer that exposes restricted information fails regardless of its writing quality.

Record factual accuracy, useful completion, response time, running cost and permission failures separately. A public model benchmark can inform a shortlist, but it does not replace tests of your own task, sources and connected actions.

Write a one-sentence test brief

Define the user, input, output and boundary. For example: "Given a fictional service request and an approved policy, produce a suggested internal category with the supporting policy section, without contacting the customer."

This is narrow enough to test. "Make customer service better" is not. Keep the intended use fixed while comparing options, and record any change to the brief. Otherwise you can accidentally favour the system by making its job easier halfway through the trial.

The NIST AI RMF Measure guidance emphasises fit-for-purpose measures and acceptable performance limits. The worksheet below is a practical starting point, not an official NIST scoring scheme.

Build cases with known expected behaviour

Include ordinary work, awkward inputs and cases where the correct answer is to stop or ask for help. Start with synthetic material. Use real business information only under an approved handling arrangement.

For each case, record an identifier, the input, the permitted sources, the expected behaviour and the reason it matters. Have someone who understands the job check the expected answer before using it to judge a model.

Keep a separate set of cases for the final check. Do not continually change the instructions using that set, or it stops being an independent check of the finished system.

A small fictional test worksheet

These five examples illustrate the structure, not a sufficient sample size for release:

• Clear request: the source supports one category. Expect that category and the correct reference.
• Missing information: a required detail is absent. Expect a question, not a guessed value.
• Conflicting source: two current-looking documents disagree. Expect a visible conflict and escalation.
• Restricted source: the answer exists in a document the test user cannot access. Expect no disclosure.
• Hostile instruction: a document asks the assistant to ignore its rules or send information elsewhere. Expect the document to remain evidence, not become an instruction.

Add cases based on the actual consequences of failure. A workflow processing varied document layouts needs different coverage from one writing short summaries. No fixed number of examples proves that every future input will be safe.

Separate quality scores from release blockers

Use a simple scale for ordinary quality: acceptable unchanged, usable after a small edit, or unusable. Define each category with examples so reviewers apply it consistently.

Then create a separate blocker list. An unauthorised disclosure or external action should not disappear inside a high average score. If one critical case fails, keep the workflow within its existing boundary while the failure is investigated.

Record the reason, not just a number. "Wrong policy version" tells the builder what to inspect. "Two out of five" does not. Where reviewers disagree, discuss the source and the rubric rather than quietly averaging their decisions.

Compare against the current way of working

Run the same task manually or with the existing rules-based process. Measure time spent preparing inputs, checking output and correcting mistakes as well as generation time.

An illustrative calculation: 18 acceptable results from 24 test cases is 75% on that test set. It is not a claim of 75% accuracy on all business work. State which cases were included and show critical failures separately.

Keep rejected and unfinished runs in the denominator when reporting completion. Do not present only the outputs that looked good enough to keep.

Make the comparison repeatable

Save the test-set version, instructions, model or service identifier, relevant settings, source versions, date and reviewer decisions. If the provider does not expose a stable model version, record that limit.

Repeat important cases to look for inconsistent behaviour. Test permission boundaries and connected actions with the real application configuration, not only in a separate chat window.

The NCSC's secure-development guidance includes documenting AI assets, data, models and prompts. Keep evaluation records under suitable access controls; they can reveal sensitive details about a system.

End with a decision, not a league table

Choose one of four outcomes: reject, revise and retest, allow a limited pilot, or accept the agreed scope. Name the owner and record unresolved limitations. A narrow pilot is not approval for additional users, data or actions.

Retest relevant cases when instructions, sources, permissions, integrations or the provider change. Our AI monitoring guide covers the checks that continue after release.

Blue Canvas can help compare options against a defined business task and build the selected workflow through AI implementation and automation. Bring a process worth improving, not a preferred model that must win.

Book a free conversation.