Has AI hit a plateau? Test the claim against your workflow

“Has AI hit a plateau?” can describe several different problems: a benchmark stops improving, a product feels much the same, or a business gets little additional value from its setup. These questions need different evidence. A disappointing workflow does not establish what is happening across AI research.

Say what you think has stopped improving

Name the task and outcome. Is the system still missing facts in long documents? Is the team spending the same time reviewing drafts? Has a connected workflow become less reliable? Define the limit in terms you can test.

Keep capability, product usability and business benefit separate. A better answer may still take too long to review. A convenient interface may still use the wrong source. An expensive model may offer no useful improvement on a task that ordinary rules already handle.

Use the same test before and after a change

Keep a small set of representative examples with checked expected results. Record the model or product version, enabled tools, instructions and source material. Re-run the examples after a meaningful change.

If the source documents, prompt, reviewers and workload all change at once, it becomes difficult to explain a difference. Record those changes rather than attributing every movement to the model. Retain difficult cases as well as successful ones.

Find the bottleneck before replacing the model

• Source quality: is the correct, current information available?
• Permissions: can the authorised user access it without exposing restricted material?
• Instructions: is the required output and its boundary clear?
• Workflow: does the result reach the right person or system?
• Review: can somebody check it efficiently against the evidence?
• Recovery: can waiting work be handled when a dependency fails?

A fictional plateau investigation

Imagine a team using an assistant to draft answers to delivery questions. Changing models has not reduced corrections. A review finds that the knowledge folder contains two conflicting delivery policies.

The next experiment should clarify which policy is current, give it an owner and rerun the same questions. Buying a more expensive model before fixing that conflict may leave the original problem intact. This is an illustrative diagnosis, not a Blue Canvas case study or proof about a model’s performance.

Change one useful part at a time

Try a clearer source set, a narrower task, a better output format or a different review screen. Measure whether the change improves correctness and the total effort needed. Keep the previous version available so the team can recover if the experiment fails.

A more elaborate agent workflow is another hypothesis to test. Additional tools and steps can introduce new failure points. Assess the end-to-end result, including what was actually accepted by the destination system.

Do not turn a benchmark into a business guarantee

A published benchmark measures a defined task under particular conditions. Check what was tested, which version was used and whether the evaluation resembles your work. Headline scores may be useful for shortlisting, but your acceptance evidence still needs to match your own task.

The NIST AI Risk Management Framework offers a structured way to consider AI risks and evaluation. It does not certify a vendor or predict that your project will create a return.

Decide whether to improve, narrow or stop

Continue when the evidence shows a worthwhile improvement and the team can maintain it. Narrow the task when a limited use works but a wider one creates unreliable outputs. Stop when review, errors or operating cost outweigh the benefit.

A decision to keep the current tool or remove an AI step can be a useful result. The objective is a dependable business process, not adopting every new model announcement. Use our AI evaluation worksheet to keep the comparison clear.

For help scoping the work, see AI consultancy or book a free consultation. Agree the scope, responsibilities and written quote before work starts.