How to scope and test a customer-service chatbot

A customer-service chatbot needs a defined job, approved information and a clear route to a person. Begin with those requirements before choosing a product. A system that produces fluent answers can still give the wrong policy, expose a record or leave a customer stuck.

This guide covers scoping, testing and operating a bounded chatbot. Use the acceptance criteria to compare proposals and decide whether the first release is ready for customers.

Choose one useful job

Start with a small set of questions such as delivery options, store information or how to request a booking. Name the information source and its owner. Keep complaints, unusual requests and decisions requiring specialist judgement on a clear human route.

Decide whether the chatbot provides information, prepares a request or can change a record. A booking enquiry is different from a confirmed booking. A draft refund request is different from authorising a payment. Give each action its own permission and test.

Build a controlled source set

Use current policy and product information with a named owner and review date. Remove obsolete versions and resolve conflicting instructions. The assistant should identify when a source cannot answer the question, rather than inventing a policy.

For order or account information, authenticate the user and apply the appropriate access controls throughout the workflow. A customer knowing a name or order number is not, by itself, proof that they may view every related record.

Agree the answer contract

• Answer only within the approved topic and data scope.
• Keep conditions, exclusions and uncertainty visible.
• Use a source or system record that the business can verify.
• Avoid promises about availability, delivery or refunds that the source does not support.
• Offer a usable escalation route with the context the customer has already supplied.
• State clearly when a request has been recorded and when an action has actually completed.

Compare platforms using requirements

Ask whether the proposed system can support your channels, identity checks, source updates, permissions and handover. Confirm how messages and logs are retained, which suppliers process them and how the business can export or delete material as appropriate.

Compare the written scope, current feature availability and total cost. Avoid old price lists or blanket claims that a platform is automatically compliant because of its hosting region. The right choice depends on the processing and controls actually used.

Test awkward conversations before release

Create an approved set of normal questions, missing details, contradictory sources and requests outside scope. Include a user asking the chatbot to ignore its instructions or disclose another customer’s information.

Check follow-up questions and conversation history, not just the first answer. Test unavailable integrations, failed actions and duplicate requests. Confirm that retrying an interrupted operation does not create a second booking or send another message.

A fictional returns-policy exercise

Imagine a retailer whose approved policy explains ordinary returns but says nothing about a customer’s damaged electrical item. The assistant should identify the limit and route the case to the responsible team. It should not invent repair instructions or promise a refund.

Ask the reviewer to compare the response with the source and confirm that the escalation contains the customer’s relevant question. This is a test scenario, not advice about a real product or a Blue Canvas client result.

Make handover work for people

Tell customers how to reach a person and what to expect next. If the team is unavailable, do not describe a queued message as a live transfer. Preserve the useful context so customers do not have to start again.

Give staff a view of unanswered questions, repeated failures and conversations needing review. Assign an owner who can update information, pause the system and organise a correction when an answer is wrong.

Check data handling deliberately

List the information the chatbot needs and exclude material that does not serve the agreed purpose. Review supplier terms, access, retention and the appropriate privacy information. Where the workflow raises regulated or sensitive questions, involve the responsible specialist before launch.

The ICO’s AI guidance is a useful starting point for assessing personal-data use. An AI notice or a vendor badge does not settle all the organisation’s responsibilities.

Measure service quality and total effort

Track correct answers, unsupported answers, completed handovers, unresolved requests and the time staff spend reviewing and repairing the work. Report important access failures separately from satisfaction scores.

Include configuration, software usage, source maintenance and support when assessing cost. A fictional calculation might compare the time needed to resolve the same sample manually and with the assistant, including every correction. Do not present that estimate as realised savings until it has been observed.

Launch within the tested scope

Begin on one channel with agreed limits and an available fallback. Keep the source register and evaluation set current. Recheck relevant behaviours when the model, integrations or policies change.

For broader operations, read our customer-service automation guide. For configuration, integration and handover, see AI implementation services.

Book a free consultation to discuss the task and what would need checking before a build.