Evaluate and improve AI responses

This guide includes features awaiting release. If an option is not available in your workspace, contact StackShift Support.

What you can do

Compare AI behavior with examples and use outcomes to choose improvements. Where to find it: AI → Evaluations, Improvements and Procedure outcomes

Create or open an evaluation and add realistic examples, including questions AI should hand to a person.

Select the relevant configuration or procedure version and run the evaluation.

Read individual results and their evidence. A combined score can hide one serious mistake.

Use Improvements and Procedure outcomes to investigate repeated failure patterns; revise the knowledge, policy or procedure that caused them.

Run the same examples again before broadening customer-reply scope. Review real outcomes afterward.

Include both a routine delivery question and a request for an unauthorized refund; expect an answer to one and escalation for the other.

Evaluation results do not guarantee every future answer.
An improvement suggestion needs review before changing a published procedure or customer policy.

You have a repeatable way to assess a change instead of assuming a new prompt is better.

Run cannot start: check provider availability, credits and inputs.
Good score but poor replies: include examples from the actual failing conversation and verify the evaluated version.

Enable customer-facing replies for a chosen scope, with rules for handing work to a person.

Create, test and publish support procedures with approved actions and human handover.