Aide's replies to order questions are now graded against a written definition of an acceptable reply, per type of request. Every reply a person confirms as unacceptable becomes a test case, and changes to how Aide replies are checked against those cases before they ship. No claim about answer accuracy is made until the bar has been met on enough real tickets to mean something.
How it behaves
- What "acceptable" means: a reply a teammate at your store would send as is, or with cosmetic edits only, that gets every fact right for this customer (order status, dates, tracking, amounts, what was changed), follows your store's policy (windows, fees, what may be promised), offers, takes or correctly refuses the right action, answers the customer's actual question in their language, and promises no refund, credit or timeline without a source. Anything else is unacceptable; there is no partial credit.
- A wrong fact fails the reply: a reply that states something wrong is unacceptable however good its tone and next step are, so a polished reply can't pass on style.
- Refusing correctly is right: an address change asked for after the order is fulfilled should be refused and explained, not attempted, and is graded that way.
- Handoffs: a handoff is acceptable only when it was warranted: the information or the permission to answer was missing, your policy requires a person, or the customer asked for one. An unnecessary handoff is unacceptable, so the rate can't be raised by passing conversations to your team. The share resolved without a person is reported next to it.
- Measured per request type: each graded ticket gets one label: where is my order, address change, cancel, refund, return, shipping policy or other. Aide proposes the label and the person grading confirms or corrects it. The label is used for grading only; nothing is added to your tickets.
- The bar: first, 90% of replies acceptable on where-is-my-order, address change and cancel requests, on at least 100 graded tickets each. After that, 99% on at least 200 each (no more than 2 unacceptable out of 200). Refund, return and shipping policy replies are measured the same way but not held to the bar.
- How sure a rate is: every rate is reported with the number of tickets and the lowest rate the data is reasonably consistent with, so a small sample isn't read as more certain than it is: 90 acceptable out of 100 could mean a true rate as low as about 82.6%.
- Who grades: an AI judge, run separately from the model that writes replies, grades each sampled reply first. A person then checks every reply the judge failed and a random quarter of those it passed, and the person's verdict is final. If the judge wrongly passes more than 2% of the replies checked, every pass is checked by hand until it is back under.
- Which replies: chat and email replies Aide sent for stores in its first cohort, sampled each week per request type. Copilot drafts are graded the same way and reported separately, along with whether your teammate sent them unchanged, edited them or discarded them. Test replays on a development store with seeded orders give early signal but never count toward the bar.
- Your customers' data: test cases are scrubbed of names, email addresses, phone numbers, street addresses, order numbers and tracking numbers before they are saved, and a case containing an email address, phone number or card-like number is rejected. The quality report lists tickets by their ID only, never their messages.
- No new screen or setting: the quality report is internal, run by the Aide team, and none of this changes how Aide writes replies today.
- Not yet: there are no figures. Grading against the bar starts with replies from the first cohort, and the set of test cases grows as failures are confirmed.