Test against the tally

You have the exact questions people ask, in their own phrasings, from Module 01. That is the test set. Put every phrasing in and check every answer against the source document. Then put in questions it should not answer and check that it refuses and redirects. Then put in questions where the answer is not in the documents and check it says so.

Record it in a simple table: question, expected answer, what it said, pass or fail. Fix failures at the source: usually a document that needs a clearer title or an answer nearer the top, sometimes an instruction that needs tightening. Retest. Aim for every version-one question passing in every phrasing before anyone else sees it.

Then give it to two colleagues, not the whole team, for a week. Ask them to send you anything wrong. Fix, retest, and only then open it up.

In practice. A test run of 22 questions in 61 phrasings: 49 correct, 8 wrong, 4 refused when they should have answered. All 8 wrong answers traced to two documents with vague titles. Retitled, retested: 60 of 61. The last one was a question that genuinely needed a human, and the test set was corrected.

At your desk. Build the test table from your tally. Run every phrasing. Record pass or fail. Fix at the source. Retest. Give it to two people for a week.

Write it down.

• Phrasings tested
• passed:
• The cause of most failures:
• My two test colleagues: