Analyse customer survey comments without inventing the numbers

Priya has asked for two different things. The first is a summary: what are customers saying most, with quotes that fairly represent it. The second is triage: is anything in here urgent? These need different kinds of attention. A theme that appears eight times is worth reporting. A single comment about a product that smells of burning is worth acting on today.

The board will take the numbers in your update at face value, so they must be numbers you can defend. "Eight of 48 comments mention delivery problems" is defensible if you can list the eight. "35% of customers are unhappy with delivery" isn't, however confident it sounds.

Good fit for the first sort. Reading 48 comments and proposing themes is the kind of pattern-finding AI does quickly, and it can do a first pass at tagging every comment.

It's unreliable at two things this task depends on: counting accurately, and quoting exactly. Language models often miscount items in a long list and may "tidy up" a quote into something nobody wrote. Plan to check both.

Survey comments look anonymous, but these aren't. C07, C18, C32 and C43 name Fernway staff. C25 identifies the respondent and her workplace. C24 ("the driver who does our Thursday drop") and C36 ("the lady who answers the Midlands phone line") name nobody but point at one person each. Removing names doesn't always make data anonymous (the ICO's anonymisation guidance explains why): anyone at Fernway could work out who C24 and C36 are about.

Before analysis, replace those references with a role in square brackets, for example [account manager], [engineer], [driver], [staff member], [respondent] and [customer's business]. The themes don't depend on who it was. Praise and complaints about named people go to their manager as a separate job, not into a board report.

Use a tool your organisation has approved for customer data, signed in with your work account, even after removing names. Your organisation's policy and approved tools still take priority.

Keep the comment IDs. They're what make every count and quote checkable. Keep the rating, region and customer type, since they help the AI read sarcasm (a rating of 2 next to "Oh brilliant") and let you ask later whether a theme is stronger in one region.

Remove the names as described above. The first line of the practice file is a label saying the data is fictional; delete it so the column headings are read as headings. Check the file is complete: 48 rows in, 48 rows referred to in the output.

The approach is the same everywhere: tag first, count second, and make the tool show which comments it put where. How the counting happens differs.

Break it into two short prompts. Asking for themes, tags, counts, quotes and triage at once in one long prompt usually makes every part worse.

The attached file has 48 customer survey comments, each with an ID. Suggest 8 to 12 themes that cover them. Then make a table with one row per comment: ID, themes (one or more), and whether the comment is positive, negative or mixed. Read the rating alongside the text, as some comments are sarcastic. Every ID from C01 to C48 must appear once.

Using your table, list each theme with the IDs of the comments in it and the count. Don't give percentages. For each theme, choose one quote that represents it and copy it word for word from the comment, with its ID. Then, separately, list every comment that mentions a possible safety risk, a promise we didn't keep, or a customer who says they may leave, even if it appears only once.

This is the kind of answer the weak instruction can produce.

Before you check anything, count the claims you'd need to verify: five percentages, one quote, an average and "most customers positive". None of them comes with a comment ID.

The error that matters most isn't a number:

• The safety comment has disappeared. C10 describes a shredder plug getting very hot with a burning smell. Folded into "product quality (10%)", it looks like a minor theme. It needs action today, whatever the theme counts say.
• The broken promise has gone too. C05 was promised a manager's call back in June and never got one. That's one comment, and it's exactly what the Customer Response Project exists to fix.
• Minority feedback has vanished. The driver complaint (C24) and servicing cancellations (C23, C48) appear nowhere.
• Sarcasm check. Look at how C02, C09, C21 and C34 were read. Each sounds positive on the surface and has a rating of 2.

Don't ask the AI to "check its numbers". Ask it to show them.

Replace every percentage with a count and list the comment IDs behind each count. Check each quote appears word for word in the file; replace any that don't with an exact quote and its ID. Add a separate section for any comment about safety, a promise we didn't keep or a customer threatening to leave. Recalculate the average rating and show the working.

Then check it yourself. Spot-check at least two themes by reading every comment in the ID list and asking "does this belong here?", then search the file for words like "late", "invoice" or "website" to find any that were missed. Find every quote in the file with your search tool. Recalculate the average: the ratings add up to 149, and 149 divided by 48 is 3.10.

Your theme names and counts may differ slightly from the finished version below, because deciding which theme a comment belongs to is a judgement. What matters is that your counts match your own ID lists, and that your quotes are exact.

Yes, for the first sort. Reading 48 comments and proposing a set of themes by hand takes a while, and the AI's tag table gave you something to correct instead of a blank page.

The checking took real time: re-reading every comment in two or three themes, searching for each quote, redoing the average. That's the part you can't skip, because the unchecked version had an invented quote, a wrong percentage and no mention of a safety issue. If you'd sent it, the board would have seen a neat summary and nobody would have phoned the customer with the hot plug.