The decision AI should not make

Written by AI Tools AcademyChecked against the sources listed below on 27 September 2026

This is a fictional example. Fernway Group is a made-up UK company, and everyone in it, including the applicants, is invented.

The other pieces in this series are about AI doing a job badly. This one is about a job AI shouldn't be given at all: deciding which people get an opportunity. The same reasoning applies to ranking staff for a bonus, choosing who is at risk of redundancy or grading performance. The tool may well produce something that looks reasonable. What nobody can do is properly explain or stand behind it.

Before

Tom Elliott, Fernway's Sales Lead, advertised for a Sales Coordinator. Sixty-four people applied. Tom had a renewal deadline, a team to run and a week to produce a shortlist of eight for interview.

Fernway uses Microsoft 365 Copilot on work accounts, which is its approved tool, so Tom didn't paste the applications into a personal chatbot. That matters, and it rules out one problem. It doesn't touch the bigger one.

The AI approach

Tom put the 64 CVs and cover letters in a folder and asked:

The instruction Tom usedAny AI tool

Review these applications for our Sales Coordinator role (job description attached). Score each candidate out of 100, rank them, and give me the top eight to interview with a one-line reason for each. Also give a short reason for the ones you'd reject.

Why this works: This is the version that caused the problem: it hands the whole judgement to the tool, with no criteria, and asks for a ranking of people.

What went wrong

The output was fast and tidy. Every applicant had a score and a reason. Some of the reasons are shown below, with the applicants' names removed.

What the AI produced (extract) (fictional)

Shortlisted

Applicant 47, 91/100: Strong communicator with a team-sports background (university rugby captain). Likely to be an excellent cultural fit for a sales team.
Applicant 14, 88/100: Recent graduate with energy and a digital-native approach to CRM tools.

Not shortlisted

Applicant 8, 61/100: Extensive experience, but may be overqualified and less adaptable to new systems.
Applicant 31, 58/100: Two-year gap in employment (2022 to 2024) may indicate reduced commitment.
Applicant 22, 55/100: Qualifications from an overseas institution; UK sales experience limited.

Read those reasons again with an employer's eyes.

"Overqualified and less adaptable to new systems" is the kind of thing said about older applicants, and age is protected under the Equality Act 2010.
A career gap can reflect maternity leave, caring for children or relatives, illness or disability. Treating it as low commitment risks disadvantaging people because of sex, pregnancy and maternity, or disability.
"Overseas institution" can stand in for nationality, which falls under race in the Act.
"Cultural fit" from team sports and "digital-native" graduates tilt the list towards a narrow group without anyone intending it.

The tool wasn't told to do any of this. It learned from a great deal of human writing about hiring, and some of that writing carries exactly these assumptions. Removing names doesn't remove them, because the signals sit in dates, gaps, institutions and hobbies.

Then two things happened that Tom couldn't answer.

Applicant 31 asked for feedback on why she wasn't invited to interview. The honest answer was "a tool gave you 58 and mentioned your career gap". Tom couldn't give that answer, and he couldn't give any other, because he hadn't made the judgement himself.

And when Tom ran the same request again the next morning to double-check, the order of the top eight changed. Two applicants dropped out and two came in. The scores looked like measurements, but they weren't stable enough to be.

A better approach

Do it yourself instead: A person decides, against criteria written first

Shortlisting, hiring, bonuses, performance ratings and redundancy selection are decisions with significant effects on people. They belong to a person who can explain them, and ideally to two people, one checking the other.

A fair shortlist is not complicated:

Write the criteria before you read any applications. Four to six things the role genuinely needs, each one something you can see evidence of in an application.
Score each application against those criteria, with a few words noting the evidence. "Criterion 2: ran a customer database for three years at previous job" is an explanation. "58/100" is not.
Have a second person check a sample, especially the borderline ones and the rejections.
Keep the notes. If anyone asks why, you can show them.

There is still useful work for AI around the decision, as long as no applicant's data goes near it:

Check your job advert and criteria for requirements the role doesn't really need, or wording that might put some groups off.
Turn your criteria into interview questions, each tied to a criterion.
Draft a neutral rejection template and a feedback template you then fill in yourself.

Two points on the law, in plain words. This isn't legal advice, and your HR team, data protection officer or legal adviser has the final say. First, the Equality Act 2010 makes it unlawful to discriminate because of protected characteristics including age, disability, race, religion or belief, sex, and pregnancy and maternity. Using software to sort applications doesn't move that responsibility away from the employer. Second, UK data protection law gives people specific rights where significant decisions about them are made solely by automated means, and the ICO expects organisations to be able to explain decisions made with AI's help. Its guidance Explaining decisions made with AI, written with The Alan Turing Institute, sets out what a meaningful explanation looks like. Tom's shortlist would have struggled to meet it.

Your organisation's policy and approved tools still take priority. An approved tool is the right place for work data, but approval covers where the data goes. It doesn't make the tool the right one to decide about people.