Pairwise comparisons and order effects

Learning outcome: Compare outputs without letting presentation decide the winner.

Pairwise evaluation asks which of two outputs better meets a defined criterion. It can make a difficult rating task easier, but introduces its own biases. Reviewers may favour the first answer, a longer answer or a familiar style.

Hide irrelevant model labels where feasible and vary presentation order. Allow a tie or “both unacceptable” result. Use the same source material and task for both outputs. Record the reason for preference against the rubric, not just a vote. A win rate on one test set is not a universal ranking of models.

Worked example

A fictional comparison shows a detailed answer first and a concise answer second. The detailed answer repeats an unsupported date; the concise answer correctly says the date is unknown. If the rubric is source fidelity, the second answer should win despite appearing less complete. Reversing order helps reveal whether position is influencing reviewers.

Try it yourself

Design a blind comparison sheet with criterion, order, preferred answer, reason and a both-unacceptable option. Use two outputs with different failure types.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

Why allow “both unacceptable”?

1. It prevents any useful comparison
2. It guarantees reviewers will agree
3. A forced winner can conceal that neither output is usable

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• Anthropic: evaluating agents
• NIST: AI Risk Management Framework
• scikit-learn: metrics