Evaluation, red teaming and uncertainty · Module 1: Evaluation, red teaming and uncertainty in practice
Rubrics and disagreement between reviewers
Learning outcome: Make a quality judgement inspectable.
A rubric turns a broad judgement into criteria with observable anchors. “Good answer” is too vague for consistent evaluation. Criteria might cover factual support, task completion and preservation of uncertainty. Define critical failures separately from stylistic preferences.
Have reviewers score a shared sample and discuss disagreements before evaluating the full set. Preserve disagreements that reflect genuine ambiguity in the task. An AI judge can help scale a defined rubric, but should be checked against human-reviewed examples and known traps. Do not assume fluent explanations from a judge prove its scores are reliable.
Worked example
Two fictional reviewers disagree about an answer that is concise but invents an approval. One rewards clarity; the other fails it. The rubric resolves this by making unsupported approval a critical failure regardless of style. Concision remains a separate criterion, so a useful attribute is recorded without hiding the decisive error.
Try it yourself
Write a three-criterion rubric with clear pass/fail anchors and one critical failure. Score two fictional answers that trade elegance against factual support.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
The unsupported approval must fail the critical criterion. Record whether the concise writing itself met its criterion. This makes the result useful for repair without allowing an average score to disguise an unacceptable action.
Check your understanding
How should an invented approval affect the evaluation?
- It matters only if the answer is also long
- As the defined critical failure, even if the prose is strong
- It should be cancelled out by a high style score
Show the answer and explanation
Answer: As the defined critical failure, even if the prose is strong. The rubric defines unsupported approval as a critical failure. Style can be recorded separately, but averaging it with factual support would conceal the decisive problem.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.