Inside language models · Module 1: Inside language models in practice
Sampling and repeatability
Learning outcome: Design a fair comparison when outputs vary.
Text generation can choose among candidate next tokens. Sampling controls affect that choice; some settings encourage more variation than others. Availability and meaning of individual controls depend on the model and service. A lower-variation setting is not a truth setting, and repeated runs can still differ.
For a useful comparison, keep the task, source material and scoring rule fixed. Save the model identifier and settings when available. Run more than one trial if variation matters. Assess the outputs against a criterion defined before looking at them. Otherwise, selecting the most flattering answer makes the experiment look better than the normal experience.
Worked example
A fictional team compares two summarisation prompts. Prompt A is tried once; prompt B is tried five times and only its best answer is retained. That is not a fair comparison. Give each prompt the same cases and number of trials, then record both missing facts and invented details, including the less impressive runs.
Try it yourself
Design a comparison using two prompts, three fictional documents and two runs per document. Write the scoring criteria before producing any outputs.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
There are twelve outputs: two prompts times three documents times two runs. Retain all twelve. Report which prompt meets the criteria more consistently, rather than claiming certainty from one attractive answer.
Check your understanding
Which comparison is fairer?
- Equal cases and trials with criteria set in advance
- One run of A against the best of ten runs of B
- Whichever answer looks most polished
Show the answer and explanation
Answer: Equal cases and trials with criteria set in advance. Equal inputs and trial counts prevent one prompt receiving extra chances to look good. Criteria set beforehand reduce selective judging. Retain failed runs as well as successful ones.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.