Active learning and selective labelling

Learning outcome: Choose which examples need human labels most carefully.

Active learning selects examples for labelling to make better use of limited annotation effort. One approach asks for labels on uncertain cases, but uncertainty alone may repeatedly select near-duplicates or unusual examples that are not useful for the target task.

Combine a selection strategy with checks for coverage and label quality. Keep an independent evaluation set so the labelling process does not consume the test. Write clear annotation guidance and allow ambiguous cases to be discussed. Track which examples were selected and why. The number of labels collected is less informative than whether they improve performance on the intended distribution.

Worked example

A fictional classifier asks reviewers to label twenty almost identical messages containing an unfamiliar product code. Meanwhile, an entire category of short requests remains unrepresented. A revised selection process limits duplicates and includes coverage checks, so reviewer time is spent on a more varied set.

Try it yourself

You can label ten of thirty fictional messages. Propose a selection rule using uncertainty, diversity and coverage. Reserve an evaluation sample outside that process.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

Why might uncertainty-only sampling waste annotation effort?

1. It can repeatedly select very similar cases
2. Uncertain cases never teach anything
3. Human labels are unnecessary for active learning

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• scikit-learn: common pitfalls
• scikit-learn: model evaluation
• NIST: AI Risk Management Framework
• scikit-learn: time series splits
• scikit-learn: novelty and outlier detection
• modAL: uncertainty sampling
• Neo4j: entity resolution