Synthetic data and circular evidence

Learning outcome: Use generated examples without treating them as ground truth.

Synthetic data is constructed rather than directly observed from the target setting. It can help test formats, rare scenarios or workflows without using real personal records. Its usefulness depends on how well it represents the properties that matter for the task.

A model-generated label is not automatically a reliable reference answer. If the same system creates and grades the examples, shared mistakes may be hidden. Keep provenance, review the labels and evaluate on independently established cases where possible. Do not claim a system works for a population because it performed well on fictional people designed by another model.

Worked example

A fictional assistant generates fifty “difficult customer” messages and labels every short message rude. A classifier trained on that set reproduces the shortcut. A short, polite request from an actual test specification exposes the problem. More generated rows with the same shortcut would reinforce it rather than diversify the evidence.

Try it yourself

Design six fictional messages that deliberately separate length from politeness. Label them yourself with a written rule and have another reviewer challenge ambiguous cases.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

Why is self-generated, self-graded data a weak sole evaluation?

1. The generator and grader may share the same mistakes
2. Synthetic examples can never be useful
3. Fictional examples always reveal personal information

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• scikit-learn: common pitfalls
• scikit-learn: evaluation metrics