Data leakage and tomorrow’s information

Learning outcome: Keep evaluation information out of model development.

Data leakage occurs when information unavailable at the intended prediction time influences development or evaluation. It can enter through a future field, duplicated examples across splits or preprocessing fitted using the held-out test data.

Define the moment at which the prediction will be made. List which fields genuinely exist then. Split data in a way that reflects future use, including grouping related records where necessary. Fit learned preprocessing on the training data and apply it to the held-out data. Keep a final test set separate from repeated tuning so it remains a meaningful check.

Worked example

A fictional model predicts which support tickets need escalation at arrival. One input is “final resolution team”, populated after the ticket closes. An excellent test score may simply reflect that future information. Removing the field and evaluating on later tickets gives a more relevant estimate, even if the number looks less impressive.

Try it yourself

List five possible ticket fields and label each as available at arrival or available later. Include one field that indirectly reveals the outcome.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

Which field is unsuitable for an arrival-time prediction?

1. The message received at arrival
2. The team that eventually resolves the ticket
3. The arrival timestamp

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• scikit-learn: common pitfalls
• scikit-learn: evaluation metrics