Missing data is a question, not a zero

Learning outcome: Choose a missing-value treatment that fits the meaning.

A blank cell may mean not measured, not applicable, withheld or genuinely zero. Replacing all blanks with zero changes the meaning of the dataset. Before calculating, define what each field represents, its unit and the allowed reasons for missingness.

Examine who or what is missing. If only easy cases have completed measurements, an average of the available rows can exaggerate performance. Report how many records contributed to a result and how many were excluded. Any imputation method should be documented and checked for its effect on the conclusion. Keep the original values so the treatment is reversible.

Worked example

A fictional pilot has processing times of 8, 10, blank and 12 minutes. The mean of observed times is 10 minutes across three records. Replacing the blank with zero gives 7.5 minutes across four rows, falsely implying a completed task took no time. The missing record needs investigation, not a convenient numerical value.

Try it yourself

Create a small table with a true zero, a missing time and a not-applicable field. Write the calculation rule and a sentence reporting its denominator.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

A blank processing-time cell should automatically become which value?

1. No automatic value; establish what the blank means
2. Zero
3. The fastest observed time

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• scikit-learn: common pitfalls
• scikit-learn: evaluation metrics