Before-and-after is not automatically causation

Learning outcome: Identify alternative explanations for a pilot result.

A change after an AI rollout does not by itself establish that AI caused it. Task difficulty, staffing, seasonal demand, training and measurement changes can all affect the result. Start by stating exactly what was measured and which comparison was made.

Use comparable cases and record changes in the surrounding process. A controlled comparison can strengthen a causal argument when feasible and appropriate, but it also needs a fair design. With a small observational pilot, describe the result as an association and state the uncertainty. Include review and correction time rather than measuring only the first draft.

Worked example

A fictional team's average drafting time falls from 20 to 14 minutes during a pilot. The second week contains shorter letters and an extra reviewer. The observed six-minute difference is real for those records, but the share caused by AI is unresolved. A next test matches letter type and counts total effort, including corrections.

Try it yourself

Write a cautious two-sentence conclusion for the pilot. Then propose one design change that would make the next comparison more informative.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

What can this before-and-after result establish on its own?

1. The same saving will occur in every team
2. A measured difference with unresolved alternative explanations
3. AI caused the entire reduction

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• scikit-learn: common pitfalls
• scikit-learn: evaluation metrics