A model flags an account as likely to cancel because its activity has fallen. That can help a team decide where to investigate, but it leaves the central explanation open. The customer may be disappointed, between projects or moving work to another account. Prediction concerns what is likely to happen under the conditions represented in the data, not necessarily why it will happen.

Predictive analysis uses data and statistical or machine-learning models to estimate unknown outcomes. Those outcomes are often future events, such as cancellation or demand, although prediction can also concern an unobserved current value.

Define the prediction at a specific moment

In a fictional retention model, specify which accounts are scored, the date of prediction and the future window in which cancellation counts. Use only information available at that moment. Including a cancellation-request field created afterwards would make retrospective performance look impressive while providing no useful advance warning.

The scikit-learn guidance on data leakage explains why information from evaluation data must not leak into training and preprocessing. Split data in a way that reflects deployment, including time and repeated records from the same people or accounts where relevant.

A random row split can be misleading when near-duplicate records from the same account appear on both sides. The evaluation should test the kind of new case the model will actually face.

Evaluate performance against a practical baseline

Compare the model with a simple alternative, and choose performance measures that fit the decision. If cancellation is rare, predicting that nobody cancels can achieve high overall accuracy while failing to identify any customer at risk.

Ranking and calibration answer different questions. A model may rank higher-risk accounts well while overstating their probabilities. If a score is presented as a 70% risk, examine whether comparable scored cases experience the outcome at roughly that rate, with suitable attention to sample size and conditions.

Check performance across relevant groups and periods. Product changes, new audiences and altered data collection can make an earlier evaluation a poor guide to current behaviour.

Separate the score from the intervention

A strong predictor is not automatically a useful lever. Increasing logins through reminders may change a recorded symptom without addressing the reason someone was leaving. Driver analysis and qualitative research can help frame hypotheses, but causal claims need their own evidence.

If scores trigger outreach, the intervention changes the future data and should be recorded. Comparing actual outcomes with the model’s predictions alone does not necessarily establish the outreach effect.

Use predictions to inform a defined process, with appropriate oversight and monitoring for its consequences. A useful model should improve the decision it supports, rather than merely produce an accurate-looking score on historical records.