How to Evaluate a Machine Learning Model

To evaluate a model, test it on examples that look like the situations where you will use it, and compare it with a simple baseline. A score only means something when the data split, the metric and the comparison all match the task. Start by writing down the decision the model supports.

What decision does the model support?

Suppose a fictional service predicts which customers may leave next month. State when the prediction is made and what a team would do with it, for example send a check-in message. Exclude cancellation confirmations and other future information from the features. If a field is only filled in after a customer leaves, it must not be an input.

Choose a baseline, such as predicting no churn for everyone. It may be accurate when churn is rare, but it will find none of the customers you care about. That is why accuracy alone is a weak guide here.

How should you split the data?

Keep development data and final evaluation data separate. Three common patterns:

• Random split, fine when rows are independent.
• Chronological split, better for future predictions because the model is tested on later data than it trained on.
• Grouped split, needed when related records from one person or organisation could appear on both sides.

Fit preprocessing, such as scaling or filling missing values, on training data only. Then apply it to the test data. Inspect leakage from duplicates, target-derived fields and information recorded after the outcome. scikit-learn's pipelines help keep these steps inside the training fold.

How do you read a confusion matrix?

Imagine 100 fictional customers: 20 actually churn. The model flags 25, and 15 of those churn. That gives 15 true positives, 10 false positives, 5 false negatives and 70 true negatives.

Precision is 15/25, or 60%. Recall is 15/20, or 75%. Accuracy is 85/100, or 85%. Each describes a different aspect. Precision answers "of those we flagged, how many were right?" Recall answers "of those who left, how many did we catch?" Discuss outreach cost and missed churn before choosing a threshold.

Notice that the "predict no churn" baseline scores 80/100 accuracy on this data. The model's 85% looks modest next to that, while its recall of 75% is where the real gain sits.

What else should you inspect?

Review errors by meaningful groups and periods. A model can look fine overall and fail for one city, plan or month. For regression, compare absolute error with a naive prediction and look at unusually large misses.

Small evaluation sets produce uncertain estimates. Do not overstate a tiny improvement. Cross-validation, described in the scikit-learn documentation, gives a spread of scores rather than one number, which is more honest.

Use validation data for model choices and reserve a final test for the finished approach. Repeatedly choosing the best test score turns the test into another development set.

Step-by-step routine

1. Write the decision and the moment of prediction. 2. Pick a baseline and score it. 3. Split the data in a way that matches deployment. 4. Fit preprocessing and the model on training data only. 5. Choose the metric from the error costs, then pick a threshold on validation data. 6. Score once on the held-out test set. 7. Read the worst errors by hand.

A short scenario

A learner trains a churn model, sees 92% accuracy and stops. A reviewer asks for the baseline. Predicting "no churn" for everyone also scores about 90% on that data, because few customers leave. The learner now reports recall and precision, finds recall is low, and adjusts the threshold on validation data. The lesson is that a number needs a comparison before it means anything.

Common mistakes

• Tuning on the test set until the number looks good.
• Scaling or imputing on the full dataset before splitting.
• Quoting accuracy on imbalanced data.
• Reporting one score without any sense of its uncertainty.

Reporting checklist and self-check

Record data provenance, split rules, baseline, metric definition, threshold and limitations. Include example failures and the conditions under which you would reconsider use.

To check you got it, hand your report to someone else and ask them to reproduce one number from your code. If they cannot, something is missing from the record.

What to build next

Write a one-page evaluation report for the fictional churn example. Recommend whether the model should be used, changed or studied further, and say what information is still missing. Then read regression, classification and clustering explained to match metrics to tasks, and see the Machine Learning track for the wider path.

Sources & further reading

• scikit-learn model selection and evaluation
• scikit-learn metrics and scoring
• scikit-learn cross-validation

Examples and practice recommendations are educational illustrations. Tool documentation and programme inclusions should be checked for your intended use.