Model quality is a property of the evaluation population, prediction target and loss or decision rule. Training accuracy, ranking discrimination, calibrated probabilities and time-to-event behavior each answer a different question.
In this chapter
- Validation, leakage, metrics, and tree modelsUnderstand overfitting, safe dataset splits and ensemble behavior.
- Precision, recall, ROC-AUC and decision thresholdsChoose classification measures according to prevalence and business cost.
- Gradient boosting, LSTM and survival predictionCompare tabular and sequential risk modeling with correct censoring and evaluation.
- Regression and time-series forecasting evaluationMAE, RMSE, WAPE, bias, rolling-origin splits and economic impact.
- Model selection, calibration and error analysisBaselines, group/time splits, probability quality and error cohorts.
- Neural-network optimization and architecture choicesBackpropagation, SGD/Adam, CNNs, LSTMs and training diagnostics.
- Linear models, PCA and clustering baselinesUnderstand prediction objectives, coefficient penalties, variance projections and distance-based groups.