Simulating Classification Models for Ex-Ante Evaluation of Predict-Then-Optimize Methods

Abstract

Predict-Then-Optimize combines machine learning predictions with downstream optimization to support decision-making when problem parameters are unknown at the time of solving. However, better predictive performance does not necessarily lead to better decisions, making it useful to assess this relationship before investing in the development of a prediction model. Existing simulation-based approaches enable such ex-ante evaluation, but are limited to binary classification and may require solving the downstream optimization problem many times. We generalize this methodology to optimization problems with categorical uncertain parameters by introducing a method for simulating multiclass predictions at prescribed performance levels and using it to construct a prediction-error-to-decision-regret mapping. To reduce the computational effort required to obtain this mapping, we also propose a first-order approximation based on the regret caused by individual misclassifications. Computational experiments confirm that the proposed prediction simulation algorithm reproduces the target classification performance and that the first-order approximation closely matches the simulation-based error-to-regret mapping for some problems. Its accuracy decreases when interactions between simultaneous misclassifications become more important. These results demonstrate the potential of the proposed approach and identify new questions about when simple approximations of the error-to-regret relationship are sufficiently accurate.