Explain To Decide: A Human-Centric Review on the Role of Explainable Artificial Intelligence in AI-assisted Decision Making

Abstract

The unprecedented performance of machine learning models in recent years, particularly Deep Learning and transformer models, has resulted in their application in various domains such as finance, healthcare, and education. However, the models are error-prone and cannot be used autonomously, especially in decision-making scenarios where, technically or ethically, the cost of error is high. Moreover, because of the black-box nature of these models, it is frequently difficult for the end user to comprehend the models' outcomes and underlying processes to trust and use the model outcome to make a decision. Explainable Artificial Intelligence (XAI) aids end-user understanding of the model by utilizing approaches, including visualization techniques, to explain and interpret the inner workings of the model and how it arrives at a result. Although numerous research studies have been conducted recently focusing on the performance of models and the XAI approaches, less work has been done on the impact of explanations on human-AI team performance. This paper surveyed the recent empirical studies on XAI's impact on human-AI decision-making, identified the challenges, and proposed future research directions.