arXiv Science⌕ Search

arXiv subjects

Richard M. Golden

Publications and source records attributed to Richard M. Golden.

6 recordsLinked to original sources

Detection of Cognitive Diagnostic Model Misspecification using New Lancaster-Chesher Information Matrix Tests

Model specification tests play a crucial role in evaluating the appropriateness of probability models for estimation and inference. Existing methods for the detection of model misspecification such as the chi-square goodness-of-fit (GOF) tests and more recently the M2 statistic \autocite{MaydeuOlivares2005,MaydeuJoe2014} tend to result in test statistics with excessive degrees of freedom for models with larger numbers of parameters. An alternative approach is based upon the Information Matrix (IM) equality. The IM equality asserts that if a probability model is correctly specified, the asymptotic covariance matrix of the maximum likelihood estimators can be asymptotically estimated using a methodology based upon either the first or second derivatives of the log-likelihood function. Using a contrapositive argument, White (1982) \nocite{Wh82} proposed a misspecification test methodology based upon comparing these two alternative covariance matrix estimators. Extending this work, Presnell and Boos (2004) \nocite{Presnell2004} showed how to develop a misspecification test which only requires one degree of freedom regardless of the complexity of the model or data. In this paper, we extend prior work and additionally apply methods of Golden et al. (2013, 2016) \nocite{golden2013}\nocite{Golden2016} to derive and evaluate misspecification tests for Cognitive Diagnostic Models (CDMs) which only require 1 or 2 degrees of freedom regardless of model or data complexity. Analytic formulas for the tests are derived so they can be applied without requiring computationally intensive bootstrap simulation methods. Our simulation studies show the asymptotic statistical tests have good level (type 1 error) and power performance for CDM models and data which might be encountered in practice.

stat.ME↗

Human-Likeness is Multi-Dimensional: Evaluating LLM Generated Virtual Patients Against Simulated Patient Dialogue

Large language models (LLMs) are increasingly used to generate virtual patients (VPs) for medical training. As these systems move toward broader deployment in medical education, reliable evaluation is needed to identify potential failure modes and ensure that generated patient responses remain consistent with human simulated patient (SP) behavior. Inaccurate, unrealistic, or inconsistent responses may introduce educational risks by exposing learners to misleading patient interactions. However, current evaluation approaches rely heavily on human judgment, making large-scale evaluation difficult. In this work, we present a controlled generation pipeline for systematically comparing VP responses generated by 10 LLMs across multiple decoding temperatures with human respiratory SP responses. To support scalable evaluation, we introduce a multi-dimensional framework for measuring similarity between LLM-generated and human SP dialogue across semantic, conversational grounding, linguistic, and emotional dimensions. Across semantic, conversational, linguistic, and emotional dimensions, no single model consistently produced the most human-like responses. From a practical perspective, Gemini 2.5 Flash demonstrated the most consistently human-like performance across semantic, linguistic, grammatical, and emotional dimensions, while Claude Sonnet 4.6 achieved the strongest semantic fidelity. These findings highlight the importance of multi-dimensional evaluation and demonstrate how automated metrics can provide a scalable approach for assessing virtual patient systems before broader deployment in medical education.

cs.HC↗

Can You Spot the Virtual Patient (VP)? Expert Evaluation, Turing Test, Linguistic Analysis, and Semantic Similarity Analysis

Communication is a core clinical competency and essential for safe, effective patient care. Virtual patients (VPs) powered by large language models (LLMs) offer a promising alternative to traditional formats such as standardized patients (SPs). However, few studies have quantitatively as well as qualitatively evaluated their realism across diverse clinical scenarios using behavioral studies and linguistic dialog analyses. This study evaluated the realism of GPT-4o-generated VPs through expert review, Turing testing, linguistic analysis, and semantic analysis. We generated 44 VPs using structured illness scripts derived from doctor-simulated patient dialogues across 17 clinical categories. Experts annotated responses for hallucinations, omissions, and repetitions, with high interrater reliability (ICC > 0.77). A Turing test revealed that participants struggled to distinguish VP responses from those of real patients, with average classification accuracy for VP responses falling below chance level. Lexical and syntactic analyses of over 2,000 conversation turns showed that VP responses were generally realistic, more formal and lexically consistent, while human responses displayed greater emotional variation. Semantic similarity using BioClinicalBERT and cosine similarity averaged 0.871 at the response level and 0.842 at the transcript level. Our findings support the integration of LLM-based VPs in communication training while highlighting areas for trust, transparency, and refinement, contributing to both GenAI deployment in healthcare and its safety evaluation.

cs.HC↗

Assessment of Misspecification in CDMs Using a Generalized Information Matrix Test

If the probability model is correctly specified, then we can estimate the covariance matrix of the asymptotic maximum likelihood estimate distribution using either the first or second derivatives of the likelihood function. Therefore, if the determinants of these two different covariance matrix estimation formulas differ this indicates model misspecification. This misspecification detection strategy is the basis of the Determinant Information Matrix Test ($GIMT_{Det}$). To investigate the performance of the $GIMT_{Det}$, a Deterministic Input Noisy And gate (DINA) Cognitive Diagnostic Model (CDM) was fit to the Fraction-Subtraction dataset. Next, various misspecified versions of the original DINA CDM were fit to bootstrap data sets generated by sampling from the original fitted DINA CDM. The $GIMT_{Det}$ showed good discrimination performance for larger levels of misspecification. In addition, the $GIMT_{Det}$ did not detect model misspecification when model misspecification was not present and additionally did not detect model misspecification when the level of misspecification was very low. However, the $GIMT_{Det}$ discrimation performance was highly variable across different misspecification strategies when the misspecification level was moderately sized. The proposed new misspecification detection methodology is promising but additional empirical studies are required to further characterize its strengths and limitations.

stat.ME↗

Stochastic Descent Analysis of Representation Learning Algorithms

Although stochastic approximation learning methods have been widely used in the machine learning literature for over 50 years, formal theoretical analyses of specific machine learning algorithms are less common because stochastic approximation theorems typically possess assumptions which are difficult to communicate and verify. This paper presents a new stochastic approximation theorem for state-dependent noise with easily verifiable assumptions applicable to the analysis and design of important deep learning algorithms including: adaptive learning, contrastive divergence learning, stochastic descent expectation maximization, and active learning.

stat.ML↗

Generative Modeling of Hidden Functional Brain Networks

Functional connectivity refers to the temporal statistical relationship between spatially distinct brain regions and is usually inferred from the time series coherence/correlation in brain activity between regions of interest. In human functional brain networks, the network structure is often inferred from functional magnetic resonance imaging (fMRI) blood oxygen level dependent (BOLD) signal. Since the BOLD signal is a proxy for neuronal activity, it is of interest to learn the latent functional network structure. Additionally, despite a core set of observations about functional networks such as small-worldness, modularity, exponentially truncated degree distributions, and presence of various types of hubs, very little is known about the computational principles which can give rise to these observations. This paper introduces a Hidden Markov Random Field framework for the purpose of representing, estimating, and evaluating latent neuronal functional relationships between different brain regions using fMRI data.

stat.ML↗