arXiv subjects

James Liley

Publications and source records attributed to James Liley.

9 recordsLinked to original sources

arXiv2025

Calibration of answer probabilities in verbal autopsies: Working Paper

We consider the problem of calibrating a posterior estimator of a categorical latent variable given a fixed-length ternary string using only unlabelled observations and without a specified likelihood. We primarily consider a setting in which the estimator is parametrised by estimated conditional probabilities of elements of the string given the latent variable, with calibration if the estimates are correct. Our motivating application is the `Verbal Autopsy' procedure, whereby a cause of death is probabilistically inferred following a structured interview with associates of the deceased. More generally, our setting applies to circumstances where experts can more readily describe posterior beliefs than likelihoods, due to similarity with diagnostic practices. We argue combinatorially that in general the problem is intractable without a simplifying assumption on the data distribution, though some posterior estimators can be ruled out as incompatible. We propose an assumption of block-conditional independence on substrings, allowing calibration procedures based on substring frequency, imputation, and pairwise and three-way distributions of string elements. We give theoretical results on identifiability, on consistency for distributions of either fixed support or fixed entropy, and on robustness to assumptions, finding essentially that three conditional independence blocks of size at least the number of latent categories are necessary and sufficient for calibration. We empirically evaluate methods on data simulated to resemble realistic verbal autopsy questionnaires, and find substantial promise for the approach in the practical problem of calibrating posterior estimates for causes of death.

stat.ME↗

arXiv2024

AMUSE: Adaptive Model Updating using a Simulated Environment

Prediction models frequently face the challenge of concept drift, in which the underlying data distribution changes over time, weakening performance. Examples can include models which predict loan default, or those used in healthcare contexts. Typical management strategies involve regular model updates or updates triggered by concept drift detection. However, these simple policies do not necessarily balance the cost of model updating with improved classifier performance. We present AMUSE (Adaptive Model Updating using a Simulated Environment), a novel method leveraging reinforcement learning trained within a simulated data generating environment, to determine update timings for classifiers. The optimal updating policy depends on the current data generating process and ongoing drift process. Our key idea is that we can train an arbitrarily complex model updating policy by creating a training environment in which possible episodes of drift are simulated by a parametric model, which represents expectations of possible drift patterns. As a result, AMUSE proactively recommends updates based on estimated performance improvements, learning a policy that balances maintaining model performance with minimizing update costs. Empirical results confirm the effectiveness of AMUSE in simulated data.

cs.LG↗

arXiv2024

Ethical considerations of use of hold-out sets in clinical prediction model management

Clinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some patients if a model is updated on this data. One suggested resolution to this is the use of hold-out sets, in which a set of patients do not receive model derived risk scores, such that a model can be safely retrained. We present an overview of clinical and research ethics regarding potential implementation of hold-out sets for clinical prediction models in health settings. We focus on the ethical principles of beneficence, non-maleficence, autonomy and justice. We also discuss informed consent, clinical equipoise, and truth-telling. We present illustrative cases of potential hold-out set implementations and discuss statistical issues arising from different hold-out set sampling methods. We also discuss differences between hold-out sets and randomised control trials, in terms of ethics and statistical issues. Finally, we give practical recommendations for researchers interested in the use hold-out sets for clinical prediction models.

cs.LG↗

arXiv2022

Safe machine learning model release from Trusted Research Environments: The SACRO-ML package

We present SACRO-ML, an integrated suite of open source Python tools to facilitate the statistical disclosure control (SDC) of machine learning (ML) models trained on confidential data prior to public release. SACRO-ML combines (i) a SafeModel package that extends commonly used ML models to provide ante-hoc SDC by assessing the vulnerability of disclosure posed by the training regime; and (ii) an Attacks package that provides post-hoc SDC by rigorously assessing the empirical disclosure risk of a model through a variety of simulated attacks after training. The SACRO-ML code and documentation are available under an MIT license at https://github.com/AI-SDC/SACRO-ML

cs.LG↗

arXiv2022

GRAIMATTER Green Paper: Recommendations for disclosure control of trained Machine Learning (ML) models from Trusted Research Environments (TREs)

TREs are widely, and increasingly used to support statistical analysis of sensitive data across a range of sectors (e.g., health, police, tax and education) as they enable secure and transparent research whilst protecting data confidentiality. There is an increasing desire from academia and industry to train AI models in TREs. The field of AI is developing quickly with applications including spotting human errors, streamlining processes, task automation and decision support. These complex AI models require more information to describe and reproduce, increasing the possibility that sensitive personal data can be inferred from such descriptions. TREs do not have mature processes and controls against these risks. This is a complex topic, and it is unreasonable to expect all TREs to be aware of all risks or that TRE researchers have addressed these risks in AI-specific training. GRAIMATTER has developed a draft set of usable recommendations for TREs to guard against the additional risks when disclosing trained AI models from TREs. The development of these recommendations has been funded by the GRAIMATTER UKRI DARE UK sprint research project. This version of our recommendations was published at the end of the project in September 2022. During the course of the project, we have identified many areas for future investigations to expand and test these recommendations in practice. Therefore, we expect that this document will evolve over time.

cs.LG↗

arXiv2022

Holdouts set for safe predictive model updating

Predictive risk scores for adverse outcomes are increasingly crucial in guiding health interventions. Such scores may need to be periodically updated due to change in the distributions they model. However, directly updating risk scores used to guide intervention can lead to biased risk estimates. To address this, we propose updating using a `holdout set' - a subset of the population that does not receive interventions guided by the risk score. Balancing the holdout set size is essential to ensure good performance of the updated risk score whilst minimising the number of held out samples. We prove that this approach reduces adverse outcome frequency to an asymptotically optimal level and argue that often there is no competitive alternative. We describe conditions under which an optimal holdout size (OHS) can be readily identified, and introduce parametric and semi-parametric algorithms for OHS estimation. We apply our methods to the ASPRE risk score for pre-eclampsia to recommend a plan for updating it in the presence of change in the underlying data distribution. We show that, in order to minimise the number of pre-eclampsia cases over time, this is best achieved using a holdout set of around 10,000 individuals.

stat.ML↗

arXiv2021

Stacking interventions for equitable outcomes

Predictive risk scores estimating probabilities for a binary outcome on the basis of observed covariates are common across the sciences. They are frequently developed with the intent of avoiding the outcome in question by intervening in response to estimated risks. Since risk scores are usually developed in complex systems, interventions usually take the form of expert agents responding to estimated risks as they best see fit. In this case, interventions may be complex and their effects difficult to observe or infer, meaning that explicit specification of interventions in response to risk scores is impractical. Scope to modulate the aggregate model-intervention scheme so as to optimise an objective is hence limited. We propose an algorithm by which a model-intervention scheme can be developed by 'stacking' possibly unknown intervention effects. By repeatedly observing and updating the intervention and model, we show that this scheme leads to convergence or almost-convergence of eventual outcome risk to an equivocal value for any initial value of covariates. Our approach deploys a series of risk scores to expert agents, with instructions to act on them in succession according to their best judgement. Our algorithm uses only observations of pre-intervention covariates and the eventual outcome as input. It is not necessary to know or infer the effect of the intervention, other than making a general assumption that it is 'well-intentioned'. The algorithm can also be used to safely update risk scores in the presence of unknown interventions and concept drift. We demonstrate convergence of expectation of outcome in a range of settings and show robustness to errors in risk estimation and to concept drift. We suggest several practical applications and demonstrate a potential implementation by simulation, showing that the algorithm leads to a fair distribution of outcome risk across a population.

stat.ME↗

arXiv2020

Model updating after interventions paradoxically introduces bias

Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.

stat.ML↗

arXiv2017

Combining controls can improve power in two-stage association studies

High dimensional case control studies are ubiquitous in the biological sciences, particularly genomics. To maximise power while constraining cost and to minimise type-1 error rates, researchers typically seek to replicate findings in a second experiment on independent cohorts before proceeding with further analyses. This paper presents a method in which control (or case) samples from the discovery cohort are re-used in the replication study. The theoretical implications of this method are discussed and simulations used to compare performance against the standard method in a range of circumstances. In several common study designs, a shared-control method allows a substantial improvement in power while retaining type-1 error rate control. An important area of potential application arises when control samples are difficult to recruit or ascertain; for example in inter-disease comparisons, or studies on degenerative diseases. Using similar methods, a procedure is proposed for `partial replication' using a new independent cohort consisting of only controls. This methods can be used to provide some validation of findings when a full replication procedure is not possible. The new method has differing sensitivity to confounding in study cohorts compared to the standard procedure, which must be considered in its application. Type-1 error rates in these scenarios are analytically and empirically derived, and an online tool for comparing power and error rates is provided. Although careful consideration must be made of all necessary assumptions, this method can enable more efficient use of data in genome-wide association studies (GWAS) and other applications.

stat.ME↗