arXiv subjects

Daniel Jones

Publications and source records attributed to Daniel Jones.

13 recordsLinked to original sources

arXiv2026

The Challenge of Cell Segmentation in Spatially Resolved Transcriptomics

Spatially resolved transcriptomics (SRT) is transforming how we study tissues by measuring gene expression in cells in their spatial context. However, the field lacks robust methodological guidance on one of its most fundamental analytical steps: how to accurately segment cells and assign spatially localized transcripts to them. Major technical challenges include sparse molecular signals, transcript displacement, complex cellular morphologies, and the projection of three-dimensional tissue architecture onto two-dimensional imaging planes. These challenges make segmentation a major source of uncertainty, with errors that can propagate through downstream analyses and ultimately lead to misleading biological interpretations. Here, we argue that segmentation should be treated as a central unresolved problem in spatial omics rather than a routine preprocessing step. We review current approaches, highlight key methodological limitations, including the lack of appropriate metrics and gold-standard benchmarks, and propose a community-driven path forward. Establishing shared evaluation frameworks, scalable benchmark datasets, and transparent reporting standards will be essential for transforming SRT into a robust and reproducible foundation for biological discovery and clinical translation.

q-bio.OT↗

arXiv2025

A Systematization of Security Vulnerabilities in Computer Use Agents

Computer Use Agents (CUAs), autonomous systems that interact with software interfaces via browsers or virtual machines, are rapidly being deployed in consumer and enterprise environments. These agents introduce novel attack surfaces and trust boundaries that are not captured by traditional threat models. Despite their growing capabilities, the security boundaries of CUAs remain poorly understood. In this paper, we conduct a systematic threat analysis and testing of real-world CUAs under adversarial conditions. We identify seven classes of risks unique to the CUA paradigm, and analyze three concrete exploit scenarios in depth: (1) clickjacking via visual overlays that mislead interface-level reasoning, (2) indirect prompt injection that enables Remote Code Execution (RCE) through chained tool use, and (3) CoT exposure attacks that manipulate implicit interface framing to hijack multi-step reasoning. These case studies reveal deeper architectural flaws across current CUA implementations. Namely, a lack of input provenance tracking, weak interface-action binding, and insufficient control over agent memory and delegation. We conclude by proposing a CUA-specific security evaluation framework and design principles for safe deployment in adversarial and high-stakes settings.

cs.CR↗

arXiv2025

A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks

Recent research has demonstrated that state-of-the-art LLMs and defenses remain susceptible to multi-turn jailbreak attacks. These attacks require only closed-box model access and are often easy to perform manually, posing a significant threat to the safe and secure deployment of LLM-based systems. We study the effectiveness of the Crescendo multi-turn jailbreak at the level of intermediate model representations and find that safety-aligned LMs often represent Crescendo responses as more benign than harmful, especially as the number of conversation turns increases. Our analysis indicates that at each turn, Crescendo prompts tend to keep model outputs in a "benign" region of representation space, effectively tricking the model into fulfilling harmful requests. Further, our results help explain why single-turn jailbreak defenses like circuit breakers are generally ineffective against multi-turn attacks, motivating the development of mitigations that address this generalization gap.

cs.CR↗

arXiv2025

Lessons From Red Teaming 100 Generative AI Products

In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.

cs.AI↗

arXiv2024

Controllable Synthetic Clinical Note Generation with Privacy Guarantees

In the field of machine learning, domain-specific annotated data is an invaluable resource for training effective models. However, in the medical domain, this data often includes Personal Health Information (PHI), raising significant privacy concerns. The stringent regulations surrounding PHI limit the availability and sharing of medical datasets, which poses a substantial challenge for researchers and practitioners aiming to develop advanced machine learning models. In this paper, we introduce a novel method to "clone" datasets containing PHI. Our approach ensures that the cloned datasets retain the essential characteristics and utility of the original data without compromising patient privacy. By leveraging differential-privacy techniques and a novel fine-tuning task, our method produces datasets that are free from identifiable information while preserving the statistical properties necessary for model training. We conduct utility testing to evaluate the performance of machine learning models trained on the cloned datasets. The results demonstrate that our cloned datasets not only uphold privacy standards but also enhance model performance compared to those trained on traditional anonymized datasets. This work offers a viable solution for the ethical and effective utilization of sensitive medical data in machine learning, facilitating progress in medical research and the development of robust predictive models.

cs.CL↗

arXiv2023

Investigating the Effect of Misalignment on Membership Privacy in the White-box Setting

Machine learning models have been shown to leak sensitive information about their training datasets. Models are increasingly deployed on devices, raising concerns that white-box access to the model parameters increases the attack surface compared to black-box access which only provides query access. Directly extending the shadow modelling technique from the black-box to the white-box setting has been shown, in general, not to perform better than black-box only attacks. A potential reason is misalignment, a known characteristic of deep neural networks. In the shadow modelling context, misalignment means that, while the shadow models learn similar features in each layer, the features are located in different positions. We here present the first systematic analysis of the causes of misalignment in shadow models and show the use of a different weight initialisation to be the main cause. We then extend several re-alignment techniques, previously developed in the model fusion literature, to the shadow modelling context, where the goal is to re-align the layers of a shadow model to those of the target model. We show re-alignment techniques to significantly reduce the measured misalignment between the target and shadow models. Finally, we perform a comprehensive evaluation of white-box membership inference attacks (MIA). Our analysis reveals that internal layer activation-based MIAs suffer strongly from shadow model misalignment, while gradient-based MIAs are only sometimes significantly affected. We show that re-aligning the shadow models strongly improves the former's performance and can also improve the latter's performance, although less frequently. Taken together, our results highlight that on-device deployment increases the attack surface and that the newly available information can be used to build more powerful attacks.

cs.CR↗

arXiv2022

Bayesian Estimation of Differential Privacy

Algorithms such as Differentially Private SGD enable training machine learning models with formal privacy guarantees. However, there is a discrepancy between the protection that such algorithms guarantee in theory and the protection they afford in practice. An emerging strand of work empirically estimates the protection afforded by differentially private training as a confidence interval for the privacy budget $\varepsilon$ spent on training a model. Existing approaches derive confidence intervals for $\varepsilon$ from confidence intervals for the false positive and false negative rates of membership inference attacks. Unfortunately, obtaining narrow high-confidence intervals for $\epsilon$ using this method requires an impractically large sample size and training as many models as samples. We propose a novel Bayesian method that greatly reduces sample size, and adapt and validate a heuristic to draw more than one sample per trained model. Our Bayesian method exploits the hypothesis testing interpretation of differential privacy to obtain a posterior for $\varepsilon$ (not just a confidence interval) from the joint posterior of the false positive and false negative rates of membership inference attacks. For the same sample size and confidence, we derive confidence intervals for $\varepsilon$ around 40% narrower than prior work. The heuristic, which we adapt from label-only DP, can be used to further reduce the number of trained models needed to get enough samples by up to 2 orders of magnitude.

cs.LG↗

arXiv2021

Training Data Leakage Analysis in Language Models

Recent advances in neural network based language models lead to successful deployments of such models, improving user experience in various applications. It has been demonstrated that strong performance of language models comes along with the ability to memorize rare training samples, which poses serious privacy threats in case the model is trained on confidential user content. In this work, we introduce a methodology that investigates identifying the user content in the training data that could be leaked under a strong and realistic threat model. We propose two metrics to quantify user-level data leakage by measuring a model's ability to produce unique sentence fragments within training data. Our metrics further enable comparing different models trained on the same data in terms of privacy. We demonstrate our approach through extensive numerical studies on both RNN and Transformer based models. We further illustrate how the proposed metrics can be utilized to investigate the efficacy of mitigations like differentially private training or API hardening.

cs.CR↗

arXiv2014

Modelling of dependence in high-dimensional financial time series by cluster-derived canonical vines

We extend existing models in the financial literature by introducing a cluster-derived canonical vine (CDCV) copula model for capturing high dimensional dependence between financial time series. This model utilises a simplified market-sector vine copula framework similar to those introduced by Heinen and Valdesogo (2008) and Brechmann and Czado (2013), which can be applied by conditioning asset time series on a market-sector hierarchy of indexes. While this has been shown by the aforementioned authors to control the excessive parameterisation of vine copulas in high dimensions, their models have relied on the provision of externally sourced market and sector indexes, limiting their wider applicability due to the imposition of restrictions on the number and composition of such sectors. By implementing the CDCV model, we demonstrate that such reliance on external indexes is redundant as we can achieve equivalent or improved performance by deriving a hierarchy of indexes directly from a clustering of the asset time series, thus abstracting the modelling process from the underlying data.

q-fin.ST↗

arXiv2010

Effect of promoter architecture on the cell-to-cell variability in gene expression

According to recent experimental evidence, the architecture of a promoter, defined as the number, strength and regulatory role of the operators that control the promoter, plays a major role in determining the level of cell-to-cell variability in gene expression. These quantitative experiments call for a corresponding modeling effort that addresses the question of how changes in promoter architecture affect noise in gene expression in a systematic rather than case-by-case fashion. In this article, we make such a systematic investigation, based on a simple microscopic model of gene regulation that incorporates stochastic effects. In particular, we show how operator strength and operator multiplicity affect this variability. We examine different modes of transcription factor binding to complex promoters (cooperative, independent, simultaneous) and how each of these affects the level of variability in transcription product from cell-to-cell. We propose that direct comparison between in vivo single-cell experiments and theoretical predictions for the moments of the probability distribution of mRNA number per cell can discriminate between different kinetic models of gene regulation.

q-bio.MN↗

arXiv2006

Evolution of Characteristic Quantities for Dark Matter Halo Density Profiles

We investigate the effect of an assembly history on the evolution of dark matter (DM) halos of 10^{12} Msun/h using Constrained Realizations of random Gaussian fields. Five different realizations of a DM halo with distinct merging histories were constructed and evolved. Our main results are: A halo evolves via a sequence of quiescent phases of a slow mass accretion intermitted by violent episodes of major mergers. In the quiescent phases, the density is well fitted by an NFW profile, the inner scale radius Rs and the mass enclosed within it remain constant, and the virial radius (Rvir) grows linearly with the expansion parameter "a". Within each quiescent phase the concentration parameter ("c") scales as "a", and the mass accretion history (Mvir) is well described by the Tasitsiomi etal. fitting formula. In the violent phases the halos are not in a virial equilibrium and both Rs and Rvir grow discontinuously. The violent episodes drive the halos from one NFW dynamical equilibrium to another. The final structure of a halo, including "c", depends on the degree of violence of the major mergers and on their number. Next, we find a distinct difference between the behavior of various NFW parameters taken as averages over an ensemble of halos and those of individual halos. Moreover, the simple scaling relations c--Mvir do not apply to the entire evolution of individual halos, and so is the common notion that late forming halos are less concentrated than early forming ones. The entire evolution of the halo cannot be fitted by single analytical expressions.

astro-ph↗

arXiv2005

Constrained Simulations of Dark Matter Halos

The formation and structure of dark matter halos is studied by constrained simulations. A series of experiments of the formation of a 10^12 Msun/h halo is designed to study the dependence of the density profile on its merging history. We find that the halo growth consist of several quiescent phases intermitted by violent events, with the density well approximated by the NFW profile during the former phases. We find that (1) the NFW scale radius R_s stays constant during the quiescent phase and grows abruptly during the violent one. In contrast, the virial radius grows linearly during the quiescent and abruptly during the violent phases. (2) The central density stays unchanged during the quiescent phase while dropping abruptly during the violent phase, and it does not reflect the formation time of the halo. (3) The clear separation of the evolution of an individual halo into quiescent and violent phases implies that its entire evolution cannot be fitted by simple scaling relations.

astro-ph↗

arXiv2005

Constrained Cosmological Simulations of Dark Matter Halos

The formation and structure of dark matter (DM) halos is studied by means of constrained realizations of Gaussian fields using N-body simulations. A series of experiments of the formation of a 10^{12} Msun halo is designed to study the dependence of the density profile on its merging history. We confirm that the halo growth consists of violent and quiescent phases, with the density well approximated by the Navarro-Frenk-White (NFW) profile during the latter phases. We find that (1) the NFW scale radius R_s stays constant during the quiescent phase and grows abruptly during the violent one. In contrast, the virial radius grows linearly during the quiescent and abruptly during the violent phases. (2) The central density stays unchanged during the quiescent phase while dropping abruptly during the violent phase. (3) The value of \rs reflects the violent merging history of the halo, and depends on the number of violent events and their fractional magnitudes, independent of the time and order of these events. It does not reflect the formation time of the halo. (4) The fractional change in R_s is a nonlinear function of the fractional absorbed kinetic energy within R_s in a violent event.

astro-ph↗