arXiv Science⌕ Search

arXiv subjects

Soyon Choi

Publications and source records attributed to Soyon Choi.

2 recordsLinked to original sources

SHERLOC: Statistical Hypothesis Evaluation and Recovery for Learning Opponent Characteristics

When used in automated decision-making systems, machine learning (ML) models are vulnerable to data-manipulation attacks. Some defense mechanisms (e.g., adversarial regularization) directly affect the ML models while others (e.g., anomaly detection) act within the broader system. However, many defenses operate under a predefined threat model that assumes characteristics of the adversary, including their knowledge, capabilities, and objectives, are known and fixed. In practice, these characteristics may be unknown or evolve over time, creating a mismatch between the assumptions of the defender and the true attacker. In this paper, we study a new perspective on adversarial defense: focusing on the attacker, rather than the attack. We present and demonstrate SHERLOC, a framework for adversarial learning forensics -- namely, a detective that learns about the attacker from observed attack(s). We prove that, without additional knowledge, the attacker is non-identifiable --- multiple potential attackers would perform the same observed attack. This reveals an important distinction between reconstructing an attacker's behavior and recovering the attacker that generated it. We demonstrate empirically that SHERLOC is able to reconstruct a hypothesis attacker that is consistent with the observed attacks against various defender models (i.e., generalized least-squares linear regressors, logistic regressors, and polynomial classifiers) across various security-critical domains. Finally, we show that these inferred attackers need not match the ground truth attacker exactly to have downstream utility. Defenses using the SHERLOC-inferred attackers achieve approximately 90-100% of robustness improvement that would be achieved if the defender had access to the ground-truth attacker.

cs.LG↗

Data Manipulation Attacks Against Machine Learning Integrated Cyber-Physical Systems

There is a well-established body of work demonstrating that individual machine learning (ML) algorithms can be manipulated or deceived. However, in many real-world systems, multiple ML algorithms are composed to enable automated decision-making, particularly in Cyber-Physical Systems (CPS). Often such systems use a composition of vision, planning, and other ML models to affect their internal state and ultimate take actions in the physical world. While there is a huge literature on the (in)security of ML algorithms, it is not immediately clear how an attacker might leverage their ability to trick individual learners when attacking the composition as a whole. We develop methods for creating an attack policy (a mapping from state to attack) for a data-manipulating attacker affecting a CPS or other state-machine-based AI compositions. We describe our framework, a basic threat model and corresponding methods for computing an optimal attack policy, as well as planned extensions. The goal of this work-in-progress paper is to stimulate conversation and forge new collaborations regarding the security of compositional learning.

cs.LG↗