Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center

James Wen; Sahil Nalawade; Zhiwei Liang; Catherine Bielick; Marisa Ferrara Boston; Alexander Chowdhury; Adele Collin; Luigi De Angelis; Jacob Ellen; Heather Frase; Rodrigo R. Gameiro; Juan Manuel Gutierrez; Pooja Kadam; Murat Keceli; Srikanth Krishnamurthy; Anne Kwok; Yanan Lance Lu; Heather Mattie; Liam G. McCoy; Katherine Miller; Allison C. Morgan; Marlene Louisa Moerig; Trang Nguyen; Alexander Owen-Post; Alex D. Ruiz; Sreekar Reddy Puchala; Soujanya Samineni; Takeshi Tohyama; Varun Ullanat; Carmine Valenza; Camilo Velez; Pengcheng Wang; Anna Wuest; Yuxiang Zhou; Yingde Zhu; Jason M. Johnson; Naomi Lenane; Jennifer Willcox; Francis J. Vitiello; Leo Anthony G. Celi; Renato Umeton

arXiv · 2506.22523

Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center

View original source ↗Read source document ↗Export metadata ↓

Abstract

Background: Generative artificial intelligence (AI) deployment in academic medical settings raises copyright compliance concerns. Dana-Farber Cancer Institute implemented GPT4DFCI, an internal generative AI tool utilizing OpenAI models, that is approved for enterprise use in research and operations. Given (1) the exceptionally broad adoption of the tool in our organization, (2) our research mission, and (3) the shared responsibility model required to benefit from Customer Copyright Commitment in Azure OpenAI Service products, we deemed rigorous copyright compliance testing necessary. Case Description: We conducted a structured red teaming exercise in Nov. 2024, with 42 participants from academic, industry, and government institutions. Four teams attempted to extract copyrighted content from GPT4DFCI across four domains: literary works, news articles, scientific publications, and access-restricted clinical notes. Teams successfully extracted verbatim book dedications and near-exact passages through various strategies. News article extraction failed despite jailbreak attempts. Scientific article reproduction yielded only high-level summaries. Clinical note testing revealed appropriate privacy safeguards. Discussion: The successful extraction of literary content indicates potential copyrighted material presence in training data, necessitating inference-time filtering. Differential success rates across content types suggest varying protective mechanisms. The event led to implementation of a copyright-specific meta-prompt in GPT4DFCI; this mitigation has been in production since Jan. 2025. Conclusion: Systematic red teaming revealed specific vulnerabilities in generative AI copyright compliance, leading to concrete mitigation strategies. Academic medical institutions deploying generative AI should implement continuous testing protocols to ensure legal and ethical compliance.

Explore related subjects

cs.CY cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

James Wen, Sahil Nalawade, Zhiwei Liang, Catherine Bielick, Marisa Ferrara Boston, Alexander Chowdhury, Adele Collin, Luigi De Angelis, Jacob Ellen, Heather Frase, Rodrigo R. Gameiro, Juan Manuel Gutierrez, Pooja Kadam, Murat Keceli, Srikanth Krishnamurthy, Anne Kwok, Yanan Lance Lu, Heather Mattie, Liam G. McCoy, Katherine Miller, Allison C. Morgan, Marlene Louisa Moerig, Trang Nguyen, Alexander Owen-Post, Alex D. Ruiz, Sreekar Reddy Puchala, Soujanya Samineni, Takeshi Tohyama, Varun Ullanat, Carmine Valenza, Camilo Velez, Pengcheng Wang, Anna Wuest, Yuxiang Zhou, Yingde Zhu, Jason M. Johnson, Naomi Lenane, Jennifer Willcox, Francis J. Vitiello, Leo Anthony G. Celi, Renato Umeton. 2025-07-02. Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center. https://arxiv.org/abs/2506.22523

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Funding the runners-up beats a golden ticket

A golden ticket allows one reviewer to advance a proposal that the rest of a panel would reject. We assess its score-based rationale by comparing minority-support ranking with other rules for allocating the same number of places. We do so on 10,625 rejected submissions to a computer-science conference between 2017 and 2024, using later citation percentiles as the outcome. Minority-support ranking measured the gap between the two highest scores after adjustment for the panel mean and reviewer count. At a budget of 10% of the year's accepted count, panel-mean ranking selected submissions with a mean citation percentile 7.75 points higher than minority-support ranking (95% bootstrap interval 4.35 to 11.09). The difference was larger in the other sample periods and citation windows examined. At that budget, minority-support ranking was equivalent to a lottery within a margin of plus or minus 5 percentile points. Two further analyses, adjusted highest-score ranking and a matched comparison, found no clear citation advantage from minority support once the panel mean was accounted for. Ranking on score variance also outperformed minority-support ranking in a paired comparison. Estimates cover the 69.8% of eligible rejections whose later citations could be verified. These comparisons identify panel-mean ranking as a benchmark for evaluating whether golden tickets select work that would otherwise be overlooked.

cs.CY↗

arXiv2026

Who Decides? Agency and Legitimacy in Digital Educational Systems

Digital Educational Systems (DES) enable millions of individuals to learn or to acquire new skills. Their promises vary from one platform to another: some grant genuine degrees, others pledge rapid progress, drawing for this purpose on a range of techniques, from the recommendation of learning resources through complex models to a plain course textbook uploaded onto a learning management system. In this article, we argue that categorizing DES along two dimensions allows to map differences in the way these DES structurally conceptualise the learning process. These underlying dimensions are at once dimensions of the student model and mechanisms built in the DES's designs; we identify them as agency and legitimacy. Indeed, DES are, on the one hand, anchored in an institutionalised regime that grants them legitimacy and, in turn, legitimises the learning undertaken on these platforms; on the other hand, they make design choices as to the share of decision left to the learner in the learning process. We map a curated sample of common DES onto a plane composed of these two dimensions translated into indicators. This allows us to highlight underpopulated regions of the plane and to envisage how a DES might regulate these two dimensions algorithmically, in the interest of learning.

cs.CY↗

arXiv2026

Governance-as-Code: Translating EU AI Act Technical Requirements into Executable Compliance Pipelines for Generative AI Systems

The EU AI Act (Regulation 2024/1689) imposes technical obligations on high-risk AI providers, yet Articles 8-15 were drafted for predictive AI and leave seven technical gaps when applied to generative systems, spanning non-deterministic data governance, training-data provenance, continuous conformity, human oversight, open-ended robustness, emergent risk, and generative fairness. We deliver Governance-as-Code (GaC), a framework of 43 machine-checkable acceptance criteria across six compliance modules that run in a CI/CD pipeline and emit Article-indexed audit evidence, and we show the actual Rego policy code rather than merely describing it. Our central commitment is that the Act's open-textured standards ("appropriate levels," "possible biases") become declared, auditable numbers: robustness thresholds are derived from the provider's documented baseline and a state-of-the-art floor, and framing bias is collapsed into eight measurable proxies tested by counterfactual demographic probing. We also correct who owes what, since under Article 25 and Chapter V a downstream deployer relies on the upstream provider's Article 53 training-data summary and documents only the layers it controls, so GaC verifies that summary rather than demanding per-sample documentation the deployer never had. We validate on two enterprise deployments, a high-risk advisory chatbot and a limited-risk content generator, benchmarking against a manual expert audit rather than documentation artifacts that were never designed to enforce compliance. GaC reproduces all of the manual audit's findings, including three penalty-triggering violations, while cutting audit labor by roughly 75%.

cs.CY↗