arXiv subjects

Kevin Bennett

Publications and source records attributed to Kevin Bennett.

3 recordsLinked to original sources

arXiv2024

Comparing Template-based and Template-free Language Model Probing

The differences between cloze-task language model (LM) probing with 1) expert-made templates and 2) naturally-occurring text have often been overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets -- 4 template-based and 6 template-free -- in general and biomedical domains to answer the following research questions: (RQ1) Do model rankings differ between the two approaches? (RQ2) Do models' absolute scores differ between the two approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and domain-specific models? Our findings are: 1) Template-free and template-based approaches often rank models differently, except for the top domain-specific models. 2) Scores decrease by up to 42% Acc@1 when comparing parallel template-free and template-based prompts. 3) Perplexity is negatively correlated with accuracy in the template-free approach, but, counter-intuitively, they are positively correlated for template-based probing. 4) Models tend to predict the same answers frequently across prompts for template-based probing, which is less common when employing template-free techniques.

cs.CL↗

arXiv2023

Emerging Challenges in Personalized Medicine: Assessing Demographic Effects on Biomedical Question Answering Systems

State-of-the-art question answering (QA) models exhibit a variety of social biases (e.g., with respect to sex or race), generally explained by similar issues in their training data. However, what has been overlooked so far is that in the critical domain of biomedicine, any unjustified change in model output due to patient demographics is problematic: it results in the unfair treatment of patients. Selecting only questions on biomedical topics whose answers do not depend on ethnicity, sex, or sexual orientation, we ask the following research questions: (RQ1) Do the answers of QA models change when being provided with irrelevant demographic information? (RQ2) Does the answer of RQ1 differ between knowledge graph (KG)-grounded and text-based QA systems? We find that irrelevant demographic information change up to 15% of the answers of a KG-grounded system and up to 23% of the answers of a text-based system, including changes that affect accuracy. We conclude that unjustified answer changes caused by patient demographics are a frequent phenomenon, which raises fairness concerns and should be paid more attention to.

cs.CL↗

arXiv2018

Importance of Amplifier Physics in Maximizing the Capacity of Submarine Links

The throughput of submarine transport cables is approaching fundamental limits imposed by amplifier noise and Kerr nonlinearity. Energy constraints in ultra-long submarine links exacerbate this problem, as the throughput per fiber is further limited by the electrical power available to the undersea optical amplifiers. Recent works have studied how employing more spatial dimensions can mitigate these limitations. In this paper, we address the fundamental question of how to optimally use each spatial dimension. Specifically, we discuss how to optimize the channel power allocation in order to maximize the information-theoretic capacity under an electrical power constraint. Our formulation accounts for amplifier physics, Kerr nonlinearity, and power feed constraints. Whereas recent works assume the optical amplifiers operate in deep saturation, where power-conversion efficiency (PCE) is high, we show that given a power constraint, operating in a less saturated regime, where PCE is lower, supports a wider bandwidth and a larger number of spatial dimensions, thereby maximizing capacity. This design strategy increases the capacity of submarine links by about 70% compared to the theoretical capacity of a recently proposed high-capacity system.

physics.app-ph↗