arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2008.13637

Should policy makers trust composite indices? A commentary on the pitfalls of inappropriate indices for policy formation

Matthias Kaiser·Andrew Tzer-Yeu Chen·Peter Gluckman
View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper critically discusses the use and merits of global indices, in particular, the Global Health Security Index or GHSI (Cameron et 2019) in times of an imminent crisis, like the current pandemic. The index ranked 195 countries according to their expected preparedness in case of a pandemic or other biological threat. The Covid-19 pandemic provides the background to compare each country's predicted performance from the GHSI with the actual performance. In general, there is an inverted relation between predicted versus actual performance, i.e. the predicted top performers are among those that are the worst hit. Obviously, this reflects poorly on the potential policy uses of the index in imminent crisis management. The paper also uses two different data sets, one from the Worldmeter on the spread of the Covid-19 pandemics, and the other one from the INGSA policy tracker, to make comparisons between the actual introduction of pandemic response policies and the corresponding death rate in 29 selected countries.

Explore related subjects

cs.CYstat.OT

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Matthias Kaiser, Andrew Tzer-Yeu Chen, Peter Gluckman. 2021-03-03. Should policy makers trust composite indices? A commentary on the pitfalls of inappropriate indices for policy formation. https://doi.org/10.1186/s12961-021-00702-4

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Towards interactive evaluations for interaction harms in human-AI systems

Current AI evaluation methods, which rely on static, model-only tests, fail to account for harms that emerge through sustained human-AI interaction. As AI systems proliferate and are increasingly integrated into real-world applications, this disconnect between evaluation approaches and actual usage becomes more significant. In this paper, we propose a shift towards evaluation based on \textit{interactional ethics}, which focuses on \textit{interaction harms} - issues like inappropriate parasocial relationships, social manipulation, and cognitive overreliance that develop over time through repeated interaction, rather than through isolated outputs. First, we discuss the limitations of current evaluation methods, which (1) are static, (2) assume a universal user experience, and (3) have limited construct validity. Drawing on research from human-computer interaction, natural language processing, and the social sciences, we present practical principles for designing interactive evaluations. These include ecologically valid interaction scenarios, human impact metrics, and diverse human participation approaches. Finally, we explore implementation challenges and open research questions for researchers, practitioners, and regulators aiming to integrate interactive evaluations into AI governance frameworks. This work lays the groundwork for developing more effective evaluation methods that better capture the complex dynamics between humans and AI systems.

cs.CY↗
arXiv2026

Enhancing Large Language Model-Based Systems for End-to-End Circuit Analysis Problem Solving

LLMs have shown strong performance in data-rich domains such as programming, but their reliability in engineering tasks remains limited. Circuit analysis is particularly challenging because it requires both multimodal understanding and precise mathematical reasoning. This paper presents an enhanced end-to-end circuit problem-solving framework using Gemini 2.5 Pro as the backbone model for scalable engineering-education applications. We systematically evaluate Gemini 2.5 Pro on undergraduate circuit-analysis problems and identify two major failure modes: circuit-recognition hallucinations, especially source-polarity errors, and reasoning-process hallucinations, such as incorrect current-direction assumptions. To reduce recognition errors, we integrate a fine-tuned YOLO detector with OpenCV-based processing to isolate voltage and current sources for polarity re-identification. To mitigate reasoning errors, we introduce an ngspice-driven verification loop that supports iterative refinement with optional human feedback. On 83 problems, the proposed pipeline achieves 97.59% accuracy, compared with 79.52% for baseline Gemini. Across four hand-drawn diagram variations, accuracy improves from 60.61%--71.21% to 89.39%--92.42%, with statistically significant gains (p<0.005). On 43 problems from a different textbook, accuracy increases from 58.14% to 83.72%, further supporting cross-textbook generalizability. Error analysis shows that circuit recognition remains the dominant source of residual failures, particularly under varying diagram representations. Overall, the framework substantially improves the robustness, scalability, and generalizability of LLM-based circuit problem solving for engineering education and practical circuit analysis.

cs.CY↗
arXiv2026

GreenPassport: Request-Level Carbon Accounting for Cross-Border AI Inference

AI inference often crosses regional boundaries as prompts travel to remote data centers and generated tokens return to users. Regional averages cannot represent the resulting differences in serving hardware, electricity, and network delivery. Request-level accounting needs a common boundary for the service, serving site, route, local comparator, uncertainty, and data provenance. GreenPassport Carbon Accounting (GPCA) associates these inputs with each request. It estimates serving and route carbon, then selects a reporting level from the available documentation. Our public-data implementation covers data-center instances, accelerators, model families, electricity mixes, routes, and cloud-region carbon intensity. Against six accounting baselines and four energy-prediction baselines, GPCA reduced median absolute percentage error by 56.3% and median absolute error by 15.5% relative to EcoLogits under the aligned accelerator-energy boundary. It produced zero rule overstatement in the deterministic conformance tests. In the buyer case, the clean-electricity CN-West scenario produced 0.0148 gCO2e per request, 88% below the local service at 0.1220 gCO2e per request.

cs.CY↗