Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam

Xinran Wang; Boran Zhu; Shujuan Zhou; Ziwen Long; Dehua Zhou; Shu Zhang

Home/Archive/arXiv

arXiv · 2511.20526

Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam

Xinran Wang·Boran Zhu·Shujuan Zhou·Ziwen Long·Dehua Zhou·Shu Zhang

View original source ↗Read source document ↗Export metadata ↓

Abstract

Background: As large language models (LLMs) become increasingly integrated into digital health education and assessment workflows, their capabilities in supporting high-stakes, domain-specific certification tasks remain underexplored.In China, the national pharmacist licensure exam serves as a standardized benchmark for evaluating pharmacists' clinical and theoretical competencies. Objective: This study aimed to compare the performance of two LLMs: ChatGPT-4o and DeepSeek-R1 on real questions from the Chinese Pharmacist Licensing Examination (2017-2021), and to discuss the implications of these performance differences for AI-enabled formative evaluation. Methods: A total of 2,306 multiple-choice (text-only) questions were compiled from official exams, training materials, and public databases. Questions containing tables or images were excluded. Each item was input in its original Chinese format, and model responses were evaluated for exact accuracy. Pearson's Chi-squared test was used to compare overall performance, and Fisher's exact test was applied to year-wise multiple-choice accuracy. Results: DeepSeek-R1 outperformed ChatGPT-4o with a significantly higher overall accuracy (90.0% vs. 76.1%, p < 0.001). Unit-level analyses revealed consistent advantages for DeepSeek-R1, particularly in foundational and clinical synthesis modules. While year-by-year multiple-choice performance also favored DeepSeek-R1, this performance gap did not reach statistical significance in any specific unit-year (all p > 0.05). Conclusion: DeepSeek-R1 demonstrated robust alignment with the structural and semantic demands of the pharmacist licensure exam. These findings suggest that domain-specific models warrant further investigation for this context, while also reinforcing the necessity of human oversight in legally and ethically sensitive contexts.

Explore related subjects

cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Xinran Wang, Boran Zhu, Shujuan Zhou, Ziwen Long, Dehua Zhou, Shu Zhang. 2025-11-25. Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam. https://arxiv.org/abs/2511.20526

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignals

Biosignals collected from wearable devices are widely utilized in healthcare applications. Machine learning models used in these applications often rely on features extracted from biosignals due to their effectiveness, lower data dimensionality, and wide compatibility across various model architectures. However, existing feature extraction methods often lack task-specific contextual knowledge, struggle to identify optimal features in high-dimensional combinatorial feature space, and are prone to automated code generation and execution errors. In this paper, we propose DeepFeature, the first LLM-empowered, context-aware feature generation framework for wearable biosignals. DeepFeature introduces a multi-source feature generation mechanism that integrates the inherent ability of LLMs, expert knowledge and inter-feature interactions. It also employs an iterative feature refinement process that uses feature assessment-based feedback for feature re-selection. Additionally, DeepFeature utilizes a robust multi-layer filtering and verification approach for feature description-to-code translation to ensure that the feature extraction functions run without crashing. Experimental evaluation results show that DeepFeature achieves the highest average AUROC across eight tasks under both sample-level and subject-level settings, outperforming the best baselines by 4.60% and 4.61%, respectively. DeepFeature achieves the most pronounced gains on the PPG-BP tasks, while remaining competitive with the best-performing baselines on Epilepsy, WESAD, and our self-collected SEN dataset.

cs.AI↗

arXiv2026

Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendation

Multimodal recommenders combine collaborative behavior with visual and textual item evidence, whose usefulness varies across user-item interactions. Independently trained source-specific diagnostic probes partition held-out interactions into behavior-, appearance-, semantics-, and mixed-evidence regimes across five benchmarks, within which capacity-matched fixed fusion rules exhibit systematic regime-dependent performance crossovers. This diagnostic observation motivates MAGNET, a multimodal graph recommender with two core mechanisms. First, a calibrated expert bank organizes trainable experts by anchor source (behavior, appearance, or semantics) and fusion family (dominant, balanced, or complementary), while an interaction-conditioned router selects a sparse composition using all three evidence sources. Second, an entropy-triggered, coverage-aware progressive schedule decouples population-level routing-mass coverage from per-instance decisiveness, transitioning from broad routing exploration to confident specialization once sufficient coverage is sustained, while preserving coverage thereafter. MAGNET separately encodes the observed interaction graph and a filtered content-induced structural view, applying cross-view alignment after independent propagation. We evaluate MAGNET on four Amazon domains and the non-Amazon MicroLens-100K benchmark, including tail-item and low-history warm-start user evaluation alongside matched expert-design controls. Under the shared-feature, fixed-split protocol, MAGNET-DV exceeds the strongest protocol-compatible non-MAGNET baseline in every reported main-table cell, supported by seed-wise difference tests. Both the fixed and KL-anchored structured variants achieve higher five-seed mean NDCG@20 than matched homogeneous and free-mixture alternatives across all five datasets. MAGNET supports route-level diagnostics through explicit expert semantics.

cs.AI↗

arXiv2026

From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.

cs.AI↗