Can Large Language Models Become Policy Refinement Partners? Evidence from China's Social Security Studies

Abstract

The rapid development of large language models (LLMs) is reshaping operational paradigms across multidisciplinary domains. LLMs' emergent capability to synthesize policy-relevant insights across disciplinary boundaries suggests potential as decision-support tools. However, their actual performance and suitability as policy refinement partners still require verification through rigorous and systematic evaluations. Our study employs the context-embedded generation-adaptation framework to conduct a tripartite comparison among the American GPT-4o, the Chinese DeepSeek-R1 and human researchers, investigating the capability boundaries and performance characteristics of LLMs in generating policy recommendations for China's social security issues. This study demonstrates that while LLMs exhibit distinct advantages in systematic policy design, they face significant limitations in addressing complex social dynamics, balancing stakeholder interests, and controlling fiscal risks within the social security domain. Furthermore, DeepSeek-R1 demonstrates superior performance to GPT-4o across all evaluation dimensions in policy recommendation generation, illustrating the potential of localized training to improve contextual alignment. These findings suggest that regionally-adapted LLMs can function as supplementary tools for generating diverse policy alternatives informed by domain-specific social insights. Nevertheless, the formulation of policy refinement requires integration with human researchers' expertise, which remains critical for interpreting institutional frameworks, cultural norms, and value systems.