arXiv · 2611.22006
Supporting Researcher Control in AI-Assisted Social Media Analysis: A Mixed-Methods Study of AI4OSN
Abstract
High-quality annotated datasets are critical for advancing computational social science (CSS) research, yet collecting, curating, and analyzing large-scale data from dynamic sources such as online social networks (OSNs) remains a persistent bottleneck. Researchers must also translate substantive questions into queries, relevance criteria, and defensible interpretations. Large language models (LLMs) can assist, but they can also obscure the judgments, potentially reducing researcher oversight and critical evaluation. We present AI4OSN, a mixed-initiative platform that keeps researchers in control of data collection, relevance labeling, and sensemaking while making AI assistance available as an inspectable advisory resource. In a systematic user study (N=18) using X/Twitter data, participants reported above-benchmark perceived usability (mean SUS score of 79.7), increased confidence in LLM-assisted annotation and analysis tasks, and low cognitive workload during AI-supported sensemaking. Notably, novice participants showed descriptively greater alignment and higher reported agreement with AI output than experienced participants, underscoring the need to support differentiated verification practices across expertise levels. Overall, this work contributes both a human-centered system for social data analysis and empirical insights into how AI-assisted interfaces shape researcher behavior, trust, and decision-making. Our findings highlight design opportunities and tensions in building accessible, transparent, and agentic AI systems for social media and web research.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Qile Wang, Abhishek Karwankar, Shukria Muhammadi, Srikanth Kumar Perumalla, Abhin Suresh, Sunita Chandrasekaran, Kenneth E. Barner, Matthew Louis Mauriello. 2026-11-16. Supporting Researcher Control in AI-Assisted Social Media Analysis: A Mixed-Methods Study of AI4OSN. https://arxiv.science/abs/2611.22006
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.