Who Checks the Checkers? Exploring Source Credibility in Twitter's Community Notes

Abstract

In recent years, the proliferation of misinformation on social media platforms has become a significant concern. Initially designed for sharing information and fostering social connections, platforms like Twitter (now rebranded as X) have also unfortunately become conduits for spreading misinformation. To mitigate this, these platforms have implemented various mechanisms, including the recent suggestion to use crowd-sourced non-expert fact-checkers to enhance the scalability and efficiency of content vetting. An example of this is the introduction of Community Notes on Twitter. While previous research has extensively explored various aspects of Twitter tweets, such as information diffusion, sentiment analytics and opinion summarization, there has been a limited focus on the specific feature of Twitter Community Notes, despite its potential role in crowd-sourced fact-checking. Prior research on Twitter Community Notes has involved empirical analysis of the feature's dataset and comparative studies that also include other methods like expert fact-checking. Distinguishing itself from prior works, our study covers a multi-faceted analysis of sources and audience perception within Community Notes. We find that the majority of cited sources are news outlets that are left-leaning and are of high factuality, pointing to a potential bias in the platform's community fact-checking. Left biased and low factuality sources validate tweets more, while Center sources are used more often to refute tweet content. Additionally, source factuality significantly influences public agreement and helpfulness of the notes, highlighting the effectiveness of the Community Notes Ranking algorithm. These findings showcase the impact and biases inherent in community-based fact-checking initiatives.