arXiv subjects

Dimity Stephen

Publications and source records attributed to Dimity Stephen.

6 recordsLinked to original sources

arXiv2025

Implicit reporting standards in bibliometric research: what can reviewers' comments tell us about reporting completeness?

The recent surge in bibliometric studies published has been accompanied by increasing diversity in the completeness of reporting these studies' details, affecting reliability, reproducibility, and robustness. Our study systematises the reporting of bibliometric research using open peer reviews. We examined 182 peer reviews of 85 bibliometric studies published in library and information science (LIS) journals and conference proceedings, and non-LIS journals. We extracted 968 reviewer comments and inductively classified them into 11 broad thematic categories and 68 sub-categories, determining that reviewers largely focus on the completeness and clarity of reporting data, methods, and results. We subsequently derived 49 recommendations for the details authors should report and compared them with the GLOBAL, PRIBA, and BIBLIO reporting guidelines to identify (dis)similarities in content. Our recommendations addressed 60-80% of the guidelines' items, while the guidelines covered 45-65% of our recommendations. Our recommendations provided greater range and specificity, but did not incorporate the functions of guidelines beyond addressing academic content. We argue that peer reviews provide valuable information for the development of future guidelines. Further, our recommendations can be read as the implicit community standards for reporting bibliometric studies and could be used by authors to aid complete and accurate reporting of their manuscripts.

cs.DL↗

arXiv2024

Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with quality

This study investigates the viability of distinguishing articles in questionable journals (QJs) from those in non-QJs on the basis of quantitative indicators typically associated with quality. Subsequently, I examine what can be deduced about the quality of articles in QJs based on the differences observed. I contrast the length of abstracts and full-texts, prevalence of spelling errors, text readability, number of references and citations, the size and internationality of the author team, the documentation of ethics and informed consent statements, and the presence erroneous decisions based on statistical errors in 1,714 articles from 31 QJs, 1,691 articles from 16 journals indexed in Web of Science (WoS), and 1,900 articles from 45 mid-tier journals, all in the field of psychology. The results suggest that QJ articles do diverge from the disciplinary standards set by peer-reviewed journals in psychology on quantitative indicators of quality that tend to reflect the effect of peer review and editorial processes. However, mid-tier and WoS journals are also affected by potential quality concerns, such as under-reporting of ethics and informed consent processes and the presence of errors in interpreting statistics. Further research is required to develop a comprehensive understanding of the quality of articles in QJs.

cs.DL↗

arXiv2022

Medical articles in questionable journals are less impactful than those in non-questionable journals but still extensively cited

A key feature of questionable journals is a lack of adequate peer review of their articles. Content of thus unknown quality may be utilised by unsuspecting practitioners or incorporated into peer-reviewed research, becoming legitimised. It is therefore necessary to examine the citation patterns of articles in questionable journals to understand the impact and reach of research in questionable journals. Similar research has tended to focus on authors from low- and middle-income countries. As such, this study investigates the profile and impact of research in questionable journals by authors in Germany. Questionable journals were identified by matching journals with articles by authors at German institutions from Dimensions to Cabell's Predatory Reports. Metadata for these articles and a comparative sample of articles in non-questionable journals were extracted from Dimensions and the 3 year citations, self-citations, uncited rate, profile of co-authoring and citing countries, and institution type of authors were compared between groups. Nearly 600 articles in 88 questionable journals were published by German authors in 2010-2020. Three-quarters were in the medical and health sciences. Medical articles in questionable journals received significantly fewer citations than similar articles in non-questionable journals. However, articles in questionable journals were still extensively cited in 1,736 primarily non-questionable journals. Self-citations accounted for only 12% of these citations. Authors from non-university medical facilities were over-represented in articles in questionable journals. System-level changes are necessary to eliminate questionable journals and shift high-quality research into reputable venues.

cs.DL↗

arXiv2021

Peer reviewers equally critique theory, method, and writing, with limited effect on manuscripts' content

Peer review aims to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content to be variable, but generally modest and positive. I qualitatively analysed 1,430 peer reviewers' comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers' comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers' comments. Reviewers' comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

cs.DL↗

arXiv2021

From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers

Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners' indexation philosophies. In light of these differences, we explore here, using a citation network analysis and assessment of normalised citation impact of "duplicate" publications, whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of core and exclusive 2016-2018 publications revealed a large set of core publications indexed in all three databases that are highly self-referential. In comparison, each database selected a set of exclusive publications that appeared to hold similarly low levels of relevance to the core set and to one another, with slightly more internal communication between exclusive publications in Scopus and Dimensions than WoS. Our comparison of normalised citations for 41,848 publications indexed in all three databases found that German sectors were valuated as more impactful in Scopus and Dimensions compared to WoS, particularly for sectors with an applied research focus. We conclude that the databases do present structurally different perspectives, although Scopus and Dimensions with their additional circle of applied research vary more from the more base research-focused WoS than they do from one another.

cs.DL↗

arXiv2021

A study of referencing changes in preprint-publication pairs across multiple fields

Manuscripts have a complex development process with multiple influencing factors. Reconstructing this process is difficult without large-scale, comparable data on different versions of manuscripts. Preprints are increasingly available and may provide access to the earliest manuscript versions. Here, we matched 6,024 preprint-publication pairs across multiple fields and examined changes in their reference lists between the manuscript versions as one aspect of manuscripts' development. We also qualitatively analysed the context of references to investigate the potential reasons for changes. We found that 90 percent of references were unchanged between versions and 8 percent were newly added. We found that manuscripts in the natural and medical sciences undergo more extensive reframing of the literature while changes in engineering mostly focused on methodological details. Our qualitative analysis suggests that peer review increases the methodological soundness of scientific claims, improves the communication of findings, and ensures appropriate credit for previous research.

cs.DL↗