Welcome to DU!
The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards.
Join the community:
Create a free account
Support DU (and get rid of ads!):
Become a Star Member
Latest Breaking News
General Discussion
The DU Lounge
All Forums
Issue Forums
Culture Forums
Alliance Forums
Region Forums
Support Forums
Help & Search
Algorithm ranks peer reviewers by reputation -- but critics warn of bias
https://www.nature.com/articles/d41586-024-01235-wNATURE INDEX 25 April 2024
Algorithm ranks peer reviewers by reputation — but critics warn of bias
There are questions about whether the tool, which could be used by editors to find and shortlist peer reviewers, would disadvantage inexperienced candidates or those in certain locations.
By Dalmeet Singh Chawla
An algorithm ranks the reputation of peer reviewers on the basis of how many citations the studies they have reviewed attracted.
The tool, outlined in a study published in February1, could help to identify which papers could become high impact during peer review, its creators say. They add that, during peer review, authors should put the most weight on the recommendations and feedback from reviewers of previous papers that have been highly cited.
The study authors extracted citation data from 308,243 papers published by journals of the American Physical Society (APS) between 1990 and 2010 that had accumulated more than 5 citations each. Information about the referees of these papers was not available, so the authors used an algorithm to create imaginary reviewers, which rated papers on the basis of an algorithm that was trained on citation data from the APS data set. Using the review scores that these papers received in real life (a score of 1 being poor and 5 being outstanding), the study authors compared how closely the imaginary reviewers’ scores correlated to the actual scores the papers received.
To rank the imaginary reviewers, the study authors tracked the citations accumulated by the papers published between 1990 and 2000 and checked the review scores they were given. Imaginary reviewers that gave high review scores to papers that went on to attract a high number of citations were given a high ranking.
[...]
Algorithm ranks peer reviewers by reputation — but critics warn of bias
There are questions about whether the tool, which could be used by editors to find and shortlist peer reviewers, would disadvantage inexperienced candidates or those in certain locations.
By Dalmeet Singh Chawla
An algorithm ranks the reputation of peer reviewers on the basis of how many citations the studies they have reviewed attracted.
The tool, outlined in a study published in February1, could help to identify which papers could become high impact during peer review, its creators say. They add that, during peer review, authors should put the most weight on the recommendations and feedback from reviewers of previous papers that have been highly cited.
The study authors extracted citation data from 308,243 papers published by journals of the American Physical Society (APS) between 1990 and 2010 that had accumulated more than 5 citations each. Information about the referees of these papers was not available, so the authors used an algorithm to create imaginary reviewers, which rated papers on the basis of an algorithm that was trained on citation data from the APS data set. Using the review scores that these papers received in real life (a score of 1 being poor and 5 being outstanding), the study authors compared how closely the imaginary reviewers’ scores correlated to the actual scores the papers received.
To rank the imaginary reviewers, the study authors tracked the citations accumulated by the papers published between 1990 and 2000 and checked the review scores they were given. Imaginary reviewers that gave high review scores to papers that went on to attract a high number of citations were given a high ranking.
[...]
