In this series of posts exploring how trust in peer review is established, maintained, and delivered, we began yesterday by discussing the author’s role in establishing trust through honest research and reporting practices.

Today, we will explore the responsibilities of the reviewers to further develop that trust through unbiased and quality review practices that lead to an ultimate goal of publishing quality work that is accepted and trusted by the readers.

A complex commitment

In last week’s Scholarly Kitchen article, Todd Carpenter stated, “as the amount of literature expands, and the analytical approaches to solving problems become more complicated, peer reviewing a paper is growing in complexity and commitment”. This, in itself, is why trust starts with the researcher/author of the work being reviewed – it is unrealistic to expect a reviewer to be fully aware and versed in the increasing body of work in their discipline. 

That said, the reviewer has an obligation to develop themselves to a point of competent review in the field. With an acceptable level of trust in the author’s work, they should be prepared to objectively evaluate the value of the contribution to the field and specific publication venue. New reviewers should take extra precaution to ensure that they are competent to serve in the role as a reviewer.

A checklist approach to peer review

One concern threaded through many of the contributions to last week’s Scholarly Kitchen article was that of transparency and clear identification of roles and responsibilities throughout the peer review process. Charlie Rapple offered two suggestions for improving trust in the reviewers’ ability to fulfill their responsibilities: “a checklist provided to reviewers and published alongside the article, and an indication of the reviewer’s experience / qualifications for reviewing”.

Carpenter supports a level of standardization of process as well, stating, “Clarity of expectations, communication about those expectations, recognition for, and potentially metrics around those contributions would go a long way to improving peer review.”

Developing trust requires delivery of results

It’s the metrics that matter. The reviewers should produce actionable suggestions that improve the publication potential of the submitted manuscript. As noted yesterday, it is not the reviewer’s responsibility to validate the research data presented, but rather the author’s responsibility to present data honestly and accurately.

If this trust in the author’s work can be assumed, however, the reviewer can then focus on helpful and timely suggestions. According to Haseeb Irfanullah, “Editors and authors trust reviewers to make constructive, unbiased suggestions, and, of course, to send the reviewer’s comments on time.”

Addressing biases in the review process

The key to trustworthy reviewer performance is that the review is conducted in an unbiased way on the manuscript, not the author. Two factors of potential bias in the review process were specifically noted in the Scholarly Kitchen article last week: identification of the author(s) by reviewers and open access APCs.

The issue of reviewers knowing or being able to identify the author(s) of the submitted work, noted by Robert Harington, is one of unconscious bias and is a factor in the overall trust of the review process. Harington’s solution is a shift to more prominent use of the double-blind review process. He said, “double-blind peer review introduces a pause that essentially should help reviewers and editors navigate unconscious bias”.

Similarly, bias exists when there is a perceived “pay to play” factor in publishing where acceptance might be incentivized. Tim Vines noted this effect of publication fees in the open access environment as additional concerns for trust in the review process. “On the Open Access side, moving away from APCs (which incentivize acceptance) to submission fees (which incentivize a quality review experience) would also be a vital shift.”

Vines also made a statement that seemed to summarize the role of the reviewer best, “Ultimately, trust in peer review is built when individual researchers have a great review experience (regardless of the outcome); all we need to do is make that the norm.” This, however, may be easier said than done.

---

Eric Schmieder is the Membership Marketing Manager for TAA. He has taught computer technology concepts to curriculum, continuing education, and corporate training students since 2001. A lifelong learner, teacher, and textbook author, Eric seeks to use technology in ways that improve results in his daily processes and in the lives of those he serves. His latest textbook, Web, Database, and Programming: A foundational approach to data-driven application development using HTML, CSS, JavaScript, jQuery, MySQL, and PHP, First Edition, is available now through Sentia Publishing.