While preprint feedback is beneficial for the authors, reviewers, readers and other stakeholders, public commenting on preprints has so far remained relatively low. Cultural barriers likely influence participation in public preprint feedback. Authors fear that competitors will leave unfair criticism, or that even fair criticism will bias journal editors and evaluators: while nearly every paper will be thoroughly criticized during journal peer review, the rarity of this feedback being out in the open might lead some to believe that the paper receiving it is especially problematic. Potential reviewers, especially those who rely on more senior colleagues for career advancement, are concerned about retribution for public criticism, or simply harming their reputation by leaving uninformed feedback.

In order to overcome these concerns, we convened a Working Group to discuss how to alleviate the social friction associated with public feedback by developing a set of behavioral norms to guide constructive participation in preprint review. The Working Group brought together relevant stakeholders (researchers, editors, preprint review platform representatives, funders) to discuss the challenges around participation in preprint review and explore what cultural norms could enable and foster further participation in public commentary and feedback. 

FAST principles

The Working Group has developed a set of 14 principles for creating, responding to, and interpreting preprint feedback, clustered around four broad themes: Focused, Appropriate, Specific, Transparent (FAST). The principles are outlined below, and a full description of the FAST principles is available in the preprint ‘FAST principles for preprint feedback.’

FAST principles

The principles apply to the different actors in a preprint feedback environment, including:

Author – Any of those listed in the authorship line of the preprint.

Reviewer – Anyone other than the author(s) who provides comments/feedback on the content and/or claims of the preprint; the feedback may relate to the full paper or to specific parts of the work.

Community – Anyone who engages with the preprint and the feedback on that work. This includes readers but also members of the research community or the public who interact with the authors and/or reviewers, for example, to provide praise, to call out inappropriate behavior or to comment on uses of the preprint and/or preprint feedback by non-specialists.

Examples of statements compliant with FAST principles

Focused

Focus on the content of the preprint
‘This study just reports an observation, there is no mechanistic insight. The authors need to explore and report the mechanism behind the results.’
FAST version‘I realize it is beyond the scope of this study, but as possible follow-up steps, you could explore whether the observation may be driven by mechanism x.’

Focus on the science and not the journal 
‘This study reports interesting results, but as it stands, does not reach the level required for journal x.’
FAST version‘This study reports interesting results, I provide below some suggestions to strengthen the methodology, and to increase the clarity in the interpretation of the results.’

Focus on the scientific and not the personal 
‘The authors show sloppy methodology and a poor knowledge of the state-of-the-art in this field.’
FAST version‘I have some concerns about the methodological approach in the study; in order to ensure that the conclusions are supported and possible artefacts are ruled out, the study should incorporate controls x and y’.

Appropriate

Polish the tone

‘I don’t get why the authors are not reporting the images behind graph x, and the statistical tests employed in the analyses, what are they hiding?’
FAST version‘In order to strengthen the results and increase the reproducibility of the work, I recommend that the manuscript includes the images associated with graph x and report (in the figure legend or the Methods section) which statistical tests were applied.

Reflect on possible biases

‘As others have noted, this is a poorly designed paper and the authors show a poor command of the English language. Also, I disagree with the authors’ interpretation of the findings.’

FAST versionI recognize that English may not be the authors’ first language, to increase the clarity in the presentation, I suggest polishing the written text slightly, particularly in sections x and y. This will help ensure the contributions of the work are clearly understood by readers. I have expertise in this area and previously published work that reported findings that do not align to the interpretation reported in the paper. From that perspective, I offer comments in relation to the results and interpretation reported here’

Engage in scientific discourse

‘We did not solicit this feedback on our preprint and will not respond to the comments.’

FAST versionWe thank the commenter(s) for reading our work and for provisioning comments. We provide a brief response here on some of the most salient points raised, including where we view some of the comments as falling beyond the scope of our paper.’

Behave responsibly and with integrity

‘That comment on the preprint appears over the top and aggressive, but I do not want to get involved’

FAST version ‘I have read the preprint and the feedback by commenter A. While I agree there may be some concerns in relation to the study methodology x, it is important for our feedback to remain polite and constructive, and I encourage commenter A to rephrase the comments.’

Specific

Make it useful
‘The study design is flawed, so it is unclear what to conclude from this work.’
FAST version‘I have some questions about the study design, in particular, a concern about the lack of a control group involving population without disease x. Without this control group, it does not appear possible to conclude whether the outcome reported is due to the intervention, or to other confounding factors.’

Indicate if critical or optional
‘Here are some comments about the paper, in no particular order.’
FAST version‘We outline some questions and comments about the study below, summarising which ones are major items related to study design and methodology, followed by minor items aimed at aiding the clarity in reporting and presentation.’

Assess whether claims are supported by the data
‘I do not trust the conclusions in the paper.’

FAST version‘There is a question as to whether the results, as currently reported, fully support conclusion y. The study could incorporate experiment x to adequately support that conclusion, or the text could be revised to ensure the conclusions tightly align to the findings currently included’.

Be candid
‘I note that the manuscript does not report experiment x to support conclusion y, but this is a very interesting study with novel conclusions.’

FAST versionI have a major concern about one aspect of the study. I think an important additional experiment x needs to be provided to support conclusion y; I realize this may require non-trivial additional work, but it appears important to provide this data in order to ensure that no questions arise about the validity of the conclusions’.

Transparent

Sign (or not) your feedback

‘Here are some anonymous comments on the paper.’
FAST version‘This review is posted on behalf of a colleague who evaluated the preprint, they have expertise in image data analysis and computational methods.’

Acknowledge oversights

‘We did not solicit feedback on our preprint, so we will not address this comment.’

FAST version‘We thank the commenter for pointing out this omission in our manuscript, which was due to an oversight. We have now updated the preprint to include the relevant information.’

Credit contributors

‘Here are some comments on the paper.’

FAST versionThis review was developed as part of the discussions at our journal club; the comments reflect contributions by all members of the journal club.’

Resources

The Working Group reviewed a number of existing resources and materials around journal peer review, preprint review, and public commenting on scholarly work to inform the development of the FAST principles. We provide a summary of the resources reviewed below, and welcome suggestions for additional relevant resources; if you would like to contribute a resource, please submit it via this form.

Preprint commentary & review

Journal peer review

Public commentary and critiques on online platforms and social media

Working Group participants

  • Sharon Ahmad (The Company of Biologists)
  • Tim Behrens (University of Oxford)
  • Gautam Dey (EMBL)
  • Sandra Franco Iborra (New York Genome Center)
  • Maryrose Franko (Health Research Alliance)
  • Samantha Hindle (PREreview)
  • Shriyaa Mittal (Harvard Medical School/MGH)
  • Sara Monaco (Review Commons / EMBO)
  • Dyche Mullins (UCSF)
  • Jessica Polka (ASAPbio)
  • Iratxe Puebla (ASAPbio)