Challenges of Designing and Implementing Simulation Models of Peer Review

Science relies on peer review. Through this mechanism, manuscripts are selected for publication and grant proposals for funding. However, the processes of peer review do not operate in a vacuum; they reflect the priorities, norms, and practices of the institutions in which they are embedded, such as scientific communities, funding agencies, publishers, and scholarly societies, each with their own perspectives and logics (Bollen et al. 2014; Benner & Sandstrom 2000). Peer review is a multi-level system. At the macro level a funding agency sets its priorities and goals for funding based on national priorities and legal mandates. At the meso level, funding agencies use peer review to select which proposals to fund, but also integrate their own strategic objectives (gender balance, geographical diversity, disciplinary needs for example) into the selection process. At the micro level, individual reviewers and panels bring their own perspectives to bear on the review processes. In particular, the dynamics of meso- and micro-level complexity provides an area of exploration that could benefit from simulation studies for two reasons. Simulation studies help us understand what features of the peer review process emerge from different norms, relationships, attitudes and behaviors of the actors and organizations involved. These methods also allow us to develop and test policy recommendations for the improvement of peer review in these same organizations.

In our own project we started by mapping existing simulation models of peer review and identified knowledge gaps in the literature, then started developing a simulation model to address these gaps. We found that numerous researchers had studied peer review systems by means of formal and computational modeling, such as agent-based models (ABM) (Squazzoni & Takács 2011). We counted 44 papers on simulation models of peer review published since 1969: some were used to compare the efficiencies of alternative peer review systems (e.g. Kovanis et al. 2017); some compared different behavioral strategies of authors, editors or reviewers (e.g. Thurner & Hanel 2011; Squazzoni & Gandelli 2013); some sought the origin of the issues of peer review, such as biases, high costs and inefficiencies (e.g. Righi & Takács 2017).