^{1}

^{1}

^{1}

^{2}

^{3}

^{4}

The authors have declared that no competing interests exist.

With the volume of manuscripts submitted for publication growing every year, the deficiencies of peer review (e.g. long review times) are becoming more apparent. Editorial strategies, sets of guidelines designed to speed up the process and reduce editors’ workloads, are treated as trade secrets by publishing houses and are not shared publicly. To improve the effectiveness of their strategies, editors in small publishing groups are faced with undertaking an iterative trial-and-error approach. We show that Cartesian Genetic Programming, a nature-inspired evolutionary algorithm, can dramatically improve editorial strategies. The artificially evolved strategy reduced the duration of the peer review process by 30%, without increasing the pool of reviewers (in comparison to a typical human-developed strategy). Evolutionary computation has typically been used in technological processes or biological ecosystems. Our results demonstrate that genetic programs can improve real-world social systems that are usually much harder to understand and control than physical systems.

Peer reviewed publications remain the cornerstone of the scientific world [

One of the biggest obstacles to research on peer review is the scarce availability of data. Although it is relatively simple to obtain information about citations or publication dates, the inner workings of the editorial process are usually kept secret. However, thanks to the cooperation with our partners at PEERE, we were able to acquire a very detailed dataset from one of the sub-editors of the Journal of the Serbian Chemical Society (JSCS). The dataset contained information about 58 articles submitted to the journal for publication and allowed us to study the review process from the viewpoint of a journal editor. We were able to separate the process into distinct phases (

a: Transition probabilities, showing the percent of review threads that passed through the edge. b: An example of a review thread, showing a path through the decision diagram, coupled with the duration of each phase.

Specifically, the main focus of the present study was how to improve the workflows of editors and decrease the review time. (i.e. the average number of days needed to acquire the required number of reviews for a submitted manuscript). In the simulations, we assumed that two reviews were required, as is the case with the JSCS.

Editorial workflows usually comprise many time-consuming tasks. First, editors need to find appropriate reviewers for each manuscript. Although this can be done with the help of various scientific databases, it is still a very laborious process. Second, they send invitations, handle the communication with reviewers and evaluate the received reports. Finally, they have to deal with situations where they do not receive enough reviews from the invited reviewers or the reviewers give conflicting recommendations.

Before we delve into a more in-depth explanation of how editorial workflows can be optimised, we would like to take a closer look at the artificial review threads. A realisation of the review process is initiated when the editor issues an invitation to a potential reviewer. Thus, to mirror the behaviour of real-life realisations, artificial review threads begin their life in the INVITATION phase (

To make the simulations as realistic as possible, all the review threads strictly followed the decision diagram presented in

Informal discussions with editors revealed two editorial strategies, both of which are visualised in

a: Strategy of Editor A, where the editor waits for all review threads to end before starting new ones. b: Strategy of Editor B, where the number of active review threads is always constant, and a new invitation is sent immediately after one of the active threads ends.

In determining which strategy is better, one should consider that two opposite forces affect the efficiency of editorial strategies. On the one hand, both authors of articles and editors would like to receive reviews as soon as possible. On the other hand, the editors prefer to minimise the number of invitations sent to reviewers. Studies show that authors become impatient if the review process takes too long and may withdraw the manuscript [

Taking the aforementioned factors into account, the strategies can be compared using the simple diagram presented in

Strategies were evaluated by simulating the review process of a large number of articles (using the same procedure that was employed in the fitness function) and while the process itself is stochastic, deviations from these efficiency curves are negligible. (a) Review time as a function of the effective number of reviewers (the average number of reviewers needed to achieve a given review time). Each point on the plot corresponds to a single batch size—the first point on each curve represents the batch of two reviewers and for subsequent points the batch size increases by 1. (b) Comparison of the review time of various strategies for eight effective reviewers. (c) Review time as a function of the batch size. (d) The effective number of reviewers as a function of the batch size.

The strategies of Editor A and Editor B are simple and intuitive. Many new editors would likely adopt one of these strategies as the first go-to solution in the peer review process. However, editors, especially in smaller journals, usually organise their workflows on their own without any point of reference. The development of an automated tool could provide more efficient strategies, thereby making the editor’s job much easier, shortening review times and increasing the satisfaction of authors of manuscripts with the review process.

Cartesian Genetic Programming (CGP; implementations in various programming languages are readily available at

The editorial strategies in CGP can be encoded as a grid of nodes (

BATCH SIZE, the size of the batch; ACTIVE REVIEWERS, the number of active review threads (reviewers for whom the review process has not ended yet); NEEDED REVIEWS, the number of reviews required per article; RECEIVED REVIEWS, the number of reviews received thus far; SUB, subtraction; DIV, division; MUL, multiplication; ADD, addition; MOD, modulo.

Editorial strategies were evolved using a very simple algorithm, called the ‘4 + 1 evolutionary algorithm:’ [

At the beginning of the simulation five strategies are created by randomly assigning functions and links (connections) to nodes. Each of these random strategies is evaluated using a fitness function and the best strategy is chosen as the parent – the seed of the simulation.

At each step of the simulation the current parent is copied four times and the mutation operator is applied to these copies. The mutation operator traverses all inputs of all nodes in a strategy and changes them randomly with a given probability (called the mutation probability). It also traverses all function realised by nodes and changes them randomly with the same probability. It means that both the function and inputs can be changed by the mutation operator at the same time. Then, each of the mutated copies is evaluated using the fitness function. If one of the copies is at least as good (fit) as the parent, then it becomes the new parent. Otherwise, if all copies are worse strategies than the parent, then they are discarded and the parent carries over into the next iteration. We tested mutation probability values in the range _{mut} ∈ [0.01, 0.10]. Since we were able to reach optimised solutions for all probabilities in this range, we decided to use _{mut} = 0.03 in all simulations.

It is worth mentioning at this point that while a graph representing an editorial strategy may be comprised of a large number of nodes, usually only a small fraction of these nodes (<10%) are actually used to calculate the output value. That is, most of the nodes in the graph are not parts of paths that connect inputs with the output. We call such nodes

Step 2 is repeated until a strategy characterised by a sufficiently low fitness function value is found (when compared to real strategies developed by human editors). There is no natural stopping condition for this algorithm – in theory it may try to optimise the strategy indefinitely.

The fitness function used to evaluate strategies is one of the most important components of the optimisation process. The numerical value it assigns to each program is a measure of effectiveness and can be used to compare two strategies. Our approach is based entirely on real-world data. With the help of artificial review threads, the fitness function we designed performs numerical simulations of the review process for batch sizes between two and twenty. The effective number of reviewers and the review time are calculated by averaging the results of simulations of the review process of ten thousand articles per one batch size. The pseudocode for the fitness function and the function responsible for the simulation of the review process can be found in the Supporting Information (

During simulations we assume that time is discrete and that one time step corresponds to one day. However, we also assume editors send new invitations (and consult editorial strategies) only when one of the running review threads finishes its execution (that is, when it enters the NO RESPONSE, REPORT or REJECTION phase). A valid strategy must obey certain constraint. First, it is assumed that at the beginning of the review process (i.e. the zeroth day) invitations are sent to the entire batch of reviewers. Second, the batch size must never be exceeded. Third, at least one invitation must be sent when there are no active (running) review threads. If at least one of these constraints is not met by the strategy being evaluated (the first condition is imposed automatically during simulations), then the fitness function returns a very large value (proportional to the number of batch sizes for which the strategy is not valid), unobtainable by a proper strategy. Otherwise, when the aforementioned conditions are fulfilled, the fitness function calculates the area under the strategy efficiency plot (

Although the simulations performed to evaluate the values of the fitness function were based on ‘full’ review threads (i.e. with full information about phases of the process that constitute each thread), the results of these simulations can be recreated using data presented in the Supporting Information –

With the help of CGP equipped with the aforementioned fitness function, we searched for evolved strategies that would be better than strategies conceived by humans. The evolved strategy can be visualised in a couple of ways. The most natural representation is presented in

Although the program in

Since editorial strategies are essentially deterministic functions that assign integer numbers to states of the system (to each possible input), one can recreate the mathematical formula that corresponds to a strategy by following nodes from output to inputs and writing down the nested functions encountered in nodes. It is important to keep in mind that all functions realised by nodes are protected and normal rules of simplifying mathematical expressions often do not apply. The function equivalent to the evolved strategy can be expressed in the following way:
_{NRT} is the number of new review threads that should be started, _{RQ} is the required number of reviews, _{RR} is the number of received reviews, _{ART} is the number of active review threads, and _{B} is the batch size.

The evolved strategy performed well when compared to that of Editor B and Editor A (

It should be noted that our method of optimisation has no bearing on the quality of peer review. Editorial strategies determine when and how many invitations must be sent to reviewers but they do not interfere with the actual process of reviewing the manuscript by said reviewers. It means that it is still up to the editor to uphold the standard of the journal and the quality of the process.

We would like to end this article by mentioning one more discovery. Editorial workflows are incredibly important, but what can scientists do to speed up the review process? Surely, not everything is in the hands of editors. As it turns out, scientists can do quite a lot.

(PDF)

(PDF)

(PDF)

(PDF)

This publication is supported by the COST Action TD1306 “New frontiers of peer review”.