The initial step in developing the PRE tool required an evaluation of several pre-existing WRA screening tools [2, 15–17, 27, 33–34] to determine the most appropriate and highly predictive questions contributing to model accuracy for ornamental plants. We evaluated the existing WRA tools and selected questions that were found in two or more of these tools. Using these criteria we identified 56 questions to evaluate that ranged in topic from invasive history, climate match, difficulty of control, environmental impacts, reproductive and dispersal strategies, and growth rate (S1 Table). Each question was given the same weighted score used in the WRA tool it was taken from, with most questions having a score of 1 for a yes answer and 0 for a no answer.

We screened a small population of plant species (35 plants—21 known invasive and 14 known non-invasive plants) to do our initial evaluation of the 56 questions we selected from existing WRA models. While this sample size is relatively small, it served as a preliminary evaluation to test our criteria for including questions in the PRE tool and to determine if the questions were answerable for ornamental plants, which in many cases they were not. The invasive plants were selected from the California Invasive Plant Council’s (Cal-IPC) California Invasive Plant Inventory [6], a list of species that have known or suspected impacts in California and which represent diversity in both taxonomic relationship and phenology (Table 1). The Cal-IPC Inventory is based on an assessment that focuses on impacts and invasiveness of species already within the state, while WRA models typically consider species that are present but not yet invasive, or have yet to be introduced to a region. Thus, the behavior of the plant is more related to other regions of the country or world. The non-invasive species were chosen from the PlantRight’s Suggested Alternatives for Invasive Garden Plants [35] and the Cal-IPC Don’t Plant a Pest Alternatives List [36]. These lists of non-invasive plants are considered appropriate for California and are similar in form and function to known invasive plants.

For each plant species evaluated, a complete literature review was performed, along with searches of online databases and species fact sheets. The information gathered in the literature and internet review was used to answer yes or no for each question in the model. If no information was available to answer a specific question, that question was answered with a “?”, meaning “unknown”.

For each plant species evaluated, we calculated the total score (the sum of points from all the questions) and the percentage of questions that were answered (Table 1). To determine which questions contributed to the predictability of invasiveness and non-invasiveness, we used a two-tailed Fischer’s Exact Test (S1 Table). For each question, we calculated the percentage of times it was answered for all known invasive and non-invasive plants. We determined which questions to eliminate based on the following criteria: 1) Fischer’s Exact Test (two-tailed) P<0.05, 2) question were answered <20% of the time, and 3) question was irrelevant or biased for ornamental plants. Through this process, we developed the PRE tool by narrowing the 56 questions down to 19 (Table 2).

In a more robust evaluation, we tested the 19 question PRE tool by screening 92 additional plants (56 known invasive and 36 known non-invasive plants). Nearly all the known invasive plants were chosen from Cal-IPC California Invasive Plant Inventory [6] or are newly invasive in California (e.g., Buddleja davidii and Billardiera heterophylla) and represent a wide range of taxonomic families and growth forms (Table 3). The non-invasive species were selected from the UC Davis Arboretum All-Star Plant List [37] and PlantRight’s Suggested Alternatives for Invasive Garden Plants [35]. These plants also represent a wide range of growth forms and taxonomic groups (Table 3). The non-invasive species in both lists have many outstanding qualities for use in California gardens, including ease of propagation, few pest or disease problems, and high water use efficiency.

Typically, the testing of a risk assessment tool for accuracy includes post-hoc testing of a list of non-native species that are already known to be invasive versus non-native species that have been introduced for a substantial period of time into the area of concern, but are not considered invasive. We selected non-invasive species that had been in cultivation in California for at least 30 years with no record of escape and invasiveness. An accuracy rate can be determined by the number of correct predictions for the known invasive and non-invasive species. Overall accuracy is determined based on the equation: A₀ = (TP + TN) / (IT + NT); where TP represents the true positives or the number of invasive species correctly rejected as high risk for invasiveness and TN represents the true negatives or the number of non-invasive species correctly accepted as low risk for invasiveness; IT is the total number of invasive species tested and NT is the total number of non-invasive species tested [38]. Invasive species accuracy or sensitivity is determined by using the equation A_i = TP / (TP + FN) where FN is the total number of false negatives or invasive plants that were correctly accepted as having low risk of invasiveness [31, 33, 38]. Finally, non-invasive species accuracy or specificity is A_n = TN / (TN + FP) where FP is the total number of false positives or non-invasive plants that were rejected for high risk for invasiveness. We analyzed the data in two ways, including 1) omitting plant species that were categorized as “needs further evaluation” (considering them to be neither a FP for invasive species nor a FN for non-invasive species), and 2) including “needs further evaluation” species as FP for invasive species or FN for non-invasive species.

The final score for each species was the sum for all of the scores for the all of the questions answered. Based on the separation in scores between the known invasive and non-invasive species, the scoring scale for the 19 question PRE tool was established to be <11 as an accept (low invasive risk), 11 to 13 was classified as “needs further evaluation”, and >13 as a reject (not allowed introduction due to its high invasive risk). Plants which fell into the “needs further evaluation” category may need additional assessment by an expert panel.

Likelihood ratio (LR) was used to compare the predictive potential of the questions we evaluated [31]. The LR assesses the proportion of rejected invaders to the proportion of rejected non-invaders using the equation LR = (TP/IT) / (FP/NT). LR is independent of the base-rate of invasives and non-invasives in the environment. Higher values give better predictive ability and a value of 1 would provide no predictive capability.

Receiver operating characteristic (ROC) curve analysis is commonly used to evaluate the performance of WRA tools [15, 33]. A ROC curve represents test sensitivity (accuracy for correctly categorizing non-invasive plants as having low risk of invasiveness) against the complement of specificity (accuracy for correctly categorizing invasive plants as having high risk of invasiveness) over the range of potential cut-off levels [39]. The area under the ROC curve (AUROC) is used to quantify the performance of a WRA test in discriminating between two classes (high risk versus low risk of invasiveness) [15]. The closer the AUROC is to 1, the higher the accuracy of the screening test, while an AUROC of 0.5 indicates the tool has low accuracy for correctly categorizing plant species as either high or low risk of invasiveness [40]. AUROC is calculated as the sum of trapeziums, or the difference between the x-values multiplied by half the sum of the y-values.

In our initial evaluation of the 56 questions from existing WRA models, the cumulative scores (for all 56 questions) ranged from 21 to 44 for known invasive plants, with an average score of 31 (Table 1). The scores for known non-invasive plants ranged from 5 to 14, with an average score of 10. For each plant species screened, the percentage of questions answered for known invasive plants ranged from to 80% to 98%, with an average of 90%. The percentage of questions answered for known non-invasive plants ranged from to 86% to 95%, with an average of 89%.

The Fischer’s Exact Test identified a total of 31 questions that had a P>0.05 (S1 Table). The percentage of times each of the 56 questions was answered for known invasive plants ranged from 5% to 100%. The percentage of times each of the 56 questions was answered for known non-invasive plants ranged from to 0% to 100%. Eleven of the 56 questions were answerable less than 20% of the time both known invasive and non-invasive plants, so did not meet the question selection criteria. Of the 56 questions evaluated, 17 were eliminated because they did not provide statistically significant predictive power to separate known invasive from known non-invasive plants (S1 Table). Three questions were eliminated because of the inherent bias in the question. For example, the question was only known and answered when the phenomenon was studied, which was nearly always with known invasive species (i.e., allelopathy, palatability to animals, impacts on agricultural or grazing systems). Eight other questions showed statistical significance but were irrelevant to evaluating ornamental plants or new plant introductions. For example, “Has this species been exposed to a high level of domestication?” would not apply to new introductions and is primarily a question related to agricultural species. Species such as corn, which is highly domesticated, is known to have very low invasive potential [41]. Other irrelevant questions, such as “Does it easily establish within horticultural situations?” and “Are propagules dispersed by horticultural practices?” will always be answered yes for horticultural and ornamental species, as such species are expected to establish easily and are dispersed by humans through horticultural practices. Furthermore, the purpose of this model is not to evaluate the potential for significant impact or to prioritize species or populations for management or eradication, but rather to evaluate the potential to escape and establish outside of cultivation. Questions including, “Is this species found only in disturbed areas or is it also found in mature native vegetation as well?” and “Does this species negatively impact water relations on the site?” are included in other models designed to evaluation potential impacts [42, 43], but are not appropriate for the purpose of this model. Finally, there were three questions on the management of species that were eliminated from our final model. These included “Does it reshoot after all control methods?”, “Are effective herbicide treatment available?”, and “Does it need re-occurring control work several times per year?” Such questions do not provide information on the invasive potential of newly introduced ornamental plants, nor is there generally adequate information to answer these questions for ornamentals. These questions are more relevant to prioritization models that allow land managers a decision-making process for allocation of limited resources to manage invasive plant infestations [44].

Of the 19 questions we included in the final PRE tool, 11 showed statistical significance in separating known invasive from non-invasive species, four were the result of merging two similar questions, where both were significant or near significant (e.g., different methods of vegetative reproduction, various biotic and abiotic propagule dispersal mechanisms), one was the result of merging four similar questions (human-mediated propagule dispersal) that were significant or nearly significant (S1 Table). The remaining three questions included in the final PRE model included a new question on whether the plant produces viable seed. While there were other questions on the length of seed viability or whether seeds require pretreatment in order to germinate, these questions were often difficult to answer with ornamental species. Thus, we felt that a simpler form of the questions would be more useful and add additional power to the prediction of invasiveness and non-invasiveness. We retained one of the climate-related questions (Is the species found predominantly in a climate matching those within the targeted region?) because it is important for evaluating new plant introductions into a new geographic area. This question provided no separation in our analysis because we sampled from a population of known invasive and non-invasive species from within the geographic area of evaluation (i.e., California). We retained one question that had a P>0.05, which asks about the capability for fragments of the plant’s stems or roots to establish as new plants. We decided to retain this question to see whether it met our question selection criteria after further testing for the following reasons: 1) it represents an important aspect of vegetative reproduction for invasive plants; 2) we were already collecting information regarding vegetative reproduction for other questions; 3) it was answerable 100% of the time in our first round of testing; and 4) it had a relatively low P-value (P = 0.2816). The results of the additional screenings are presented in the next section.

In our evaluation of the 19 WRA questions, we were able to answer each question at a high frequency (similar to the same questions in the 56 question evaluation), ranging from a low of 54% for non-invasive plants regarding the question on the length of the juvenile period (question 15) to 100% for most other questions (Table 2). We were able to answer an average of 97% of the questions for both invasive and non-invasive plants (Table 3). For individual species, this ranged from 85 to 100% of the questions answered.

The results of the two-tailed Fischer’s Exact Test showed that 16 of the 19 questions were statistically significant in differentiating between the known invasive and non-invasive species. Furthermore, among those 16, the new question on viable seed (question 12) was highly significant (P<0.0001). In addition, the question on plant fragments capable of producing new plants (question 11), which was not significant at P<0.05 in the 56 question analysis, proved to be highly significant when evaluated with the final PRE tool on the 94 species (P = 0.0002). Of the remaining three questions, two were significant in the 56 question analysis, but were not significant at the P<0.05 level in the 19 question analysis. These included questions on whether seeds were quick to germinate (<2 months after dispersal) (question 14, P = 0.1296) and the length of the flowering period (question 16, P = 0.2320). Despite not being statistically significant at the P≤0.05 level, these questions did appear to contribute some level of separation between known invasive and non-invasive species. Finally, as was the case with the 56 question evaluation, the question asking if the species was found predominantly in a climate that matches those of the region of introduction (question 5) could not be evaluated with the 94 species because all the species assessed were already within the geographic area of evaluation (i.e., California).

While the scores for 16 of the 19 questions gave one point for a yes answer and zero points for a no answer, three questions were weighted more heavily (Table 2). In the climate matching questions (question 5) we used two points for a yes answer, which is consistent for other WRA models with the same question [15, 17]. However, for questions 1 to 3, we increased the weighting from one point for naturalization where it is not native (question 1), to two points for invasive elsewhere in the US or world (question 2), to three points for invasive elsewhere in the US or world, but in a similar climate (question 3). We considered that the risk of invasiveness in the region of interest was greater when a species had already become invasive in a different region of the world that had a similar climate to the region of interest.

The scores for known invasive plants ranged from 12 to 21, while the scores for known non-invasive plants ranged from 2 to 13 (Table 3). By plotting score frequencies we were able to assign scoring cutoffs for the PRE tool (Fig. 1). From this plot we determined that scores below 11 were accepted as having a low risk of invasiveness, scores above 13 were rejected as having a high risk of invasiveness, and scores between 11 and 13 would be in the “needs further evaluation” category.

For the 57 known invasive plants evaluated through the 19 question PRE tool, no species were classified as accept (Table 3). Four species, however, were included in the “needs further evaluation” category, including Carpobrotus chilensis, Elaeagnus angustifolia, Gleditsia triacanthos, and Retama monosperma. All but Gleditsia triacanthos are listed by Cal-IPC as Moderately Invasive [6]. Gleditsia triacanthos is not listed by Cal-IPC as invasive, but is included in the Weeds of California and Other Western States as having escaped cultivation in California [45]. Thus, when species within the “needs further evaluation” category were excluded, the accuracy and sensitivity of the PRE tool in prediction invasiveness was 100%. Even when these four species were considered false positives (invasive species incorrectly accepted as non-invasive) the accuracy and sensitivity was 93%.

The 19 question PRE tool gave no false negatives (non-invasive species rejected as invasive), but did classify one species (Koelreuteria paniculata) in the “needs further evaluation” category (Table 3). Koelreuteria paniculata has been reported to have escaped cultivation and become invasive in several US states, particularly Texas [46] and Florida [47], though it has not been reported as an invasive or naturalized plant in California. Thus, the percent accuracy or specificity of the model when plants classified as “needs further evaluation” are excluded is 100%. Even when the “needs further evaluation” species are considered as false negatives, the accuracy is still a very high 97%. When considering both known invasive and non-invasive species, the overall accuracy of the PRE model was 100% when “needs further evaluation “species were excluded and 95% when they were included.

Smith et al. [31] noted that the likelihood ratio (LR) can be useful in assessing the proportion of rejected invader to the proportion of rejected non-invaders and is independent of the base-rate of invasives and non-invasives in the environment. In this case, values close to one provide no predictive power in categorizing potential invasive from non-invasive species. The LR value for the PRE tool demonstrated a very high value even when including “needs further evaluation” species (5.84). When not including “needs further evaluation” the LR value was infinity as the denominator was zero.

For the ROC curve analysis, a positive result was considered anything above the threshold for “needs further evaluation” (>10) and a negative result was an “accept” (<11) outcome. As further support for the accuracy of the PRE tool, the AUROC evaluation had a score of 1, indicating it has high accuracy for correctly categorizing plant species as either high or low risk of invasiveness (Fig. 2).

We believe that the increase in accuracy with the PRE model compared to other similar models may be due to both the reduction in subjectivity associated with questions that were changed or removed, as well as the elimination of questions that added no separation power. It is possible that the inclusion of these questions reduced the ability to accurately evaluate non-invasive plants.