PLOS Clim plos plosclimate PLOS Climate 2767-3200 Public Library of Science San Francisco, CA USA 10.1371/journal.pclm.0000429 PCLM-D-24-00031 Research Article Earth sciencesAtmospheric scienceClimatologyClimate changeGlobal warming Research and analysis methodsResearch designSurvey researchSurveys Biology and life sciencesPsychologyPsychological attitudes Social sciencesPsychologyPsychological attitudes Physical sciencesMathematicsApplied mathematicsAlgorithms Research and analysis methodsSimulation and modelingAlgorithms Biology and life sciencesPsychologyBehaviorConditioned response Social sciencesPsychologyBehaviorConditioned response Social sciences Social sciencesPolitical scienceGovernmentsPolitical parties Research and analysis methodsResearch designSurvey research Can large language models estimate public opinion about global warming? An empirical assessment of algorithmic fidelity and bias Can large language models capture public opinion about global warming? https://orcid.org/0000-0001-9204-0867 Lee Sanguk Conceptualization Formal analysis Investigation Methodology Writing – original draft Writing – review & editing 1 https://orcid.org/0000-0002-2588-7491 Peng Tai-Quan Conceptualization Formal analysis Investigation Resources Supervision Writing – review & editing 2 * Goldberg Matthew H. Writing – review & editing 1 Rosenthal Seth A. Resources Writing – review & editing 1 https://orcid.org/0000-0003-4789-1384 Kotcher John E. Resources Writing – review & editing 3 https://orcid.org/0000-0003-3409-9187 Maibach Edward W. Resources Writing – review & editing 3 Leiserowitz Anthony Resources Writing – review & editing 1 Yale Program on Climate Change Communication, Yale University, New Haven, Connecticut, United States of America Department of Communication, Michigan State University, East Lansing, Michigan, United States of America Center for Climate Change Communication, George Mason University, Fairfax, Virginia, United States of America Hsueh Lily Editor Arizona State University, UNITED STATES OF AMERICA

The authors have declared that no competing interests exist.

 * E-mail: winsonpeng@gmail.com 7 8 2024 2024 3 8 e0000429 22 2 2024 28 6 2024 2024 Lee et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Large language models (LLMs) can be used to estimate human attitudes and behavior, including measures of public opinion, a concept referred to as algorithmic fidelity. This study assesses the algorithmic fidelity and bias of LLMs in estimating public opinion about global warming. LLMs were conditioned on demographics and/or psychological covariates to simulate survey responses. Findings indicate that LLMs can effectively reproduce presidential voting behaviors but not global warming opinions unless the issue relevant covariates are included. When conditioned on both demographic and covariates, GPT-4 demonstrates improved accuracy, ranging from 53% to 91%, in predicting beliefs and attitudes about global warming. Additionally, we find an algorithmic bias that underestimates the global warming opinions of Black Americans. While highlighting the potential of LLMs to aid social science research, these results underscore the importance of conditioning, model selection, survey question format, and bias assessment when employing LLMs for survey simulation.

 The author(s) received no specific funding for this work. Data Availability All data used in this study are publicly available on OSF (https://osf.io/w36gn/). Introduction

Public opinion on global warming influences policy-making decisions [1] and public behavior [2]. Scholars and policymakers typically use representative surveys to measure and understand public opinion [3]. While surveys are essential tools for understanding public opinion about global warming, their resource-intensive nature often limits the depth and inclusivity of analysis, potentially resulting in biases toward majority views and neglecting minority perspectives. Moreover, the scarcity of resources makes it challenging to encompass all practically significant variables in a single survey, limiting our grasp of the connections among diverse factors influencing public opinions on global warming.

Large Language Models (LLMs) like GPT have the potential to complement traditional survey methods by simulating survey responses with fewer resources and augmenting data from underrepresented sub-populations. Moreover, unlike conventional predictive methods, such as regression models that depend solely on numeric data, LLMs excel in integrating semantic information (e.g., semantic information in a survey question). This unique ability is particularly advantageous in generating responses for untested global warming survey questions, thereby potentially complementing public opinion datasets [4]. However, these potentials of LLMs hinge upon their ability to accurately estimate public opinion on global warming. As an initial assessment, this study investigates the extent to which LLMs can accurately emulate and reflect multiple dimensions of public opinion about global warming.

LLMs have demonstrated significant potential for contributing to social science research. A recent development lies in their capacity to accurately replicate the perceptions, viewpoints, and behavior of the general population or specific subgroups, termed algorithmic fidelity [5]. Algorithmic fidelity refers to the extent to which LLMs’ intricate web of connections among ideas, attitudes, and sociocultural contexts accurately reflects those found in various human subgroups [5]. By training on an extensive corpus of human-generated data that includes human perceptions and behaviors, LLMs may possess the capability to simulate diverse facets of public opinion.

Recent studies have yielded promising results. For instance, Argyle and colleagues found strikingly high correlations in voting behaviors during presidential elections between human samples and silicon samples derived from LLMs [5]. Silicon samples refer to samples synthesized by LLMs conditioned to thousands of sociodemographic backstories sourced from real human participants in surveys [5]. Similarly, Hwang and colleagues found that LLMs were able to accurately reflect public opinions on diverse political issues, including gun control, gender perspectives, economic inequality, trust in science, and so forth [6]. However, the majority of these studies have focused on the political domain, particularly presidential elections and support for political issues.

It remains uncertain whether LLMs can accurately represent public beliefs and attitudes about other important social issues, such as global warming. Global warming perceptions differ from political opinions in that, while both are subjective, climate change itself is grounded in scientific fact. Given this distinction, LLMs might perform differently when predicting public perspectives on global warming. LLMs are engineered to prioritize factual correctness through extensive training and alignment processes [7]. This focus on correctness could limit LLMs’ ability to reflect the varied and sometimes erroneous human perspectives on global warming.

Algorithmic fidelity in Large Language Models (LLMs) is influenced by input conditioning and model choice. Accurately reflecting public opinion on global warming requires LLMs to be conditioned with detailed demographics and specific covariates to gain a more accurate perspective of individuals. Research shows that LLMs’ predictive accuracy improves when they incorporate past opinion data, as well as demographics and ideology [6]. Additionally, different LLM versions, like GPT-3.5 and GPT-4, vary in their algorithmic fidelity [7–9]. Building on these studies, we assess algorithmic fidelity under different conditions and models. Specifically, we compare LLMs conditioned solely on demographics with LLMs conditioned on both demographics and issue-related covariates. For the sake of simplicity, we categorize political ideology and party affiliation as demographic variables. Furthermore, we examine the algorithmic fidelity of distinct LLM versions: GPT-3.5 and GPT-4.

Public perceptions of global warming are complex. They include, for example, beliefs about global warming, understanding of its causes, worry about the issue, policy support, behavior, and more. To gain deeper insights into how well LLMs represent diverse psychological aspects of global warming, we assess the evaluation metrics (e.g., accuracy, F1) and distribution of LLM predictions against nationally representative survey responses. A strong performance shown by evaluation metrics indicates robust algorithmic fidelity, leading to closely matching distributions between silicon and survey samples. The study draws on nationally representative climate change survey data collected in 2017 and 2021 as benchmarks for evaluating the algorithmic fidelity of LLMs.

 Methods Survey sampling

Two nationally representative survey datasets were collected in October 2017 (N = 1,304) and September 2021 (N = 1,006). In these surveys, participants were asked to answer multiple questions related to global warming. These surveys were conducted under an exemption granted by the Institutional Review Board (IRB) of Yale University (IRB Protocol ID: 2000031972). Subsets of de-identified datasets were accessed for the purpose of this research on September 18, 2023. For each survey, the researchers obtained a distinct sample from the Ipsos KnowledgePanel, comprising U.S. adults aged 18 and over. This panel, which mirrors the U.S. population, was assembled using probability sampling methods. Panel members were recruited using various techniques, such as random digit dialing and address-based sampling, covering nearly all residential phone numbers and addresses in the U.S. Participants completed the survey forms online. Those without internet access were provided with computers and internet connectivity. Upon joining the Ipsos panel, members are informed that participation in every survey is voluntary, and that all data are collected and shared with clients in an anonymous format. Additionally, at the start of the survey, participants were notified that certain questions will pertain to their political opinions, offering them the option to withdraw if they so choose.

 Silicon sample data collection

To generate silicon sample datasets, we used two versions of GPT (GPT-3.5 vs. GPT-4) and two sets of conditional inputs (demographics only vs. demographics and issue-related covariates). Specifically, silicon samples were generated using GPT-3.5-turbo-16k and GPT-4 through the OpenAI API, setting the temperature at 0.70 based on a prior study [5]. For models conditioned solely on demographics, we fed demographic information, such as race/ethnicity, gender, age, political ideology, political party affiliation, education, and residential state, into the models via prompts. Meanwhile, for the models conditioned on both demographics and covariates, additional covariates such as issue involvement in global warming, interpersonal discussions about the topic, and awareness of the scientific consensus, were included along with demographics. These covariates were selected because they appear commonly in both waves of the survey and served as important covariates in previous studies [10–13].

We utilized an interview format adapted from Argyle et al. [5] for our prompts (prompt examples are available in Tables A and B in S1 Text). At the system level, GPTs were instructed to act as an interviewee, guided by the directive: “You are an interviewee. Based on your previous answers, respond to the last question.” Subsequently, the simulated interview began. To establish a clear timeline of survey, the first prompt was phrased as: “Interviewer: What is the current year and month of this interview? Me: October 2017.” For the 2021 survey, "September 2021" was typed in. After setting the timeline, "Me" responses leading up to the final question were provided using actual survey data. For instance, regarding race/ethnicity, the prompt was framed as: “Interviewer: I am going to read you a list of five race categories. What race do you consider yourself to be? ’White, Non-Hispanic’, ’Black, Non-Hispanic’, ’2+ Races, Non-Hispanic’, ’Hispanic’, or ’Other, Non-Hispanic.’ Me: {race from survey response}.”

The final question is the target, for which GPTs supply an answer. For instance, regarding global warming beliefs with a binary answer option, it was phrased as, “Interviewer: What do you think: Do you think that global warming is happening? Would you say ’Yes’, or ’No’?” For other target questions about global warming, we provided comprehensive answer options that matched the survey. For global warming belief with multiple response options, the target question was phrased as, “Interviewer: What do you think: Do you think that global warming is happening? Would you say ’Yes’, ’No’, ’Don’t know’, or ’Refused’ to answer?” For the causation of global warming, the target question was phrased as, “Interviewer: Assuming global warming is happening, do you think it is ’Caused mostly by natural changes in the environment’, ’Caused mostly by human activities’, ’Caused by both human activities and natural changes’, ’Neither because global warming isn’t happening’, ’Other (Please specify)’, ’Don’t know’, or ’Refused’ to answer?” For global warming worry, the target question was, phrased as “Interviewer: How worried are you about global warming? Would you say you are ’Not at all worried’, ’Not very worried’, ’Somewhat worried’, ’Very worried’, or ’Refused’ to answer?”

Occasionally, GPTs generated answers that did not precisely match the listed options. We manually corrected these hallucinations. For instance, instead of a straightforward ‘Yes,’ GPT might produce, ‘Yes, I believe global warming is happening.’ Such responses were recoded to align with the intended options. Any deviations were easily identifiable and adjusted to fit within the given answer choices.

 Survey measurements Target variables

Global warming belief. To measure belief in global warming, we provided a brief definition of global warming: “Global warming refers to the idea that the world’s average temperature has been increasing over the past 150 years, may be increasing more in the future,” and then asked “Do you believe that global warming is happening?” with three response options: “No,” “Don’t know,” and “Yes.”

Global warming cause. We used a recoded version of the survey question. Originally, the survey asked: “Assuming global warming is happening, do you think it is…” with five answer options: “Caused mostly by human activities,” “Caused mostly by natural changes in the environment,” “None of the above because global warming isn’t happening,” “Other (Please specify),” “Refused.” This measure was then recoded to incorporate open-ended responses, expanding the original five answer choices to seven categories: “Caused mostly by human activities,” “Caused mostly by natural changes in the environment,” “Caused by human activities and natural changes,” “Neither because global warming isn’t happening,” “Don’t know,” “Other (Please specify),” and “Refused.” This recoded version was used in the LLM prompt.

Global warming worry. This was measured with a question asking “How worried are you about global warming?” with four response options: “Not at all worried,” “Not very worried,” “Somewhat worried,” and “Very worried.”

 Demographics

Demographic details such as race, ethnicity, gender, age, education, and residential state were provided by Ipsos, based on answers provided when enrolling panel members. Race and ethnicity used five categories: “White, Non-Hispanic,” “Black, Non-Hispanic,” “Other, Non-Hispanic,” “Hispanic,” and “2+ Races, Non-Hispanic.” Gender included two categories: “Male,” and “Female.” Age was segmented into four groups: “18–29,” “30–44,” “45–59,” and “66+.” Education was segmented into four categories: “Less than high school,” “High school,” “Some college,” “Bachelor’s degree or higher.” Residential state includes 50 states and the District of Columbia of the U.S.

Political ideology. This was measured with a question asking “In general, do you think of yourself as…” with six response options: “Very liberal,” “Somewhat liberal,”, “Moderate, middle of the road,” “Somewhat conservative,” “Very conservative.”

Political party. We employed a two-step method to gauge political party. First, participants were asked to identify themselves as “Republican,” “Democrat,” “Independent,” “Other,” or “No party/Not interested in politics.” Those who chose “Independent” or “Other” were then asked a second question: whether they were more aligned with the “Republican party,” “Democratic party,” or “Neither.” If participants initially identified as Republican or Democrat, or if they leaned towards one of these parties in the secondary question, they were categorized accordingly. Those who answered “Independent” in the first question or “Neither” in the second question were categorized into “Independent/Other.” Participants who responded with “No party/Not interested in politics” were categorized into “No party/Not interested.”

 Global warming covariates

Issue involvement in global warming. This was measured with a question asking “How important is the issue of global warming to you personally?” with six response options: “Not at all important,” “Not too important,” “Somewhat important,” “Very important,” “Extremely important,” and “Refused.”

Interpersonal discussion about global warming. This was measured with a question asking “How often do you discuss global warming with your family and friends?” with five response options: “Never”, “Rarely”, “Occasionally”, “Often,” and “Refused.”

Awareness of scientific consensus. This was measured with a question asking “Which comes closest to your own view?” with five response options: “Most scientists think global warming is not happening”, “There is a lot of disagreement among scientists about whether or not global warming is happening”, “Most scientists think global warming is happening,” “Don’t know enough to say,” and “Refused.”

 Pilot test

As a pilot test, we assessed LLMs’ algorithmic fidelity in predicting presidential election voting behaviors to replicate a prior study [5]. We conducted this replication using our own dataset to explore the extent to which algorithmic fidelity is generalizable across different datasets. Our results largely confirm that LLMs, when conditioned on individual demographics, can replicate voting behaviors effectively. The detailed information about the pilot test is presented in S1 Text. Fig A and Table C in S1 Text provides the results of the pilot test.

 Results Algorithmic fidelity of global warming belief: From binary choice to polynomial choice

We investigated whether LLMs demonstrate a high level of algorithmic fidelity for beliefs in global warming. As an initial examination, we limited our sample to respondents who answered either "Yes" or "No" to the question of whether global warming is happening and constrained GPT models to these binary responses. The average accuracy of GPTs across the models, conditions, and years was 85% (SD = 3.41), which suggests that GPTs predict the publics’ belief that global warming is happening with high accuracy.

Accuracy, while an intuitive measure of correct predictions, can be misleading in datasets with skewed distributions, such as our data on belief that global warming is happening. For example, if a majority of survey participants respond with "Yes" and GPTs predict all cases as "Yes," they can still appear highly accurate. To address this issue, we use an additional evaluation metric called a F1 score. A F1 score accounts for both precision (i.e., the proportion of correct positive predictions among all predictions labeled as positive) and recall (i.e., the proportion of actual positive cases that are correctly identified by the model), providing a balanced evaluation of data that is unevenly distributed. We assessed the models using a F1 score and a Macro-Average F1 score (MAF1) which averages F1 scores across answer options.

When conditioned only on demographics, GPT models showed compromised prediction accuracy, with high F1 scores for "Yes" predictions (F1 range: .91-.92) but low or non-existent F1 scores for "No" predictions (F1 range: NA-.08). Interestingly, these models seem to assume a universal belief in global warming, an assumption that does not accurately reflect the diversity of real-world viewpoints in the U.S. To enhance the algorithmic fidelity of LLMs, we introduced additional covariates relevant to global warming, such as issue involvement, interpersonal discussion about global warming, and awareness of the scientific consensus about global warming. When GPT-4 was conditioned on both demographics and these additional covariates, its MAF1 improved from .49 to .82 in the 2017 survey and from unavailable to .85 in 2021. Similarly, under the same conditionings, GPT-3.5’s MAF1 increased from unavailable to .53 in 2017 and to .65 in 2021.

We then evaluated the effects of adding a third response option, "Don’t know," instead of limiting GPTs to a simple "Yes/No" decision, mirroring the approach commonly used in public opinion surveys. The introduction of an additional response option decreased the accuracy of both GPTs. The average accuracy across models, conditions, and years dropped to 75% (SD = 3.70), lower than in binary scenarios. GPTs conditioned only on demographics failed to generate "No" or "Don’t know" responses, resulting in nonexistent F1 scores for these categories. GPTs conditioned on both demographics and covariates showed improved performance. Nevertheless, GPT-4 had more difficulty with "Don’t Know" (2017 F1: .16; 2021 F1: .20) than "No" (2017 F1: .58; 2021 F1: .60). GPT-3.5 performed poorly with both "No" (2017 F1: .24; 2021 F1: .21) and "Don’t Know" (2017 F1: .32; 2021 F1: .34).

Fig 1 displays the response distributions from survey participants and silicon samples regarding their belief that global warming is happening, with two (upper panel) and three (lower panel) response choices. Similar to the binary version, models that were conditioned solely on demographics with three answer choices overestimated the proportion of individuals who believe global warming is happening, compared to the actual survey results. When GPTs were conditioned on both demographics and global warming covariates, response distributions aligned more with the survey data. The evaluation metrics for binary and multiple choice are available in Tables D and E in S1 Text, respectively.

10.1371/journal.pclm.0000429.g001 Belief that global warming is happening: Distributional comparison of survey and silicon samples.

Note: “Demo Only” represents GPTs are conditioned solely on demographics and “Demo + Cov” represents GPTs are conditioned on demographics and covariates.

 Algorithmic fidelity of global warming cause

We then evaluated GPT models’ responses regarding the causes of global warming against survey participant answers. The survey provided the options “Human,” “Nature,” “Both,” “Global warming isn’t happening,” among others (labels simplified here, full details in the Methods section). These options were replicated in our prompts. The average accuracy across GPT models, conditions, and years was 51% (SD = 7.42). MAF1 scores were not available, as models failed to produce F1 scores for some answer options, preventing MAF1 calculation.

Fig 2 compares response distributions from the survey and silicon samples regarding global warming causation. Notably, GPT-4, conditioned only on demographics, greatly overestimates the number of individuals attributing global warming to human activities compared to GPT-3.5 under the same conditions. This was unexpected, considering GPT-4’s general superiority in cognitive tasks. However, adding covariates to demographic conditioning aligns the response distribution more closely with the survey data. The evaluation metrics are available in Table F in S1 Text.

10.1371/journal.pclm.0000429.g002 Global warming cause: Distributional comparison of survey and silicon samples.

Note: “Demo Only” represents GPTs are conditioned solely on demographics and “Demo + Cov” represents GPTs are conditioned on demographics and covariates.

 Algorithmic fidelity of global warming worry: From categorical answers to ordinal assessment

In the final phase, we asked the GPT models about their estimated level of worry about global warming. In the survey, this question was structured as an ordinal variable with four distinct categories: "Very Worried," "Somewhat Worried," "Not Very Worried," and "Not At All Worried." We incorporated this ordinal scale in the prompts provided to the GPT models. The average accuracy across models, conditions, and years was 48% (SD = 13.02). GPT-4, conditioned only on demographics, matched survey data poorly (2017 MAF1 = .22, 2021 MAF1 = .22). GPT-3.5 with the same conditioning failed to produce any responses for "Not very worried" and "Not at all worried," rendering MAF1 unavailable for both years. Adding additional covariates to demographics improved fidelity, with GPT-4 outperforming GPT-3.5 (2017 MAF1 = .65 and .47; 2021 MAF1 = .54 and .50, respectively).

Fig 3 illustrates how the interplay between conditions and model version impacts algorithmic fidelity. Both GPT-4 and GPT-3.5, when conditioned only on demographics, overestimated the number of individuals worried about global warming. Consistent with earlier findings, GPT-4’s estimations were more extreme compared to GPT-3.5, particularly in overrepresenting those "very worried" about global warming. However, GPT-4 conditioned on both demographics and covariates displayed a response distribution more closely aligned with survey data than the similarly conditioned GPT-3.5. The evaluation metrics are available in Table G in S1 Text. Fig B and Table H in S1 Text summarizes the assessment of overall distributions between survey and silicon samples across models and survey items based on Kullback-Leibler Divergence (KLD).

10.1371/journal.pclm.0000429.g003 Global warming worry: Distributional comparison of survey and silicon samples.

Note: “Demo Only” represents GPTs are conditioned solely on demographics and “Demo + Cov” represents GPTs are conditioned on demographics and covariates.

 Algorithmic bias assessment across sub-populations

Here we examine how GPT models represent beliefs that global warming is happening across different sub-populations. The focus was on high-fidelity models in which GPT-4 conditioned on both demographics and covariates for binary beliefs about global warming.

Table 1 details the accuracy and MAF1 results. Interpretations were based on MAF1, where a score below 0.70 is typically considered inadequate. The overarching findings indicate that GPT-4 accurately predicted belief that global warming is happening among diverse sub-populations. However, certain sub-populations were less accurate. GPT-4 was less precise in predicting Non-Hispanic Blacks’ belief that global warming is happening in both 2017 (MAF1 = .62) and 2021 (MAF1 = .60). Further analysis reveals that GPT-4 underestimated Non-Hispanic Blacks believing that global warming is happening. GPT-4 also underrepresented Non-Hispanic Others’ belief in global warming in 2021 (MAF1 = .64), although this requires further investigation due to the limited sample size and heterogeneity of the results within the subgroup. Table I in S1 Text includes the assessment of algorithmic bias for presidential election voting behaviors. Individual F1 scores for each answer option for presidential election and global warming belief (binary choice) across sub-populations are available in Table J in S1 Text. Moreover, Accuracy and MAF1 scores for global warming belief (multiple choice), cause, and worry across sub-populations are available in Table K in S1 Text.

10.1371/journal.pclm.0000429.t001 Accuracy and MAF1 of GPTs for global warming belief across sub-populations.
GW Belief GW Belief
2017 GPT-4 Demo + Cov 2021 GPT-4 Demo + Cov
Variables Acc MAF1 Acc MAF1
Race/ethnicity
    2+ Races, Non-Hispanic .94 .86 1.00 1.00
    2017 n = 32
    2021 n = 18
    Black, Non-Hispanic .90 .62 .92 .60
    2017 n = 104
    2021 n = 73
    Hispanic .90 .77 .92 .82
    2017 n = 132
    2021 n = 101
    Other, Non-Hispanic .96 .91 .89 .64
    2017 n = 51
    2021 n = 38
    White, Non-Hispanic .88 .82 .90 .86
    2017 n = 786
    2021 n = 669
Gender
    Female .89 .80 .92 .85
    2017 n = 568
    2021 n = 454
    Male .89 .84 .90 .85
    2017 EL n = 431
    2021 EL n = 401
    2017 n = 537
    2021 n = 445
Age
    18–29 .91 .85 .89 .72
    2017 n = 145
    2021 n = 113
    30–44 .88 .78 .94 .87
    2017 n = 243
    2021 n = 171
    45–59 .88 .78 .89 .84
    2017 n = 299
    2021 n = 230
    60+ .89 .85 .91 .87
    2017 n = 418
    2021 n = 385
Political ideology
    Moderate, middle of the road .90 .74 .93 .82
    2017 n = 424
    2021 n = 380
    Somewhat conservative .81 .80 .81 .80
    2017 n = 206
    2021 n = 165
    Somewhat liberal .96 .68 .98 .83
    2017 n = 262
    2021 n = 168
    Very conservative .76 .72 .83 .82
    2017 n = 99
    2021 n = 84
    Very liberal .97 .78 .98 -
    2017 n = 104
    2021 n = 88
Political party
    Democrats .96 .62 .97 .74
    2017 n = 535
    2021 n = 423
    Independent/Other .89 .84 .91 .84
    2017 n = 108
    2021 n = 96
    No party/Not interested .81 .73 .84 .78
    2017 n = 105
    2021 n = 50
    Republicans .81 .81 .83 .83
    2017 n = 354
    2021 n = 321
Education
    Bachelor’s degree or higher .92 .84 .93 .86
    2017 n = 440
    2021 n = 358
    High school .86 .81 .88 .83
    2017 n = 272
    2021 n = 226
    Less than high school .88 .74 .98 .96
    2017 n = 64
    2021 n = 42
    Some college .88 .82 .89 .84
    2017 n = 329
    2021 n = 273
Correlational pattern correspondence assessment

GPT-generated responses “reflect underlying patterns of relationships between ideas, demographics, and behavior that would be observed in comparable human-produced data [5].” Accordingly, we investigated how GPT outputs correspond with correlations of demographics and covariates with the target variables in the survey data. We used Cramer’s V to measure association strength. Fig 4 illustrates Cramer’s V values between variables for both surveys and GPT models. GPT-4, conditioned on demographics and covariates, most closely matched survey data in association patterns. This is evidenced by the smallest difference in Cramer’s V between the model and survey across demographic variables (To calculate the average difference in Cramer’s V, we only consider Cramer’s V values associated with demographic variables. Cramer’s V values associated with global warming covariates are excluded from this calculation because they are not consistently available across all models. Including them could potentially lead to an unfair comparison between models.) and years (mean Cramer’s V difference (hereafter, diff) = .04, SD = .05). It was followed by GPT-3.5 with the same conditioning (diff = .09, SD = .12), GPT-3.5 with only demographics (diff = .10, SD = .09), and GPT-4 with only demographics (diff = .11, SD = .11). Not surprisingly, global warming covariates tend to be more strongly correlated than demographics with the target variables. Models accounting for these covariates display better performance that models without them, indicating that integrating relevant background information is essential to achieve a high level of algorithmic fidelity.

10.1371/journal.pclm.0000429.g004 Cramer’s V correlations in survey vs. GPTs.

Note: Cramer’s V could not be estimated for some models because there was no variation in the outcome (e.g., GPT-3.5 with Demo Only for GW Belief).

 Discussion

This research investigates the algorithmic fidelity and bias of Large Language Models (LLMs) by simulating public opinion about global warming and comparing the synthesized data with survey data. Built on prior research investigating algorithmic fidelity in the political realm [5], we extend our analysis to the topic of global warming. The overall findings reveal that LLMs exhibit promising capabilities in predicting public opinion about global warming. Nevertheless, our findings also identify several concerns about employing LLMs in global warming survey research.

Our study finds that incorporating issue-related covariates alongside demographics significantly enhances algorithmic fidelity in global warming research. LLMs conditioned with both are notably better at predicting individual beliefs about global warming than those using only demographic data. This improvement highlights LLMs’ ability to incorporate psychological factors, which are often more indicative of global warming beliefs than demographics alone [11]. However, LLMs that rely solely on demographic information perform poorly even though demographic factors like age, education, and political views are known to be associated with global warming beliefs [11].

The version of the LLM also influences its algorithmic fidelity. GPT-4, conditioned on demographics alone, tends to overestimate the belief that global warming is human-caused and worry about global warming, more so than GPT-3.5. This overestimation is reduced when both demographics and issue-related covariates are used, with GPT-4 then offering more accurate predictions and closer alignment with survey results. This indicates that GPT-4’s advanced training and alignment may both enhance and limit fidelity, especially for science-based subjects like global warming. GPT-4’s design, which includes a thorough alignment process, aims for improved accuracy over GPT 3.5 [7]. With adequate data, such as covariates, GPT-4’s sophisticated integration can potentially produce more accurate estimations of individual opinions than its predecessor. However, in the absence of comprehensive data, GPT-4 may overly rely on scientifically aligned opinions that could diverge from actual views. While this notion is conceivable, it lacks robust empirical support, and ongoing research is essential to understand the factors that influence the differing responses of models developed through distinct processes.

It is also crucial to acknowledge that the lack of transparency from AI developers complicates our ability to understand what influences LLM responses, particularly under uncertainty (e.g., when using only demographics). It raises questions about whether massive training data favor certain attitudes toward climate change, or if the post-training adjustments guided by human feedback lack representativeness. To enhance the reliability of LLMs for social science research, transparency in LLM development is imperative.

Our research indicates a potential algorithmic bias in LLMs regarding certain sub-populations, echoing prior findings [9]. LLMs, particularly those refined with human feedback, often reflect opinions more characteristic of liberal, higher-income, higher-educated, non-religious individuals, and those not adhering to religions such as Buddhism, Islam, or Hinduism [9]. In our study, LLMs perform poorly in predicting beliefs about global warming of non-Hispanic Black Americans. This discrepancy is not explained by sample size, as accuracy remains higher for other racial and multi-racial groups with smaller samples. LLMs often reflect the biases inherent in their training data, which can lead to biased outputs. These biases are particularly pronounced when considering groups that are underrepresented in the data sets. The implications of such biases are not confined to any single nation but are of global concern, as the perspectives of individuals from developing or economically disadvantaged countries may be underrepresented. The root of this problem lies in the limited availability and use of diverse training data, as well as potential biases in the human feedback that guides LLMs’ learning processes. This situation emphasizes the need for a more inclusive approach in gathering data and curating feedback, ensuring a wider range of voices are heard and accurately reflected in the technology’s outcomes. Therefore, a thorough investigation of LLMs’ algorithmic biases, especially against marginalized groups and countries, is essential for ensuring fairness and accuracy across broader domains of social science research.

Our study offers actionable guidance for integrating LLMs into climate change research and other fields. We suggest researchers should condition LLMs with domain-specific variables related to global warming, such as public engagement and scientific agreement, to enhance their relevance and accuracy. Because of the multifaceted nature of global warming perceptions, the inclusion of a broader range of covariates might further sharpen the models’ precision for particular investigative goals. Additionally, our findings indicate that more sophisticated models like GPT-4 show greater fidelity than GPT-3.5 when equipped with potential covariates. Furthermore, reducing the scope of answer choices, particularly vague ones such as “Don’t know,” can potentially improve estimation accuracy. Although restricting answer options reduces LLMs’ applicability in social science research, it still holds significant value in streamlining exploratory data collection. For instance, in our study, we acquired data from 1,304 synthetic samples at a considerably lower cost (i.e., approximately $2.08 using GPT-3.5 and $20.86 using GPT-4 as of September 2023) and time reduction when compared with traditional survey techniques. This underscores LLMs’ cost-efficiency and speed. However, we do not argue that LLMs can replace traditional survey methods and empirical research. Conventional surveys are and will be an essential tool for assessing public opinion, where cost-effectiveness should not overshadow scientific rigor. Hence, we propose employing LLMs as a complementary instrument for preliminary investigations, survey design, outcome forecasting, and hypothesis generation. Meanwhile, established methods should be maintained for empirical studies to ensure the rigor of research findings.

Our research has limitations that suggest avenues for future exploration. First, we focused on closed-ended questions, overlooking the depth offered by open-ended questions. Open-ended responses could provide richer qualitative data, potentially increasing LLM algorithmic fidelity. Future studies should investigate the impact of incorporating open-ended question responses as conditional inputs on LLMs. Second, our research focuses on a narrow aspect concerning the impact of prompt format (e.g., the number of answer options) on algorithmic fidelity, leaving unexplored other facets of prompt structure that may affect fidelity, such as the order of response options [14]. Other elements like a question sequence and the quantity and arrangement of target questions could also influence algorithmic fidelity. Understanding how these survey design elements affect both human and LLM responses needs more attention in future research.

In conclusion, this research provides valuable insights into the algorithmic fidelity and bias of LLMs in simulating public opinions regarding global warming. This study offers practical guidance on conditioning prompts and selecting models to maximize fidelity in social science applications while emphasizing the importance of validating LLMs, particularly for minority groups. A nuanced approach is required to harness the power of LLMs while addressing their limitations through proactive algorithm auditing and bias mitigation.

 Supporting information <p>(DOCX)</p> </caption> </supplementary-material> </sec> </body> <back> <ref-list> <title>References Bromley-Trujillo R, Poe J. The importance of salience: public opinion and state policy action on climate change. J Pub Pol. 2020;40: 280–304. doi: 10.1017/S0143814X18000375 Doherty KL, Webler TN. Social norms and efficacy beliefs drive the Alarmed segment’s public-sphere climate actions. Nature Clim Change. 2016;6: 879–884. doi: 10.1038/nclimate3025 Berinsky AJ. Measuring Public Opinion with Surveys. Annu Rev Polit Sci. 2017;20: 309–329. doi: 10.1146/annurev-polisci-101513-113724 Kim J, Lee B. AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction. arXiv; 2023. Available: http://arxiv.org/abs/2305.09620 Argyle LP, Busby EC, Fulda N, Gubler JR, Rytting C, Wingate D. Out of One, Many: Using Language Models to Simulate Human Samples. Polit Anal. 2023;31: 337–351. doi: 10.1017/pan.2023.2 Hwang E, Majumder BP, Tandon N. Aligning Language Models to User Opinions. arXiv; 2023. Available: http://arxiv.org/abs/2305.14929 Aher G, Arriaga RI, Kalai AT. Using large language models to simulate multiple humans and replicate human subject studies. 2023. OpenAI. GPT-4 Technical Report. arXiv; 2023. Available: http://arxiv.org/abs/2303.08774 Santurkar S, Durmus E, Ladhak F, Lee C, Liang P, Hashimoto T. Whose Opinions Do Language Models Reflect? arXiv; 2023. Available: http://arxiv.org/abs/2303.17548 Goldberg MH, Van Der Linden S, Maibach E, Leiserowitz A. Discussing global warming leads to greater acceptance of climate science. Proc Natl Acad Sci USA. 2019;116: 14804–14805. doi: 10.1073/pnas.1906589116 31285333 Hornsey MJ, Harris EA, Bain PG, Fielding KS. Meta-analyses of the determinants and outcomes of belief in climate change. Nature Clim Change. 2016;6: 622–626. doi: 10.1038/nclimate2943 Reser JP, Bradley GL. The nature, significance, and influence of perceived personal experience of climate change. WIREs Climate Change. 2020;11: e668. doi: 10.1002/wcc.668 van der Linden S, Leiserowitz AA, Feinberg GD, Maibach EW. The scientific consensus on climate change as a gateway belief: Experimental evidence. PLOS ONE. 2015;10: e0118489. doi: 10.1371/journal.pone.0118489 25714347 Pezeshkpour P, Hruschka E. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. arXiv; 2023. Available: http://arxiv.org/abs/2308.11483 10.1371/journal.pclm.0000429.r001 Decision Letter 0 Hsueh Lily Academic Editor 2024 Lily Hsueh This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

16 Apr 2024

PCLM-D-24-00031

Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias

PLOS Climate

Dear Dr. Peng,

Thank you for submitting your manuscript to PLOS Climate. After careful consideration, we feel that it has merit but does not fully meet PLOS Climate’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Overall, the reviewers and I are enthusiastic about the potential contribution of this paper to climate science and social science in explicating the promises of LLMs in public opinion research and analysis. That said, there are opportunities for the authors to address constructive feedback from the reviewers. Please pay close attention to Reviewer #1's recommendation to provide a more thorough discussion of the risks/limitations and benefits, including how LLMs could be complementary (and supplementary) to traditional survey research.

Along the same lines, Reviewer #2 urges the author to provide more discussion on why the models perform poorly in some cases. Reviewer #2 raises an important question about whether alignment processes between GPT-3.5 vs. GPT-4 reduce the ability of the models to replicate specific kinds of questions related to global warming. Moreover, I suggest that the authors take seriously Reviewer #2's constructive feedback on the need to explicate the differences between models that do and do not have attitudinal covariates. Reviewer #2 provide specific advice on how to do this more adequately. 

Please submit your revised manuscript by May 16 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at climate@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pclm/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Lily Hsueh, Ph.D.

Associate Professor, Arizona State University

Academic Editor, PLOS Climate

Journal Requirements:

1. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

2. Please note that PLOS Climate has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/climate/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Please upload all main figures as separate Figure files in .tif or .eps format only. For more information about how to convert and format your figure files please see our guidelines:

https://journals.plos.org/climate/s/figures

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Climate’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Climate does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to express my congratulations on your engaging and important paper concerning the suitability of utilizing large language models (LLMs) for assessing public opinion on climate change. Your study stands out for its meticulous execution and methodological robustness, contributing valuable insights to the field.

However, I would like to encourage the authors to provide a more in-depth critique of the potential risks and rewards inherent in employing LLMs for simulating public opinion. For instance, your findings reveal that GPT-4 exhibits an accuracy, from 53% to 91%, in mirroring beliefs and emotional responses concerning global warming. It is noteworthy that for certain demographic segments, this accuracy diminishes significantly. Despite the anticipation that future advancements in LLMs might enhance accuracy, the intrinsic biases present in LLMs training data and human feedback are likely to perpetuate skewed and unreliable outcomes, particularly for groups that are underrepresented in the training datasets or overlooked during model refinement through human feedback.

The concern extends to the representation of global perspectives, particularly from lower-income and developing nations, which may not be adequately captured due to limited training data and skewed human feedback. This situation echoes the limitations sometimes encountered in high-quality survey research, which, despite rigorous and replicable methodological approaches, may not fully capture public opinion. Considering that even high-quality survey-based methods encounter such difficulties, the accuracy metrics presented in your study underscore the potential for unrealistic conclusions derived from LLM-based simulations of public opinion.

I concur with your conclusion that LLMs hold promise for exploratory hypothesis generation, the design, and pre-testing of survey-and field-based research. Moreover, LLMs could potentially be useful as a supplementary data source or for validating findings in conjunction with traditional data gathering methods like surveys. Nevertheless, I harbor reservations regarding their utility in confirmatory empirical research, given the substantial risk that researchers and practitioners may increasingly gravitate towards cheaper, albeit less reliable, LLMs for simulating public opinion, potentially at the expense of more robust methods.

In conclusion, beyond the technical assessment of LLMs' capability to reflect public opinion, a more thorough exploration of the risks and benefits and the practical utility of LLMs in the realm of public opinion research on climate change is warranted. Such a discussion would greatly enhance the manuscript's contribution to the discourse on the application of LLMs in social science research.

Reviewer #2: I am reviewing the manuscript “Can Large Language Models Capture Public Opinion about Global Warming?” In this manuscript, the authors build on prior research about algorithmic fidelity and the potential for LLMs to accurately reflect political attitudes by extending the analysis to attitudes about climate change and comparing two new cutting-edge models (GPT-3.5 & GPT-4). They find that the LLMs do demonstrate a capacity for accurate reflection of public opinion about climate change, but they also highlight some important contingencies in that capacity. Specifically, the LLMs perform much better when provided with relevant attitudinal information (as opposed to relying only on demographic covariates). Also, GPT-4 performs worse than GPT-3.5 without the attitudinal covariates, and the best with them.

There are a lot of very commendable aspects to this project. The manuscript is detailed and very clearly written. Of course, there are a hundred other design and prompt decisions the authors could have made, but the ones they do make are clear, reasonable, and well-justified in the text. The authors do a very good job of walking the line between being clear about the contribution this manuscript makes (which is meaningful) without overclaiming or making a point that is too general to be supported by the studies they have conducted. The manuscript adds to both the substance (climate change opinions) and the methodological details about LLMs as silicon samples that exists in current published work. The pilot test replication of the voting data is also a very nice addition to verify their method and the models’ baseline performances. I think it is a strong manuscript that merits publication.

I have two relatively small recommendations for the authors as they finish revising. First, I understand that there is a lot that remains opaque in the training process and adjustments made to the OpenAI models. The authors' points on this in the conclusion are well-made. However, I do think the authors could do a little more to discuss why they believe the models perform poorly in some cases. For example, how should we think about the differences between 2017 and 2021, and GPT-3.5 vs. GPT-4? Are there more general principles that might serve as hypotheses for future work? Do alignment processes reduce the ability of the models to replicate specific kinds of questions related to global warming? The authors bring this up as a motivation for this study, but I think it deserves a little more follow through in the discussion of variation across models and prompts.

Second, I think the authors could do a little more to describe the differences between models that do and do not have attitudinal covariates in the prompts. On some level, it is not surprising that the models with only demographics perform relatively poorly as predictors, because – although there are small correlations for some of these items (judging from the human sample baseline on Figure 4), they are not strong predictors of the climate attitudes measured. The exception to this is of course ideology and party, which are not (as the authors note) strictly speaking demographic traits. Because of this, I would like to see Figure 4 expanded to include the attitudinal measures that are also used in the “with covariates” models. Additionally, I think if the authors spend a little more time developing that fact that the models with covariates work better precisely because the added covariates are more strongly correlated with the attitude they are predicting than the demographics were, they can work that into a broader point that the predictions work the best when provided with the most relevant background information. I occasionally encounter folks in this space who leave out the most relevant predictors because they feel like they would be too closely correlated to the outcome variable they are trying to predict. I think the authors here have an opportunity to demonstrate a really import

 10.1371/journal.pclm.0000429.r002 Author response to Decision Letter 0 Submission Version 1

7 May 2024

Submitted filename: 5.Response Letter.docx

 10.1371/journal.pclm.0000429.r003 Decision Letter 1 Hsueh Lily Academic Editor 2024 Lily Hsueh This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

2 Jul 2024

Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias

PCLM-D-24-00031R1

Dear Dr Peng,

We are pleased to inform you that your manuscript 'Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias' has been provisionally accepted for publication in PLOS Climate.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow-up email from a member of our team. 

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact climate@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Climate.

Best regards,

Lily Hsueh, Ph.D.

Associate Professor, Arizona State University

Academic Editor, PLOS Climate

***********************************************************

Reviewer Comments (if any, and for reference):

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Does this manuscript meet PLOS Climate’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Climate does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear Authors,

Thank you very much for addressing all of my comments.

I am looking forward to seeing this paper published.

Reviewer #2: The authors did a nice job addressing all of my concerns, and the ethics concerns of Reviewer 1 (I thought the suggestions and revisions here were also excellent. I think the manuscript is ready for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********