PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0262575 PONE-D-21-22428 Research Article Medicine and health sciencesOncologyCancers and neoplasmsGynecological tumorsOvarian cancer Research and analysis methodsResearch designSurvey researchSurveys Computer and information sciencesData management Science policyScience and technology workforceCitizen science Research and analysis methodsResearch designSurvey research Medicine and health sciencesEpidemiologyMedical risk factorsCancer risk factors Medicine and health sciencesOncologyCancer risk factors Medicine and health sciencesPublic and occupational healthHealth promotion Science policyScience and technology workforceCareers in researchScientists People and placesPopulation groupingsProfessionsScientists Applying citizen science to engage families affected by ovarian cancer in developing genetic service outreach strategies Applying citizen science for genetic services outreach https://orcid.org/0000-0001-7854-4372 McBride Colleen M. Conceptualization Formal analysis Funding acquisition Methodology Project administration Writing – original draft Writing – review & editing 1 * Campbell Gavin P. Conceptualization Data curation Formal analysis Writing – original draft 1 Zhao Jingsong Conceptualization Formal analysis Supervision 1 Pentz Rebecca D. Investigation Supervision Writing – review & editing 2 Escoffery Cam Conceptualization Methodology Writing – review & editing 1 Komonos Michael Formal analysis Writing – review & editing 3 Cannova Kelly Data curation Formal analysis Writing – review & editing 4 Byrne Janice L. B. Data curation 5 Paris Nancy M. Methodology Supervision Writing – review & editing 6 Shepperd James R. Conceptualization Formal analysis Methodology Writing – review & editing 7 Guan Yue Conceptualization Data curation Formal analysis Funding acquisition Writing – review & editing 1 Department of Behavioral, Social and Health Education Sciences, Emory University, Atlanta, GA, United States of America Department of Hematology and Oncology, Emory School of Medicine, Atlanta, GA, United States of America Emory School of Medicine, Atlanta, GA, United States of America The OVERRUN Ovarian Cancer Foundation, Overland Park, KS, United States of America Department of Obstetrics and Gynecology, University of Utah Health Sciences Center, Salt Lake City, UT, United States of America Georgia Center for Oncology Research & Education, Atlanta, GA, United States of America Department of Psychology, University of Florida, Gainesville, FL, United States of America Corpas Manuel Editor Universidad Internacional de La Rioja, SPAIN

No authors have competing interests.

 * E-mail: cmmcbri@emory.edu 14 2 2022 2022 17 2 e0262575 13 7 2021 30 12 2021 2022 McBride et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Citizen science (CS) approaches involving non-professional researchers (citizens) as research collaborators has been used infrequently in health promotion generally and specifically, in cancer prevention. Standardized CS approaches may be especially useful for developing communication interventions to encourage families to consider cancer genetic services. We engaged survivors of ovarian cancer and their close relatives as CS collaborators to collect and help interpret data to inform content for a website, printed invitation materials, and short-message reminders. We applied an implementation quality framework, and posed four research questions regarding the feasibility of CS: recruitment, data collection, data quality and evaluation of the experience. CS members were recruited through three networks: clinical sites, local and national cancer support organizations, and online ovarian cancer patient support groups. The professional research team operationalized theory-aligned CS tasks, five data collection options, question banks/scripts for creating surveys, structured interviews, online training and ongoing support from research coaches. 14 CS members agreed to the 12-week and 20-hour commitment for an honorarium. CS members opted to do both qualitative and quantitative assessments. CS members collected 261 surveys and 39 structured interviews. The largest number of surveys were collected for Task 1 (n = 102) to assess survivors’ reactions to different possible options for motivating survivors to visit a study website; 77% of this data were complete (i.e., no missing values). Data collected for tasks 2, 3, 4, and 5 (e.g., assessment of survivors’ and relatives’ respective communication preferences) ranged from 10 to 58 surveys (80% to 84% completeness). All data were collected within the specified time frame. CSs reported 17 hours of work on average and regarded the experience positively. Our experience suggests that CS engagement is feasible, can yield comprehensive quantitative and qualitative data, and is achievable in a relatively a short timeline.

 http://dx.doi.org/10.13039/100000054 national cancer institute U01CA240581-02 https://orcid.org/0000-0001-7854-4372 McBride Colleen M. The research was funded by a grant from the National Cancer Institute (NCI) of the NIH that has been acknowledged in the text. The following authors received some salary support from the NCI grant: Drs. McBride, Guan, Pentz, Shepperd; Mr. Campbell, Ms. Zhao and Mr. Komonos. The additional authors did not receive salary support from the NCI grant. However, Ms. Cannova and Dr. Byrne served as Citizen Scientists and received an honorarium from the NCI grant for their service. Ms. Paris volunteered her efforts. The NCI had no role in study design, data collection, data analysis, the decision to publish, or preparation of the manuscript. Data Availability The Emory Institutional Review Board has restricted the public sharing of the data. The underlying the results presented in the study are available from Mr. James Leonard, Chief Information Officer, Rollins School of Public Health, james.leonard@emory.edu. Introduction

Patient and public involvement (PPI) is recommended to be an important responsibility of health program planning and implementation globally [1–3]. PPI offers a process of engaging target audiences in a central role as “co-creaters” of intervention content, approaches, and evaluation methods. In turn, this approach can maximize the likelihood that health promotion programs will be relevant, successful, and acceptable. Indeed, PPI approaches have been associated with increased acceptance, adherence, and sustainability of intervention programs in a number of health contexts [4–8].

PPI in research comprises a continuum of approaches ranging from the public having informal and consultative roles to full control over the program development and implementation process [9, 10]. The conundrum for research collaborations involving the public is to maximize the extent of engagement while considering pragmatic concerns, such as time constraints of externally funded research. To this end, we evaluated the feasibility of engaging citizen scientists, individuals from families affected by ovarian cancer, as part of initial planning for the “Your Family Connects” intervention aimed to encourage uptake of genetic services [U01CA240581-02].

Citizen science (CS) approaches involve non-professionals researcher (henceforth, “citizens”) as collaborators in research to shape measures, refine methods, collect and interpret data [11]. While engagement of CSs in environmental sciences research has gained popularity [12, 13], CSs have been engaged infrequently in health promotion generally [14] and specifically, to cancer research [15]. In the field of genomics, a recent review of global human genomics research identified just 32 of 96 basic science and clinical initiatives reporting the use of PPI approaches defined as “active involvement in shaping and guiding research” [16] with only two using CS approaches. One used crowdsourcing (via Amazon Turk) to document the large public knowledge gap between high awareness of breast- and low awareness of ovarian cancer [17]. Nunn and colleagues [16] concluded that genomics- and related translation researchers could benefit from standardized methods to apply PPI.

Standardized CS approaches applied in the context of ovarian cancer may be especially useful for developing communication interventions to consider cancer genetic services. Up to 20% of ovarian cancers are linked to inherited genetic mutations in cancer predisposing genes (Hereditary Breast and Ovarian Cancer (HBOC)) [18]. Life-saving prevention options are available to families confirmed via genetic testing to be mutation carriers (e.g., BRCA1/2). Yet, uptake of genetic counseling and testing among those diagnosed with ovarian cancer, referred to as survivors, and their blood relatives is low [19]. Using CS approaches, researchers could collaborate with families affected by ovarian cancer to co-create optimal outreach interventions. CS approaches could enable expeditious collection of broader perspectives via support groups, foundations, and personal networks that provide insights with greater credibility to the target audience.

Moreover, concerns about insurance discrimination and privacy can hinder ovarian cancer families’ willingness to discuss inherited risk potential with researchers and health professionals. Such concern may be attenuated by involving CS members in conducting data collection. Survivors might be more willing to contact relatives and in turn, relatives more responsive to provide perspectives when approached by a CS member. Additionally, researchers who are passionate about cancer prevention and control could mentor CSs to gain greater appreciation of the value of research participation.

Current clinical practice and ethical guidelines assign individuals at high risk for inherited genetic mutations the responsibility to inform their first- and second-degree blood relatives that they too may be mutation carriers [20]. Low reported awareness of hereditary risks among relatives suggests that relying solely on a survivor outreach approach may be inadequate [21, 22]. CS engagement could enable deepened understanding of family communication barriers and receiver resistance factors.

With these considerations, we enlisted survivors of ovarian cancer and their close relatives as CS collaborators to collect and help interpret data to inform content for a website, printed invitation materials, and short-message reminders to be tested in a comparative effectiveness trial. We based the procedures for the approach on implementation quality criteria described by Heigl and colleagues [23]. In implementing the CS approach, we posed four research questions: (1) Can ovarian cancer survivors and close relatives be recruited to partner with professional researchers in a defined start-up window?; (2) Can these CS members be trained to address specific data collection tasks and meet specified data collection goals in a 12-week timeline?; (3) What is the quality of the data collected? And, (4) How do the CS members rate the experience?

 Methods

An interdisciplinary team of researchers initiated the research at Emory University. The CS protocol was reviewed and approved by the Emory University Institutional Review Board [Emory IRB: STUDY00000224]. Here we provide a description of the methods used to engage and collaboration with CS members. The CS implementation was operationalized drawing on a set of minimum criteria for good practice developed by Heigl and colleagues (2020) [Table 1] [23]. Six criteria are outlined: scientific standards, collaboration, open science, communication, ethics, and data management, along with specific criterion for each.

10.1371/journal.pone.0262575.t001 “Your family connects” citizen science (CS) methods mapped to quality metrics<xref ref-type="table-fn" rid="t001fn001">*</xref>.
Quality Criteria Specific criteria “Your Family Connects” Implementation
Scientific standards State scientific question, hypothesis or goal to be answered with the project Online Orientation Training
Methods presented in field-specific, appropriate and comprehensible way Menu of methods with descriptions
New knowledge generated Online Orientation Training
Collaboration Must be added value for CS and professional researchers CSs opportunity to shape services for families affected by ovarian cancer.
Professional researchers opportunity to increase fit of targeted intervention
Objectives are unachievable without the CS collaboration CS exclusive access to online communities of families affected by ovarian cancer
CS involved in at least one project element (e.g., data collection, data analysis & interpretation) CS involved in data instrument development, data collection and interpretation
Project definition and objective are open, clear, and communicated in a generally comprehensible manner Online Orientation Training
Assignment of tasks must be clear and transparent Online Orientation Training
Open Science Results are published in an open access format In progress
Communication Different interest groups are addressed accordingly CS teams include survivors and relatives, representing 4 regions of the U.S. and 9 state
Contact between project management and CSs possible at all times Coaches available via text and email at all times
CSs receive feedback on the progress and the results of the project Final meeting of full group of CS to discuss findings; offered co-authorship on manuscripts
Ethics Informed consent is obtained from CSs One-on-one online conferencing, review of detailed consent form, return signed consent
Research team used all data collected and expressed sincere appreciation of the CSs work
Data management Prior to data collection, all projects must have established a data management plan A detailed data analysis plan and its application to intervention development was specified as part of the approved research.

*Adapted from Heigl, Kieslinger, Paul, et al., Citizen Science: Theory and Practice, 2020 [23].

 Developing a theory-based intervention

The “Your Family Connects” intervention was conceived to align with the “Foot-in-the-Door” technique (FITD), a stepped approach we wanted to test to encourage families at high risk for ovarian cancer to seek genetic counseling services [24]. The central premise of FITD is that people are more likely to agree to a large request (e.g., providing contact information for close relatives) if they have first agreed to a small, easy request (e.g., visit a website designed exclusively for survivors of ovarian cancer) [25, 26]. The FITD technique may be effective for a variety of reasons including the possibility that it elicits consistency concerns [27] or because of self-perception processes [28] whereby people infer their value from their initial behavior [25]. While the logic of this approach is straight-forward and testable, the focus and content can take many forms that may be more or less appropriate depending on the situation. To date, no one has engaged CSs to operationalize behavioral theory (e.g., what constitutes a small- or large ask).

Ensuring the quality of CS-collected data is critical to the reliability and usefulness of the information. CS members have varying levels of expertise that are likely not comparable to professional researchers [29]. To this end, the professional research team operationalized CS tasks in accordance with scientific standards to follow the FITD theoretical logic: Task 1: Identify connected steps of contact that would motivate ovarian cancer survivors (OCS) to seek new information via a website; Task 2: Obtain OCSs’ perspectives on contacting their blood relatives regarding inherited risk; Task 3: Obtain close relatives’ perspectives on being contacted by the OCS or professionals; Task 4: Collect perspectives on information sharing within families after an ovarian cancer diagnosis; and Task 5: Characterize OCS- and close relatives’ needs regarding genetic counseling and testing.

 CS toolkit

The research team operationalized five data collection methods for the CS members to choose from: survey, structured interview, personal story, online panel, and role plays options [Table 2]. Additionally, the research team developed a bank of survey items that aligned with each of the tasks and methods including structured interview templates with scripting and open-ended questions [See S1 File for the Citizen Science Toolkit].

10.1371/journal.pone.0262575.t002 Menu of data collection methods and explanation.
Methods Explanations
Survey For this activity, you would identify a group to complete an anonymous short survey. This activity could be taken on by multiple CS participants who would do fewer surveys (10 or so) within survivors (or family members) or a single CS participant could identify a large online listserv community for administering the survey.
Structured Interview For this activity, you would identify individuals who you would interview by phone or via electronic options using a set of open-ended questions. You would be asked to keep the interviews anonymous to the researchers. Each interview would take 30–45 minutes and responses to the questions would be audio-recorded and later coded.
Personal story For this activity, you would identify pairs of individuals (a survivor paired with a close relative) and ask them to tell their story of their relationship since the cancer diagnosis. You would be provided with probing questions to guide the story telling. The stories would be audio-recorded, later coded, and remain anonymous to the researchers.
Online panel For this activity, you would identify 3–5 individual (all survivors or close relatives or a mix of the two) to participate in an online discussion via zoom or another electronic platform. You would pose a set of questions to the panel and the discussion would be audio-recorded and later coded. Panelists would be not be identified in any materials.
Role plays For this activity, you would try out tools that we are developing with survivors [and relatives] and record their comments and reactions to the tool.
CS recruitment

We initiated recruitment in September 2020. Eligible CS members were 25 years or older, spoke English, had used video-conferencing, and indicated having personal or professional survivorship connections amenable to support data collection. Although initially designed to recruit ovarian cancer survivors and their relatives living in Georgia, the onset of the COVID-19 pandemic required us to shift from an in-person approach to a virtual format. We broadened recruitment to the continental United States. CS members were recruited through three networks: clinical connections made in partnership with community organizations, local and national cancer support organizations, and online ovarian cancer patient support groups [Fig 1]. Additionally, snowball sampling or chain referral sampling was used such that individuals interested in participating were encouraged to refer others they knew who met the eligibility criteria [30].

10.1371/journal.pone.0262575.g001 Recruitment strategies.

In network 1, we informed 36 gynecologic oncologists identified by the Georgia Center for Oncology Research and Education (CORE) about the study. Interested oncologists received the IRB-approved study flyer and distributed it to patients. For network 2, the study team asked seven local ovarian cancer foundations (e.g. Lilies of the Valley at Alabama) and two national ovarian cancer organizations (Ovarian Cancer Research Alliance; Facing Hereditary Cancer Empowered) to share study flyers with their members. In network 3, the study team received approval to post study flyers on open online patient support groups and national support and discussion groups (n = 5; e.g., American Cancer Society Network Discussion Forum). The flyer directed interested people to complete an on-line eligibility screening questionnaire.

 CS implementation Informed consent

Once deemed to be eligible, a study investigator interviewed the prospective CSs via Zoom or phone call. The study investigator provided the participant with an overview of the study, tasks to be implemented, time commitment and data collection expectations (20 hours over 12 weeks), compensation ($550), and their rights as a CS. CSs were then asked to review a detailed written consent document and to recontact study investigators if they remained interested. Those willing to participate returned the electronic consent form via encrypted email at Emory.

 Orientation training

Due to COVID restrictions, we consented and trained the CS members virtually. In advance of the training session, we provided CS members with a 30-minute pre-recorded orientation presentation that described the background rationale for the five selected tasks, and descriptions of each of the five method options. CS members selected a task, data collection method, and whether to work individually or as a team.

 Coaching sessions

One of two trained research coaches (GC, JZ) worked with each of five teams in several sessions. In session 1, the coaches introduced themselves and discussed their role with the CS team. Each team received data collection goals (e.g., 50 surveys per CS member; 2–3 structured interviews per CS member). In session 2, CS members discussed how they would engage with their personal or professional networks to collect data. The coaches also guided a discussion of the team’s perceptions of what was feasible with respect to data collection so they could settle on their data collection methods and sources. Coaches also gave brief tutorials on how to use Webex to conduct structured interviews and Qualtrics for surveys. In Session 3, teams finalized the surveys and interview guides, and discussed network outreach strategies. They selected questions from a bank of questions we developed, making edits to questions and interview guides as needed. Most edits were directed at increasing clarity and personalizing the introduction they would use with their networks.

Coaches kept track of each citizen scientists’ progress, reminding them of the study deadline and their progress toward their goals. Coaches also initiated Webex structured interviews so that we could save recordings to the study team’s account. Prior to scheduled interviews, coaches sent copies of the specific interview guides and Webex instructions to the CS member. CS members who reported concern about reaching their goal consulted with the coaches to find more sources of interviews or organizations to share their survey links.

Team boundaries loosened considerably as the process of data collection progressed. For example, coaches made the CS members’ in one team aware of other team’s challenges with data collection. When Team 2 ran into difficulties linking into professional networks, Team 3 offered to include Team 2’s survey within Team 3’s broader networks. This process of collaboration was facilitated by the brevity and distinctness of the surveys.

 Statistical analysis

Descriptive statistics were used to compute counts and indicators of distribution including means, standard deviations and proportions for quantitative and qualitative data collected for each task.

 Results Recruitment of ovarian cancer survivors and close relatives

Over the course of six weeks of CS recruitment (Sept-October, 2020), 22 individuals reached out to the study team to consider serving as a citizen scientist, 21 completed the screener, 4 were deemed ineligible (never used video chat; no professional/personal networks). The majority of prospective CS members reported hearing about the opportunity through ovarian cancer support groups (n = 10). Of the 17 deemed eligible, 2 declined after discussing the study with a co-investigator and 1 was lost to follow-up; 14 chose to proceed as CS members. All CS members were female, 11 were survivors and 3 were close relatives of someone who had experienced ovarian cancer, 3 lived in the Southwestern U.S., 3 lived in the Midwest, 1 in the Northeast, and 7 were in the Southeast. The members represented six foundations.

 Data completeness

Each team opted to do both qualitative and quantitative assessments. No team opted for personal stories, online panels, or role plays. In total, CS members collected 261 surveys and 39 structured interviews [Table 3]. A summary of data collected in the designated period (by January 22, 2021) appears in Table 3 by task. Survey data were collected primarily via the CS members’ personal and professional listservs, that is electronic mailing list software applications used by U.S. based ovarian cancer advocacy groups. Team 1 collected the largest number of surveys (Task 1: n = 102) assessing survivors’ reactions to different possible options for motivating survivors to visit the study website; 77% of the data were complete (i.e., no missing values). Teams 2, 3, 4, and 5 collected between 10 and 58 surveys (80% to 84% completeness). All survey data was collected within the specified time frame.

10.1371/journal.pone.0262575.t003 Citizen science collected data and completeness.
Citizen Science Task Surveys (N) Minimum Survey Completenessa (%) Structured Interviews (N) Minimum Structured Interview Completenessa (%)
Task 1: Identify connected contact steps to motivate survivors to see new information via a website 102 77 13 82
Task 2: Obtain survivor perspective on contacting relatives 45 84 4 50
Task 3: Obtain relative perspective on being contacted 46 83 7 75
Task 4: Collect perspectives on information sharing within families after an ovarian cancer diagnosis 58 81 7 100
Task 5: Characterize survivors’ and blood relatives’ information needs regarding genetic counseling & testing 10 80 8 80

a % Completeness = Item with lowest completion/total number of items included.

Of the 39 structured interviews completed, 13 were completed for Task 1, 4 for Task 2, 7 for Task 3, 7 for Task 4, and 8 for Task 5 [Table 3]. Completeness of structured interviews was based on adherence to the interview script (i.e., asked all questions on the script). Across structured interviews, 50–82% were complete. However, the interviews varied considerably in length (Mean = 26 mins (SD = 16.3 minutes; Range: 7 to 78 minutes). Additionally, five scheduled interviews (1 survivors; 4 relatives) were not recorded at the participant’s request. Instead, the CS interviewer took notes. Seven interviews were conducted in writing because the CS member was undergoing treatment. In these cases, the duration of the interview was unknown.

To supplement areas where the data was most sparse (Task 5), the study team hosted two panels that the CS members could voluntarily attend in week 10 and week 11 of the data collection window. A member of the research team conducted the panel discussions, prompting the CS members to answer either from their own experiences or from the experiences expressed by participants interviewed over the preceding weeks. In the final week, the study team reported the study results to CS members in a virtual session.

 CS members’ rating of the experience

CS members completed a survey evaluating their experience. On average CS members reported spending 16.8 hours (SD = 8.1) on the project, less on average than the estimated 20-hour commitment [Table 4]. Ratings of experience with training, assistance from coaches, and fellow team members fell in the middle of the five-point where 1 indicated “less than expected or needed”, 3 indicated “as expected” and 5- “more than expected/needed”. Satisfaction ratings averaged in the middle of the scale for time devoted the project, what they learned, and the difficulty of the tasks, but toward the top of the scale for overall experience.

10.1371/journal.pone.0262575.t004 Citizen scientists’ evaluations of their experience.
Domains Mean SD
Time spent on project activities (hours) 16.8 8.06
Satisfaction with:
    Activity you chose 4.6 0.65
    Group/Team Interactions 4.4 0.93
    Compensation received 4.9 0.53
    Overall experience 4.6 0.85
(1-Very Dissatisfied– 5-Very Satisfied)
The amount of time I devoted to this project took: 1-Much less time than I expected– 5-Much more time than I expected 2.6 0.63
What I learned from this project was: 1- Much less than I expected– 5- more than I expected 3.4 0.87
The tasks I was asked to do were: 1-Much easier than I expected– 5-Much harder than I expected 3.1 0.73
The training I received for this project
1-Much less than I needed/wanted– 5-Much more than I needed/wanted 2.9 0.36
The help I received from the coaches
1-Much less than I needed/wanted– 5-Much more than I needed/wanted 3.0 0.00
The assistance I received from my team members
1-Much less than I needed/wanted– 5-Much more than I needed/wanted 2.8 0.60
Participating in this project enabled me to:
Make new connections within the ovarian cancer survivorship community 4.0 1.04
Learn more about the needs of ovarian cancer survivors and their families 3.9 0.95
Gain insights about the experiences of ovarian cancer survivors that will be useful to me 3.6 1.15
Make a valuable contribution to the community of ovarian cancer survivors 4.3 0.83
(1-Strongly disagree– 5-Strongly agree)
Experience with:
    Research Training 3.9 0.77
    Research Team 4.3 0.61
Coach 4.4 0.74
(1-Poor– 5-Excellent)

CS members also averaged above the scale midpoint in their ratings of how useful the experience was for making new connections within the ovarian cancer survivorship community, learning more about the needs of ovarian cancer survivors and their families, gaining insights about the experiences of ovarian cancer survivors that will be useful to me, and making valuable contribution to the community of ovarian cancer survivors.

 Discussion

Using the Heigl et al., framework, we implemented a successful effort to recruit a diverse group of CS members representing ovarian cancer survivors and their close relatives. Fourteen people participated from four regions of the U.S. CS members worked with the guidance of a research coach who convened meetings via online conferencing, provided technical support related to data collection and provided updates on progress towards their goals.

Our results show that in a 12-week timeframe we showed favorable outcome, process and feasibility indicators aligned with CS evaluation frameworks [31]. These include CSs collected more than 200 surveys, and completed 39 structured interviews, far more data than our research team could have accomplished using lower order PPI (e.g., ad hoc focus groups). The CS teams’ ready access to large online support groups dedicated to ovarian cancer survivors and their families and the ongoing technical support they received from research coaches facilitated their success. Our success compares favorably to most published program development research, where formative data collection often results in small samples (<30; e.g. [32]). Additionally, CSs reported that they were able to accomplish these goals by committing an average of 17 hours over the 12 weeks, slightly below what we had asked them to anticipate.

Relating to the scientific process and outcome indicators [31], the data collected were complete overall and comprehensively covered four of the five assigned tasks. Study investigators led two debriefing panel discussions to gauge CS members’ perspectives regarding the broader information needs related to ovarian cancer that they heard expressed or that they felt were important for the message intervention we were developing.

Relating to outcomes and impact [31], CSs rated their experiences aligned well with their expectations. They viewed the coaching and meetings to be helpful—not too little or too much. They reported feeling as though they had made a contribution to the community of ovarian cancer survivors and family, a quality criterion also noted by Heigl’s framework. Additionally, CS members left wanting additional opportunities to take on similar roles in other research projects.

Despite these positive outcomes, a few areas need further consideration. Our recruitment strategies failed to engage any male relatives to serve as CS members. CS members reported and the researchers agreed, that they could have benefited from more in-depth training in conducting structured interviews. CS members had little experience in using probing techniques, that is, using open-ended questions to promote critical thinking on the part of the interviewee. Thus, several of the structured interviews were brief and functioned more as survey data. This limited skill set may explain why CSs selected only one of the qualitative data collection options. The additional three qualitative methods all involved less structured interactions with participants. This has been reported elsewhere as a challenge for CS approaches [31].

Additionally, CS members encountered participant mistrust regarding the privacy of the information collected. Several of the CS members reported that their family members or other interviewees were not comfortable having their interviews recorded. Professional researchers and coaches provided CS members with additional rationale for recording (e.g., “the information you provide is so important, we don’t want to miss anything”) and assurances of privacy (e.g., reiterating IRB privacy requirements), but these assurances often did not assuage the concerns. Assisting CSs in garnering trust in genetics-related research likely will require focused training in conveying the nuances of Institutional Review Board requirements.

Practical implications of this report include that guided by an implementation framework CS engagement is feasible, achievable in a relatively a short timeline (even amidst a pandemic), and can provide comprehensive quantitative and qualitative data. Moreover, this approach enables CSs to meaningfully engage in developing interventions for their own communities, something that supports social justice goals and improves the fit of the intervention approach. This report also provides a detailed description of our CS methods and outputs to enable CS partnerships with professional researchers to apply these methods in their own contexts. Efforts to develop health promotion programs to apply genomic discovery may be especially benefitted by PPI approaches such as CS.

 Supporting information Citizen science toolkit.

(PDF)

We thank our citizen scientists for their important contributions in their community data collection outreach efforts: Jess BeCraft, Susan Bossert, Jan Byrne, Kelly Cannova, Jade Gibson, Phyllis Gilbert, Nancy Hicks, Aryn Kinney, Cindy McKinnon Deurloo, Regina Parker, Stacy Saravo, Marilyn Slodki, Summer Southern, and Kamilah Staggers. We gratefully acknowledge Georgia Center for Oncology Research and Education (Georgia CORE), Ovarian Cancer Research Alliance (OCRA), Facing Hereditary Cancer Empowered (FORCE), American Cancer Society Cancer Survivors Network, Ovarian Cancer Support Group, Ovarian Cancer Community for their assistance in recruiting citizen scientists. We thank the Intervention Development, Dissemination and Implementation (IDDI) Core Resource of the Winship Cancer Institute (P30CA138292) for assistance in coaching citizen scientists in survey development (Shaheen Rana, Director).

 References Newell KW, World Health O. Health by the people / edited by Newell Kenneth W. Geneva: World Health Organization; 1975. Ocloo J, Garfield S, Franklin BD, Dawson S. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Research Policy and Systems. 2021;19(1):8. doi: 10.1186/s12961-020-00644-3 33472647 Organization WH. Primary health care: report of the International Conference on primary health care, Alma-Ata, USSR, 6–12 September 1978: World Health Organization; 1978. Banna J, Bersamin A. Community involvement in design, implementation and evaluation of nutrition interventions to reduce chronic diseases in indigenous populations in the U.S.: a systematic review. International Journal for Equity in Health. 2018;17(1):116. doi: 10.1186/s12939-018-0829-6 30103753 Brewer LC, Hayes SN, Caron AR, Derby DA, Breutzman NS, Wicks A, et al. Promoting cardiovascular health and wellness among African-Americans: Community participatory approach to design an innovative mobile-health intervention. PLOS ONE. 2019;14(8):e0218724. doi: 10.1371/journal.pone.0218724 31430294 Crocker JC, Ricci-Cabello I, Parker A, Hirst JA, Chant A, Petit-Zeman S, et al. Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ. 2018;363:k4738. doi: 10.1136/bmj.k4738 30487232 Linnan LA, D’Angelo H, Harrington CB. A Literature Synthesis of Health Promotion Research in Salons and Barbershops. American Journal of Preventive Medicine. 2014;47(1):77–85. doi: 10.1016/j.amepre.2014.02.007 24768037 Rifkin SB. Examining the links between community participation and health outcomes: a review of the literature. Health Policy and Planning. 2014;29(suppl_2):ii98–ii106. doi: 10.1093/heapol/czu076 25274645 Arnstein SR. A Ladder Of Citizen Participation. Journal of the American Institute of Planners. 1969;35(4):216–24. Brunton G, Thomas J, O’Mara-Eves A, Jamal F, Oliver S, Kavanagh J. Narratives of community engagement: a systematic review-derived conceptual framework for public health interventions. BMC Public Health. 2017;17(1):944. doi: 10.1186/s12889-017-4958-4 29228932 Wiggins A, Wilbanks J. The Rise of Citizen Science in Health and Biomedical Research. The American Journal of Bioethics. 2019;19(8):3–14. doi: 10.1080/15265161.2019.1619859 31339831 Adler FR, Green AM, Şekercioğlu ÇH. Citizen science in ecology: a place for humans in nature. Annals of the New York Academy of Sciences. 2020;1469(1):52–64. doi: 10.1111/nyas.14340 32216157 Kelly R, Fleming A, Pecl GT, von Gönner J, Bonn A. Citizen science and marine conservation: a global review. Philosophical Transactions of the Royal Society B. 2020;375(1814):20190461. doi: 10.1098/rstb.2019.0461 33131446 Den Broeder L, Devilee J, Van Oers H, Schuit AJ, Wagemakers A. Citizen Science for public health. Health promotion international. 2018;33(3):505–14. doi: 10.1093/heapro/daw086 28011657 Lee YJ, Arida JA, Donovan HS. The application of crowdsourcing approaches to cancer research: a systematic review. Cancer Med. 2017;6(11):2595–605. doi: 10.1002/cam4.1165 28960834 Nunn JS, Tiller J, Fransquet P, Lacaze P. Public Involvement in Global Genomics Research: A Scoping Review. Front Public Health. 2019;7:79. doi: 10.3389/fpubh.2019.00079 31024880 Carter RR, DiFeo A, Bogie K, Zhang GQ, Sun J. Crowdsourcing awareness: exploration of the ovarian cancer knowledge gap through Amazon Mechanical Turk. PLoS One. 2014;9(1):e85508. doi: 10.1371/journal.pone.0085508 24465580 Norquist BM, Harrell MI, Brady MF, Walsh T, Lee MK, Gulsuner S, et al. Inherited mutations in women with ovarian carcinoma. JAMA oncology. 2016;2(4):482–90. doi: 10.1001/jamaoncol.2015.5495 26720728 Menko FH, Ter Stege JA, van der Kolk LE, Jeanson KN, Schats W, Moha DA, et al. The uptake of presymptomatic genetic testing in hereditary breast-ovarian cancer and Lynch syndrome: a systematic review of the literature and implications for clinical practice. Familial cancer. 2019;18(1):127–35. doi: 10.1007/s10689-018-0089-z 29846880 Henrikson NB, Wagner JK, Hampel H, DeVore C, Shridhar N, Williams JL, et al. What guidance does HIPAA offer to providers considering familial risk notification and cascade genetic testing? Journal of Law and the Biosciences. 2020;7(1). doi: 10.1093/jlb/lsaa071 34221429 Pozzar RA, Berry DL. Patient-centered research priorities in ovarian cancer: A systematic review of potential determinants of guideline care. Gynecol Oncol. 2017;147(3):714–22. doi: 10.1016/j.ygyno.2017.10.004 29029759 Roberts MC, Dotson WD, DeVore CS, Bednar EM, Bowen DJ, Ganiats TG, et al. Delivery of cascade screening for hereditary conditions: a scoping review of the literature. Health Affairs. 2018;37(5):801–8. doi: 10.1377/hlthaff.2017.1630 29733730 Heigl F, Kieslinger B, Paul KT, Uhlik J, Frigerio D, Dörler D. Co-Creating and Implementing Quality Criteria for Citizen Science. Citizen Science: Theory and Practice. 2020;5(1). Freedman JL, Fraser SC. Compliance without pressure: the foot-in-the-door technique. Journal of personality and social psychology. 1966;4(2):195. doi: 10.1037/h0023552 5969145 Burger JM. The foot-in-the-door compliance procedure: A multiple-process analysis and review. Personality and social psychology review. 1999;3(4):303–25. doi: 10.1207/s15327957pspr0304_2 15661679 Joule R-V, Girandola F, Bernard F. How Can People Be Induced to Willingly Change Their Behavior? The Path from Persuasive Communication to Binding Communication. Social and Personality Psychology Compass. 2007;1(1):493–505. Heider F. The psychology of interpersonal relations.: New York: Wiley; 1958. Bem DJ. Self-perception: An alternative interpretation of cognitive dissonance phenomena. Psychol Rev. 1967;74(3):183–200. doi: 10.1037/h0024835 5342882 López MP, Soekijad M, Berends H, Huysman M. A Knowledge Perspective on Quality in Complex Citizen Science. Citizen Science: Theory and Practice. 2020;5(1). Etikan I, Alkassim R, Abubakar S. Comparision of snowball sampling and sequential sampling technique. Biometrics and Biostatistics International Journal.3(1):55. Schaefer T, Kieslinger B, Brandt M, van den Bogaert V. Evaluation in Citizen Science: The Art of Tracing a Moving Target. The Science of Citizen Science. 2021:495. Greenberg S, Yashar BM, Pearlman M, Duquette D, Milliron K, Marvin M. Evaluating and improving the implementation of a community-based hereditary cancer screening program. Journal of community genetics. 2019;10(1):51. doi: 10.1007/s12687-018-0357-5 29508367 10.1371/journal.pone.0262575.r001 Decision Letter 0 Corpas Manuel Academic Editor 2022 Manuel Corpas This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

27 Aug 2021

PONE-D-21-22428

Applying Citizen Science to engage families affected by ovarian cancer in developing genetic service outreach strategies

PLOS ONE

Dear Dr. McBride,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please make sure that you address the minor comments suggested by reviewers, which I am sure will improve the manuscript. 

Please submit your revised manuscript by Oct 11 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Manuel Corpas, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the Methods section, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

3. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables (should remain/ be uploaded) as separate "supporting information" files.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The citizen science project was supported by National Cancer Institute grant U01CA240581-02.”

We note that you have provided funding information within the Acknowledgements Section. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“Yes, the work was funded by an NIH research grant that has been acknowledged in the text.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

6. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

7. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

8. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors aim to evaluate the feasibility of a Citizen Science quality framework, recruiting survivors of ovarian cancer and their close relatives, suggesting that CS engagement is fesible. The manuscript is well written but requries some minor revisions in order to be more comprehensive. Please find my comments as follows:

Introduction

Page 3 line 69, there is no reference for the sentence

“In the initial 63 planning year of a four-year NCI-funded study entitled “Your Family Connects” we applied citizen 64 science to generate intervention content for a message-based outreach intervention to encourage 65 families affected by ovarian cancer to consider genetic service options.”

Page 4, line 75. There is no reference for the sentence “Nunn and colleagues concluded that genomics- and 75 related translation researchers could benefit from standardized methods to apply PPI”. This regards the ref 16, which should be reported also in this sentence, otherwise it seems that is the continuum of ref 17.

Page 4, line 77-78, the sentence “Up to 20% of ovarian 78 cancers are linked to inherited genetic mutations in cancer predisposing genes”, should not be underlined.

Page 4, line 89 It is not clear the transition to concerns, i would suggest to add a sentence on the general concerns before describing each of them in details.

Page 5, line 107-108, i would suggest to include Table 1 in the Methods section and leave in the introduction the reference.

The introduction is quite long and should be shortened, with a focus on Citizens sceicne use in cancer and genetic outreach sterategies

Methods

Page 7, line 121, the authors should add the Protocol Approval Number from the Echics Committe

Page 8, line 137, Table 2 contains the results of data compelteness and should be included only in the results.

There is not a description of data statistical analysis process, eg using mean and SD

Results

In the academic writing, number less than 10 are suggested to be written in letters. The authors might conider writing these number is letters.

Discussion

The authors did not report an strenghts and limitation of their study.

The authors should describe the practical implication of their study.

Reviewer #2: Summary

The was an interesting paper, thank you for the chance to review it. Some small pick ups on consistent terminology will help improve this excellent paper.

Major issues: No major issues

Minor issues: The following points are mainly around consistency language, which if improved, will improve the readability of this paper.

Abstract:

‘nonprofessionals’ – should read non-professional researchers – as they may be professionals in other fields?

Data was/were line 38/40 – consistent?

Line 31 – 40 was complex and could be summarised more clearly or succinctly, for example, adding a heading ‘method’ and ‘results’.

Intro:

Line 54: the word citizen appears here for the first time and hasn’t been defined previously. Suggest clarification of terminology in first paragraph when defining patient and public involvement.

59: PPI collaboration or just PPI, if introducing the term collaboration does this need to be defined? Same with the word engagement.

61: ‘higher order citizen engagement’ – again, does this mean PPI? Consider using a task based approach rather than levels to describe involvement – as in ‘involvement in research design’ (high level) or just participation in survey responses (low level). A lot of this terminology will be unfamiliar to some readers so it’s important to be consistent in the terminology you are using.

63: ‘applied citizen science’ I would contest that you can ‘apply’ citizen science as a method, you could say ‘we involved people using methods informed by citizen science’ or words to that effect, if you see the distinction. At the very least you could say ‘we applied citizen science methodology’ but you’d need to carefully define what exactly you mean by that before using the shorthand later.

66: ‘Citizen science (CS) approaches involve nonprofessionals (citizens)’ suggest rewording as ‘Citizen science (CS) methods [metholodiges/paradigms?] involve non-professional researchers (citizens)

67: shouldn’t you add ‘collect data’ here too, as that is what happened with this project?

72: again, consider the word ‘approaches’ and defining it sooner. Methods, modes and tasks had very clear definitions in the review you are citing and I think this paper would be improved if terms like ‘citizen science approaches’ or ‘citizen science methods’ we used more consistently. I acknowledge this a wider issue of defining woolly terms such as CS (see my later comment about STARDIT), and you could consider using ‘participatory action research’ as an alternative term to CS.

75: regarding the line ‘Nunn and colleagues concluded that genomics- and related translation researchers could benefit from standardized methods to apply PPI’ you might be interested to learn about the project Standardised Data on Initiatives (STARDIT) which followed based on the recommendations of this scoping review and is now in a Beta stage of development. It allows stanardised reporting of involvement and the impacts of involvement (among much else). Pre-print here: https://doi.org/10.31219/osf.io/w5xj6 - and a friendly invitation to all authors to get involved with the STARDIT project if it’s of interest

82: do not advise the use of the word racial in the line ‘racial/ethnic minorities’, as race is generally considered a social construction. Ethnic acceptable as a short hand (if defined - .e.g is it cultural or are you specifically referring to inherited characteristics – e.g. groupings of people more pre-disposed to certain cancers with more awareness in those associated communities?) but perhaps also noting language communities and other cultural influences which might affect uptake. The point is you are grouping people by race in this comment, and people may not self-identify with such groupings. I note the papers you cite use this term, but in a health context I’d avoid it unless you are specifically talking about people’s lived experience of racism, which is distinct from reinforcing socially constructed racial categories. This article explains better what I’m trying to say: https://en.wikipedia.org/wiki/Race_(human_categorization)

85: This is a fantastic point and I note you’re now also introducing ‘co-create’. Again I suggest you chose a term at the start, mention all the different things it can be called, and then use that consistently as co-creation as a method might also need more explanation?

92: ‘Survivors’ is this a term that was co-defined by the people you involved, do they self-identify with that? Perhaps ‘people affected by cancer….’ Which also then includes loved-ones/carers of people with cancer etc

94: ‘Additionally, the credibility of researchers who share their passion for expanding reach could engage CS members more deeply in seeing the value of research.’ I think I know what you’re trying to say here, and I think it’s a good point. Could it be made into two sentences and made a little more explicit, perhaps with a ‘for example’? I think it’s an important point and worth expanding on.

96: ‘Individuals who learn they are at high risk for carrying inherited genetic mutations such as BRCA1/2 have the responsibility of (i.e., “duty to warn”) informing first- and second-degree blood relatives that they too may be mutation carriers’ – I would qualify this with a ‘Some individuals….may feel’ – otherwise you are prescribing a moral and ethical stance, which is dangerous waters. I know for example many people who feel it is not their duty and this might be a huge breach of ethics – unless they had asked if someone wanted to know such things first. Naturally this is a very complex area so you may wish to consult a qualified genetic councillor on writing about this accurately. If you wanted to take a CS/PAR approach to this, you’d actually say ‘CS can help understand different people’s attitudes towards sharing information about risk with relatives, and co-create learning and development interventions to support people to make informed decisions when communicating about such risk’ – or words to that effect if you see where I’m coming from?

103: typo – in sum.

107: ‘We based the procedures for the approach on implementation quality criteria

108 described by Heigl and colleagues’ – seeing as this is the basis for a large and complex table, a line or two about what this criteria is and how you applied it to this work might help the reader make more sense of the table (I had to go back and forth a couple of times and read the original paper before it clicked for me – but I might be slow!). line 175 explains it in more detail, and parts of this section could be moved earlier?

Methods and other sections

124: ‘Survivors’ – again, this term is used – is it distinct from other people involved as non-professional researchers (referred to on line 154 as ‘CS members’. Suggest this term is defined, or changed to ‘people affected by cancer (which can include people with, survivors of, people pre-disposed to, family of…etc – define as you wish). Later the acronym OCS is also used – being consistent as early as possible will help the reader navigate these complex terms.

125: the para FITD might need further review by someone more familiar with this method – this short section felt like it might need to be integrated further? In other words, why am I reading about it? Because this method was used? Could a sentence like ‘our method was informed by the FITD….because….) before hand make the link clearer?

146: stylistically, starting a new section with ‘thus’ might need looking at. I think it’s fine after a set up clause, but this is a matter of style, so take or leave.

161: ‘snowball sampling’ – while I know what you mean here, but noting that some readers might not (aside from the fact that some citizen science projects might actually sample actual snow!) perhaps a reference to this method for more information, or a short explanation of what that means?

218: ‘All CS members were female’ – does this paper require some commentary on this? Perhaps a line in the discussion around 284?

225: ‘listservs’ – define this term

252: does this line and others need putting in inverted commas ‘gaining insights about the experiences of ovariancancer survivors that will be useful to me,’?

286: ‘using probing techniques’ suggest you define this

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Ilda Hoxhaj

Reviewer #2: Yes: Jack S Nunn

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0262575.r002 Author response to Decision Letter 0 Submission Version 1

18 Oct 2021

Review Comments to the Author

Reviewer #1

Introduction

Page 3 line 69, there is no reference for the sentence

“In the initial 63 planning year of a four-year NCI-funded study entitled “Your Family Connects” we applied citizen 64 science to generate intervention content for a message-based outreach intervention to encourage 65 families affected by ovarian cancer to consider genetic service options.”

References are noted in the version that was submitted at the line numbers suggested.

Page 4, line 75. There is no reference for the sentence “Nunn and colleagues concluded that genomics- and 75 related translation researchers could benefit from standardized methods to apply PPI”. This regards the ref 16, which should be reported also in this sentence, otherwise it seems that is the continuum of ref 17.

The reference number is now added.

Page 4, line 77-78, the sentence “Up to 20% of ovarian 78 cancers are linked to inherited genetic mutations in cancer predisposing genes”, should not be underlined.

The underline has been removed.

Page 4, line 89 It is not clear the transition to concerns, i would suggest to add a sentence on the general concerns before describing each of them in details.

A transition is now added.

Page 5, line 107-108, i would suggest to include Table 1 in the Methods section and leave in the introduction the reference.

Table 1 has been moved the “Methods” section.

The introduction is quite long and should be shortened, with a focus on Citizens sceicne use in cancer and genetic outreach sterategies

The introduction has been shortened by just over half a page.

Methods

Page 7, line 121, the authors should add the Protocol Approval Number from the Echics Committe

Protocol number has been added.

Page 8, line 137, Table 2 contains the results of data compelteness and should be included only in the results.

Table 2 has been moved to the “Results” section.

There is not a description of data statistical analysis process, eg using mean and SD

A few sentences are now provided to indicate that we used simple descriptive statistics.

Results

In the academic writing, number less than 10 are suggested to be written in letters. The authors might conider writing these number is letters.

Numbers under 10 (unless a Task descriptor) have been spelled out.

Discussion

The authors did not report an strenghts and limitation of their study.

The authors should describe the practical implication of their study.

The narrative now makes it clearer what narrative was intended to describe strengths and limitations.

Reviewer #2: Summary

Minor issues: The following points are mainly around consistency language, which if improved, will improve the readability of this paper.

Abstract:‘nonprofessionals’ – should read non-professional researchers – as they may be professionals in other fields?

Data was/were line 38/40 – consistent?

Line 31 – 40 was complex and could be summarised more clearly or succinctly, for example, adding a heading ‘method’ and ‘results’.

All of the above edits have been made to maintain consistency in terms and break up complex sentences.

Intro:

Line 54: the word citizen appears here for the first time and hasn’t been defined previously. Suggest clarification of terminology in first paragraph when defining patient and public involvement.

Terminology is now clarified up front.

59: PPI collaboration or just PPI, if introducing the term collaboration does this need to be defined? Same with the word engagement.

Now defined.

61: ‘higher order citizen engagement’ – again, does this mean PPI? Consider using a task based approach rather than levels to describe involvement – as in ‘involvement in research design’ (high level) or just participation in survey responses (low level). A lot of this terminology will be unfamiliar to some readers so it’s important to be consistent in the terminology you are using.

In shortening the introduction, this statement was omitted.

63: ‘applied citizen science’ I would contest that you can ‘apply’ citizen science as a method, you could say ‘we involved people using methods informed by citizen science’ or words to that effect, if you see the distinction. At the very least you could say ‘we applied citizen science methodology’ but you’d need to carefully define what exactly you mean by that before using the shorthand later.

Agree, we now describe as “engage CSs”.

66: ‘Citizen science (CS) approaches involve nonprofessionals (citizens)’ suggest rewording as ‘Citizen science (CS) methods [metholodiges/paradigms?] involve non-professional researchers (citizens)

Change has been made as suggested.

67: shouldn’t you add ‘collect data’ here too, as that is what happened with this project?

Collected was added.

72: again, consider the word ‘approaches’ and defining it sooner. Methods, modes and tasks had very clear definitions in the review you are citing and I think this paper would be improved if terms like ‘citizen science approaches’ or ‘citizen science methods’ we used more consistently. I acknowledge this a wider issue of defining woolly terms such as CS (see my later comment about STARDIT), and you could consider using ‘participatory action research’ as an alternative term to CS.

We now use terms consistently and use the terminology we used in the consent form and all CS activities.

75: regarding the line ‘Nunn and colleagues concluded that genomics- and related translation researchers could benefit from standardized methods to apply PPI’ you might be interested to learn about the project Standardised Data on Initiatives (STARDIT) which followed based on the recommendations of this scoping review and is now in a Beta stage of development. It allows stanardised reporting of involvement and the impacts of involvement (among much else). Pre-print here: https://doi.org/10.31219/osf.io/w5xj6 - and a friendly invitation to all authors to get involved with the STARDIT project if it’s of interest

Thank you for making us aware of this resource. We were not able to figure out how to cite this effort in the narrative.

82: do not advise the use of the word racial in the line ‘racial/ethnic minorities’, as race is generally considered a social construction. Ethnic acceptable as a short hand (if defined - .e.g is it cultural or are you specifically referring to inherited characteristics – e.g. groupings of people more pre-disposed to certain cancers with more awareness in those associated communities?) but perhaps also noting language communities and other cultural influences which might affect uptake. The point is you are grouping people by race in this comment, and people may not self-identify with such groupings. I note the papers you cite use this term, but in a health context I’d avoid it unless you are specifically talking about people’s lived experience of racism, which is distinct from reinforcing socially constructed racial categories. This article explains better what I’m trying to say: https://en.wikipedia.org/wiki/Race_(human_categorization)

In shortening the manuscript, this sentence has been omitted.

85: This is a fantastic point and I note you’re now also introducing ‘co-create’. Again I suggest you chose a term at the start, mention all the different things it can be called, and then use that consistently as co-creation as a method might also need more explanation?

Terminology has been made consistent.

92: ‘Survivors’ is this a term that was co-defined by the people you involved, do they self-identify with that? Perhaps ‘people affected by cancer….’ Which also then includes loved-ones/carers of people with cancer etc

Use of this term in now justified in the narrative.

94: ‘Additionally, the credibility of researchers who share their passion for expanding reach could engage CS members more deeply in seeing the value of research.’ I think I know what you’re trying to say here, and I think it’s a good point. Could it be made into two sentences and made a little more explicit, perhaps with a ‘for example’? I think it’s an important point and worth expanding on.

This point is now made in two sentences as suggested.

96: ‘Individuals who learn they are at high risk for carrying inherited genetic mutations such as BRCA1/2 have the responsibility of (i.e., “duty to warn”) informing first- and second-degree blood relatives that they too may be mutation carriers’ – I would qualify this with a ‘Some individuals….may feel’ – otherwise you are prescribing a moral and ethical stance, which is dangerous waters. I know for example many people who feel it is not their duty and this might be a huge breach of ethics – unless they had asked if someone wanted to know such things first. Naturally this is a very complex area so you may wish to consult a qualified genetic councillor on writing about this accurately. If you wanted to take a CS/PAR approach to this, you’d actually say ‘CS can help understand different people’s attitudes towards sharing information about risk with relatives, and co-create learning and development interventions to support people to make informed decisions when communicating about such risk’ – or words to that effect if you see where I’m coming from?

As written the sentence was not making the intended point that those found to carry mutations are by clinical and ethical standards the only ones to communicate with their families. We have now reworded the sentence to make this point clearer.

103: typo – in sum.

In shortening the intro, this sentence was omitted.

107: ‘We based the procedures for the approach on implementation quality criteria

108 described by Heigl and colleagues’ – seeing as this is the basis for a large and complex table, a line or two about what this criteria is and how you applied it to this work might help the reader make more sense of the table (I had to go back and forth a couple of times and read the original paper before it clicked for me – but I might be slow!). line 175 explains it in more detail, and parts of this section could be moved earlier?

Table 1 is now moved to the “Methods” section where it can be given more detailed description.

Methods and other sections

124: ‘Survivors’ – again, this term is used – is it distinct from other people involved as non-professional researchers (referred to on line 154 as ‘CS members’. Suggest this term is defined, or changed to ‘people affected by cancer (which can include people with, survivors of, people pre-disposed to, family of…etc – define as you wish). Later the acronym OCS is also used – being consistent as early as possible will help the reader navigate these complex terms.

See above response.

125: the para FITD might need further review by someone more familiar with this method – this short section felt like it might need to be integrated further? In other words, why am I reading about it? Because this method was used? Could a sentence like ‘our method was informed by the FITD….because….) before hand make the link clearer?

As written, this section was not make the point clearly that CSs were implementing tasks to develop a FITD guided intervention. We were not using the FITD approach on the CSs. These few sentences have been edited to make this point clearer.

146: stylistically, starting a new section with ‘thus’ might need looking at. I think it’s fine after a set up clause, but this is a matter of style, so take or leave.

Omitted

161: ‘snowball sampling’ – while I know what you mean here, but noting that some readers might not (aside from the fact that some citizen science projects might actually sample actual snow!) perhaps a reference to this method for more information, or a short explanation of what that means?

Has been defined and a reference has been added.

218: ‘All CS members were female’ – does this paper require some commentary on this? Perhaps a line in the discussion around 284?

Now described as a limitation

225: ‘listservs’ – define this term

Now defined.

252: does this line and others need putting in inverted commas ‘gaining insights about the experiences of ovarian cancer survivors that will be useful to me,’?

Now added.

286: ‘using probing techniques’ suggest you define this

Now defined.

Submitted filename: CS_response_to_reviewers_PLOS_ONE_FINAL.docx

 10.1371/journal.pone.0262575.r003 Decision Letter 1 Corpas Manuel Academic Editor 2022 Manuel Corpas This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

30 Dec 2021

Applying Citizen Science to engage families affected by ovarian cancer in developing genetic service outreach strategies

PONE-D-21-22428R1

Dear Dr. McBride,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Manuel Corpas, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for the opportunity to read such an interesting manuscript , I wish you all the best and hope we can stay in touch about http://scienceforall.world/STARDIT

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Jack S Nunn

 10.1371/journal.pone.0262575.r004 Acceptance letter Corpas Manuel Academic Editor 2022 Manuel Corpas This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

4 Feb 2022

PONE-D-21-22428R1

Applying Citizen Science to engage families affected by ovarian cancer in developing genetic service outreach strategies

Dear Dr. McBride:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Manuel Corpas

Academic Editor

PLOS ONE