PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0304180 PONE-D-24-04935 Research Article Biology and life sciencesPsychologyBehaviorVerbal behaviorVerbal communication Social sciencesPsychologyBehaviorVerbal behaviorVerbal communication Social sciencesSociologyCommunicationsSocial communication Biology and life sciencesPsychologyEmotions Social sciencesPsychologyEmotions Biology and life sciencesNeuroscienceCognitive scienceCognitive psychologyLearningHuman learning Biology and life sciencesPsychologyCognitive psychologyLearningHuman learning Social sciencesPsychologyCognitive psychologyLearningHuman learning Biology and life sciencesNeuroscienceLearning and memoryLearningHuman learning Biology and life sciencesNeuroscienceCognitive scienceCognitive psychologyDecision making Biology and life sciencesPsychologyCognitive psychologyDecision making Social sciencesPsychologyCognitive psychologyDecision making Biology and life sciencesNeuroscienceCognitive scienceCognitionDecision making Physical sciencesPhysicsClassical mechanicsReflection Medicine and health sciencesHealth carePatients People and placesPopulation groupingsEthnicitiesEuropean peopleItalian people Supporting health and social care professionals in serious illness conversations: Development, validation, and preliminary evaluation of an educational booklet Educational booklet to promote serious illness conversations https://orcid.org/0000-0001-6294-9834 Gonella Silvia Conceptualization Data curation Formal analysis Investigation Methodology Project administration Validation Visualization Writing – original draft Writing – review & editing 1 * Di Giulio Paola Conceptualization Methodology Writing – review & editing 2 Riva-Rovedda Federica Conceptualization Investigation Methodology Writing – review & editing 2 * Stella Luigi Resources Writing – review & editing 3 Rivolta Maria Marcella Resources Writing – review & editing 4 Malinverni Eugenia Resources Writing – review & editing 5 Paleologo Mario Resources Writing – review & editing 1 Di Vella Giancarlo Resources Writing – review & editing 2 Dimonte Valerio Funding acquisition Methodology Supervision Writing – review & editing 1 2 City of Health and Science University Hospital Turin, Turin, Italy Department of Public Health and Pediatrics, University of Torino, Turin, Italy Fondazione Assistenza e Ricerca Oncologica (F.A.R.O.), Turin, Italy Local Health Authority of the City of Turin, Turin, Italy Fondazione Luce per la Vita Onlus, Rivoli, Turin, Italy Wiyarta Elvan Editor Universitas Indonesia Fakultas Kedokteran, INDONESIA

The authors have declared that no competing interests exist.

 * E-mail: silvia.gonella@unito.it (SG); federica.rivarovedda@unito.it (FRR) 31 5 2024 2024 19 5 e0304180 14 2 2024 8 5 2024 2024 Gonella et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Serious illness conversations aim to align the care process with the goals and preferences of adult patients suffering from any advanced disease. They represent a challenge for healthcare professionals and require specific skills. Conversation guides consistent with task-centered instructional strategies may be particularly helpful to improve the quality of communication. This study aims to develop, validate, and preliminarily evaluate an educational booklet to support Italian social and healthcare professionals in serious illness conversations. A three-step approach, including development, validation, and evaluation, was followed. A co-creation process with meaningful stakeholders led to the development of the booklet, validated by 15 experts on clarity, completeness, coherence, and relevance. It underwent testing on readability (Gulpease index, 0 = lowest-100 = maximum) and design (Baker Able Leaflet Design criteria, 0 = worst to 32 = best). Twenty-two professionals with different scope of practice and care settings evaluated acceptability (acceptable if score ≥30), usefulness, feasibility to use (1 = not at all to 10 = extremely), and perceived acquired knowledge (1 = not at all to 5 = extremely). After four rounds of adjustments, the booklet scored 97% for relevance, 60 for readability, and 25/32 for design. In all, 18 (81.8%), 19 (86.4%) and 17 (77.3%) professionals deemed the booklet acceptable, moderate to highly useful, and feasible to use, respectively; 18/22 perceived gain in knowledge and all would recommend it to colleagues. The booklet has good readability, excellent design, high content validity, and a high degree of perceived usefulness and acquired knowledge. The booklet is tailored to users’ priorities, mirrors their most frequent daily practice challenges, and offers 1-minute, 2-minute and 5-minute solutions for each scenario. The co-creation process ensured the development of an educational resource that could be useful regardless of the scope of practice and the care setting to support professionals in serious illness conversations.

 Fondazione Assistenza e Ricerca Oncologica This work was supported by Fondazione Assistenza e Ricerca Oncologica (F.A.R.O.) Onlus. The funding did not influence the study process either the study findings. The funder had a role in the data collection, but not in study design, data analysis, decision to publish or preparation of the manuscript. Data Availability All relevant data are within the paper and its Supporting Information files Introduction

Serious illness conversations represent a wide range of discussions engaging healthcare professionals, patients and/or their informal caregivers when a health condition with a high risk of mortality and a negative impact on quality of life occurs or excessively strains the caregivers [1]. Serious illness conversations need a complex skillset that has been conceptualized as both a skill and as a way of being in relation to the other [2]. Indeed, such conversations occur in a context of uncertainty and turbulent emotions, and involve communication skills as well as relational skills of empathy, flexibility, and self-reflection [3]. According to their goal, serious illness conversations require competencies in delivering news, eliciting hopes and worries, sharing or clarifying the prognosis, promoting illness understanding, exploring what is important for the patient and their goals and wishes for care, making recommendations, but also in responding to emotions and offering support and reassurance [4, 5].

Although there is consensus that effective communication plays a key role in the establishment of therapeutic alliance and partnership based on mutual trust [6] and contributes to know the patients’ valued priorities and align care with their preferences [7], communication skills training is generally not part of traditional education for healthcare professionals in Italy, especially for clinicians. As a consequence, the novices who enter the profession often feel underprepared and unequipped for difficult conversations [8], and tend to postpone them as much as possible [9]. However, there is increasing awareness that communication skills are both innate and teachable [6, 10] and communication skills training is a growing component of formal professional training for healthcare professionals in other healthcare systems such as the UK, US, Canada and Australia, where programs to promote high quality communication have been introduced in the education of healthcare students and also represent a consolidated component of continuing education for professionals [11–14].

Several institutions are recognizing the worthy contribution of effective communication in avoiding aggressive, non-beneficial treatments and delivering high-quality care [15], with increasing investments in communication skills training [16, 17]. At least at a basic level, all members of the team should be confident and capable of engaging in serious illness conversations [18].

Experts in the field of communication skills recognize that didactic material can support experiential learning [10]. Printed educational resources are among the most common approaches to translate research into clinical practice tips [19]. They may facilitate clearer communication by providing a mental model of common clinical scenarios (i.e., context, goals, challenges to overcome, strategies to be used, etc.) [6, 18]. A written conversation guide offers structure for the conversation and facilitates timely, effective conversations [17, 20], while maintaining an active role of professionals [4]. Compared to other formats such as video or audio guides, written guides are more quickly accessible in clinical contexts, make it easier to revisit a particular section, and their computerized version allows one to search for keywords that make them more discoverable [19]. When conversation guides were used, professionals perceived that they had a framework for difficult conversations [17] and their satisfaction with their role increased [20].

Written conversation guides have been introduced in different clinical contexts, such as emergency departments, intensive care units, primary care, oncology, and long-term care to provide high-quality patient-centered care [12, 21, 22]. In these experiences, the original conversation guides were adapted to the care environment and the scope of practice of the healthcare professionals who would use the guide [23].

Conversation guides that are consistent with task-centered instructional strategies may be particularly helpful to improve the flow and speed of conversation [17]. The education theory of task-centered instructional strategies assumes that optimal learning occurs with focused tasks that learners are likely to encounter in their daily practice, associated with readily accessible procedural information [24]. The former makes it clear what the learner should do, while the latter provides the learner with the information they need to perform the task and decreases their cognitive load [24].

Although it is acknowledged that there is no “magic bullet” to change professional practice [25] and that the context influences the effectiveness of strategies [26], twelve key theoretical constructs have been identified to explain healthcare professionals’ behavior change [27]. In order to promote behavior change, interventions should enhance (1) knowledge and (2) skills, (3) be consistent with social/professional role and identity, (4) serve as persuasive strategies which influence beliefs about capabilities and (5) lead to changes in beliefs about consequences, (6) stimulate reflection on motivation and goals, (7) influence memory, attention, and decision processes, (8) create a facilitative environmental context that triggers behaviors, evoke (9) emotions and (10) behaviours while providing regulation, and (11) propose desirable behaviours and (12) social norms.

In the effort to maximize the benefit of an innovative communication skills training program for different health and social care professionals [28], as an integral part of the intervention, we developed an educational booklet aimed at providing professionals with guidance in challenging communication scenarios with frail or terminally-ill patients and/or their families. Unlike previous conversation guides [29, 30] that provide a talking map with a series of signposts to cover the conversation flow with no reference to clinical cases, this booklet is built on complex communication scenarios (e.g., unawareness of disease trajectory or prognosis, ambivalent care preferences) based on realistic cases. This is likely to improve learners’ memory of content because they immediately perceive support for real-case scenarios. Moreover, previous guides employ only one communication strategy; instead, having more solutions could maximize the chance for learners to find the communication strategy that best fits their learning style (e.g., short tips, infographics, or illustrative statements). Therefore, the booklet adopts several communication strategies and covers the most common and challenging communication scenarios that professionals can experience in clinical practice to help them acquire knowledge of their real-word tasks and form mental models for how integrating the skills into their performance. In fact, relevance is a pivotal component of motivation for behavioural change [31].

The primary aim of this study was to describe the development and validation of the educational booklet. The secondary aim was to assess its acceptability, usefulness, and feasibility to use in daily practice and the acquired knowledge on communication techniques perceived by professionals.

 Material and methods

This study was carried out according to a three-step approach [32] and adhered to the guidelines outlined in the Standards for QUality Improvement Reporting Excellence in EDUcation (SQUIRE-EDU) [33] (S1 Table).

The methodology included the creation of the resource, validation, and evaluation. The first step entailed a literature review, qualitative interviews, and a co-creation process to develop the booklet. The second step involved the booklet’s validation (i.e., relevance) through health and social care professionals; moreover, completeness, clarity, realism of clinical scenarios, readability, and design were assessed. The third step concerned the preliminary evaluation of the booklet by health and social care professionals working in different clinical settings. The professionals rated the acceptability, usefulness, and feasibility of using the booklet in daily practice as well as their acquired knowledge on communication techniques and protocols after reading the booklet. These steps ensured the methodological validity and reproducibility of this study (Fig 1).

10.1371/journal.pone.0304180.g001 Development, validation, and preliminary evaluation of the educational booklet: A three-step approach.

BALD, Baker Able Leaflet Design; CVI, Content Validity Index.

Participants involved in the development of the booklet provided verbal consent that was audio and video recorded, while those involved in the validation and preliminarily evaluation steps provided written consent. The study was approved by the Ethics Committee of the University of Torino (n. 0598416/2021).

First step: Development of the booklet

A first draft of the booklet covering the most common and difficult communication scenarios was created according to a review of the literature [8, 34] and interviews with health and social care professionals with varying scopes of practice (findings published elsewhere) [35]. The first draft covered nine difficult communication scenarios against four clinical cases in different contexts of care where difficult conversations are frequent: 1. Carlo, 77 years, with advanced dementia in nursing home; 2. Giovanni, 80 years, with lung cancer in intensive care unit; 3. Ottavio, 47 years, with acute lymphoblastic leukemia in hematology; 4. Anna, 85 years, transferred to long-term services after in-hospital stay for heart failure. Communication scenarios included unawareness of disease trajectory or prognosis, turbulent emotions and prolonged silences, preferences for aggressive care, ambivalent care preferences, family carers with different care preferences, choosing care treatments when the person’s preferences are unknown, unrealistic expectations, balancing hope and realism, and remote communication. Each scenario is accompanied by i) practical hints to deal with serious illness conversations; ii) infographics summarizing the main evidence-based indications; and iii) good examples of communication scripts based on the most known communication protocols (i.e., CONNECT, SPIKES) and methods (i.e., ADAPT, NURSE, PREPARED, REMAP, TUVERI). All these protocols and methods share core components such as paying attention to the context, exploring patient/family perceptions and preferences and providing information tailored on their needs, and addressing emotions with empathic responses. Each protocol/method also has a specific focus: ADAPT and PREPARED were designed to provide a talking map in discussing prognosis; the NURSE method illustrates communicative strategies for responding to emotions; the REMAP tool may be useful to address goals of care; the CONNECT protocol was designed to organize key aspects of remote communication; finally, SPIKES and TUVERI provide guidance in disclosing unfavorable information. A detailed review of the principles or content of the main communication protocols and methods is shown in S2 Table.

An introductory text, a section on basic communication skills, and additional resources complete the booklet.

Secondly, 12 health and social care professionals and three representatives of patients associations, family carers, and volunteers in the field of dementia or palliative care were recruited between 5 September 2022 to 26 October 2022 and participated in three focus group discussions (5 participants for each focus group). The research team reached the reference person of each association that then identified their representative. Characteristics of the participants to focus groups are shown in S3 Table.

Seven to ten days prior to the focus group, each participant received the booklet by email and was asked to read it for sharing ideas on the content during the group discussion. The focus groups were conducted digitally to facilitate participation from different geographical areas [36] and had a mean duration of 78 minutes (range 68–85). A member of the research team moderated the discussion using a pre-defined set of questions and solved conflicts of opinion by seeking common ground and areas of agreement to compromise. The questions guiding the focus groups explored the completeness, acceptability, clarity, feasibility to use, and graphics of the booklet, that the literature recognizes as key features for written resources [19]. Another team member took any field notes on paraverbal and body language. Focus group discussions were audio-recorded, transcribed verbatim and transcripts checked for accuracy. Then, two researchers used an inductive thematic analysis to analyse the transcripts and identify the areas of improvement [37]. Thirdly, this analysis guided the changes of the second iteration of the booklet, which was then distributed back for a second evaluation with the request to complete a brief open-ended questionnaire assessing its content and design. Fourthly, the booklet was further adjusted according to the emerging comments.

The research team initially took the decision not to involve a design agency and to deal on their own with design issues that could have arisen. However, the team later realized that it was not possible to address satisfactorily criticalities about font, colors, and layout as they recurred on the unfolding versions of the booklet despite the efforts to solve the problems; therefore, a graphic designer was involved after the feedback from the external review board.

 Second step: Validation Content validity index

An external review board of 12 health and social care professionals who work in the community, long-term, hospital, and education settings and three representatives of two stakeholder societies reviewed the booklet and answered an online, 10-minute questionnaire from 17 January 2023 to 17 February 2023. Characteristics of the board are shown in the S4 Table.

The professionals received the booklet by email and were invited to rate each section according to its clarity, coherence, and relevance. Moreover, the booklet was explored for completeness and missing information. Clarity, coherence, and relevance of each section were assessed using a 4-point Likert scale (from 1 = not at all to 4 = much). The completeness was based on a dichotomous answer (yes/no) with the opportunity to list missing information. Free comments were also invited.

Regarding the relevance of the booklet, the content validity index (CVI) was computed for individual sections (I-CVI) and for the overall booklet (S-CVI/averaging calculation method (Ave) and S-CVI/Universal Agreement (UA)) [38].

The I-CVI was calculated for each section as the number of professionals giving a rating of either 3 or 4 (thus relevant), divided by the total number of professionals. With more than 10 validators, the I-CVI should be >0.70 [38]. Only sections scoring higher than this threshold were maintained [38].

The S-CVI/Ave was computed as the average of the I-CVIs for all sections on the booklet. A S-CVI/Ave of 0.90 or higher is recommended. The S-CVI/UA was computed as the proportion of sections of the booklet that achieved a relevance rating of 3 or 4 by all the validators. A S-CVI/UA of 0.80 or higher is recommended [38].

The clarity and coherence of each section and for the overall booklet were computed as mean with standard deviation (SD). The completeness of the booklet was computed as the number of validators who rated the booklet as complete on the total.

When a section was marked as ‘not at all’ = 1 or ‘little’ = 2 for relevance, clarity, or realism of clinical scenarios by at least two validators, the text was adjusted according to their comments.

 Product testing

The readability, layout, and design of the final booklet were evaluated. The Gulpease Index was used to assess the readability [39]. The Gulpease index considers two linguistic variables, the length of the word and the number of letters of the sentence, as shown in the following formula: 89+300*(mumberofsentences)−10*(numberofletters)numberofwords [39]. Accounting for the length of a word in characters rather than in syllables, it is proved to be more reliable in assessing the readability of Italian texts. The index ranges from 0 (lowest readability) to 100 (maximum readability). Scores below 80, 60 and 40 identify texts that are difficult to read for a 5th grade level (primary school level, 6 to 10 years), 8th grade level (junior secondary school level, 11 to 13 years), and 13th grade level (secondary school level, 14 to 18 years) [40]. Each section of the booklet was tested at the following address: https://farfalla-project.org/readability_static/.

The Baker Able Leaflet Design (BALD) criteria were used to assess the layout and design characteristics of the booklet [41]. The BALD score covers 16 attributes and ranges from 0 (worst layout and design) to 32 (best layout and design).

 Third step: Preliminary evaluation Recruitment of health and social care professionals

Health and social care professionals were recruited at a 800-bed acute care hospital and its community services and at two nursing homes in the North-west of Italy from 10 May 2023 to 15 May 2023. Professionals of any profile engaged in serious illness conversations with patients or their families were eligible.

 Questionnaire

The professionals were asked to complete a questionnaire that explored their perception on the acceptability, usefulness, and feasibility of using the booklet in daily practice. Moreover, they were asked to rate their acquired knowledge on communication techniques and protocols and could suggest changes. Finally, professionals were asked whether they would recommend the booklet to colleagues (yes/not) and to motivate it.

Acceptability is referred to by O’Connor and Cranney as ‘‘ratings regarding the comprehensibility of components of a decision aid, its length, amount of information, balance in presentation of information about options, and overall suitability for decision making” [42]. Our measure of ‘acceptability’ refers more specifically to acceptability to use, understand, read, and inform patients and their families. It also includes the themes of communication with patients/families, relationships with patients/families, and intention to use the booklet in practice. For this, we developed a 10-item scale summing rating of agreement with statements selected from the acceptability instrument by O’Connor and Cranney [42] and from a comparative evaluation of three decision aids [43]. Item scores were 1 (strongly disagree) to 5 (strongly agree) [42]. Higher total scores represent better acceptability. We regarded scores of 30 and higher as “acceptable” and 40 and higher as “highly acceptable” (mean item scores 3 and 4, respectively).

Usefulness and feasibility to use were rated on a scale from 1 (not at all useful/feasible to use) to 10 (extremely useful/feasible to use). The mean usefulness and feasibility scores were calculated; a score between 6 and 7 indicated moderate usefulness/feasibility, while a score >8 a high degree of usefulness/feasibility. Acquired knowledge was rated on a 5-point Likert scale from 1 (not at all) to 5 (extremely).

 Statistical analyses

Frequencies, percentages, and means with range or standard deviation (SD) were computed. IBM SPSS Statistics for Window Version 27.0 was employed. The sample size for the evaluation step was based on the concept of content validity and was calculated using the formula: n=z2P(1−P)e2, where ‘P’ is the expected proportion of health and social care professionals who indicated the relevance of each section of the booklet and ‘e’ represents the acceptable proportional difference compared to what would be expected [44]. An expected proportion of professionals of 85% was established with a difference of 15% and a confidence level of 95%. This results in a minimum of 22 professionals being needed [44]. This analysis is based on the methodology for the evaluation of decision aids. Decision aids and educational booklets share several similarities, including the multidisciplinary approach, evidence-based soundness, and attention to both content and design. However, some differences also occur: decision aids explicitly state the decision that needs to be considered and appear as active tools to guide the decision-making process by providing detailed focus on options and outcomes and helping people to arrive at decisions that reflect their personal values and preferences. Instead, educational booklets represent a source of information but are not focused on decision points due to their broader perspective [45].

 Results First step: Development of the booklet

The inductive thematic analysis of focus group transcripts identified eight areas for improvement: 1) Make clearer that the booklet is part of a communication skills training program; 2) Strengthen the relational component of communication; 3) Enhance the realism of clinical scenarios; 4) Draw attention to the legal issues; 5) Insert missing critical situations; 6) Employ a person-centered dialogical approach; 7) Enhance actionability; and 8) Improve graphics.

Further refinements came after re-evaluation by focus group participants and external reviewers, but without major changes.

Table 1 shows the main adjustments of the booklet during its development.

10.1371/journal.pone.0304180.t001 Adjustments of the booklet during the process of development.
Sections of the booklet First draft Second draft Third draft Final version
Title Communicating at the end of life: A practical guide for healthcare professionals dealing with difficult communications Communicating in complex situations: A practical guide for health and social care professionals No additional changes. No additional changes.
Introduction Short introduction (226 words) that describes the targeted audience and aim of the booklet, benefits of effective communication and feasibility to learn communication skills and acquire communication competence. Longer introduction (905 words) that better frames the booklet as a supportive tool promoting awareness, empowerment, accountability, and reflection on communication in complex situations. The relational aspect of communication has been valorized. The role of experience is recognized as well as the impossibility to squeeze communication in strict protocols that rather provide mental models to tailor the interaction according to the specific situation. It has been further clarified that this booklet does not cover the complex and multiple nuances of the reality but rather it is a starting point for effective communication where experience and interpersonal skill are put at stake. Redundancies across communication protocols have been noted. This led to clarify that protocols are largely overlapping for the main core contents and the choice of which protocol to adopt depends on personal preferences.
Basic communication skills The main basic communication skills and techniques (e.g., open questions, paraphrasing, summarizing, “tell me more about, “I wish”) are presented with examples. Didactic and textbook examples have been replaced with expressions drawn from the real experience of the professionals. No further changes. This section was judged too technical with little attention to the relational aspect. Therefore, the need to prepare the setting of communication as well as the pivotal role of team collaboration and the need for professionals to be physically, emotionally, and psychologically mindful has been emphasized.
Difficult communication scenarios Nine scenarios against four clinical cases (Carlo, Ottavio, Giovanni, and Anna) in different settings of care.Each scenario is accompanied by i) good examples of communication scripts based on the most popular communication protocols (i.e., CONNECT, PREPARED, REMAP, SPIKES) and methods (i.e., ADAPT, NURSE, TUVERI); ii) practical hints to deal with serious illness conversations; and iii) an infographic summarizing the main evidence-based indications. Each communication scenario now opens with an evocative sentence to convey the message that reality is complex. The realism of dialogues has been improved replacing didactic and textbook sentences with expressions drawn from the real experience of the professionals. For example, to explore family carers’ perspective about the clinical conditions of their beloved, the original sentence “Could you explain better what you mean when you say you didn’t expect it?” was changed into “I understand that you were not prepared for this worsening of your husband. Is there anything you would like to ask me?”. To expect emotion and empathize, the original sentence “I can imagine how you feel and how it’s difficult having to make a decision like this. What worries you? Would you like to talk about it?” was changed into “I realize this news has shocked you and it’s difficult having to make a decision like this. My colleagues and I are available if you would like to talk about it”. To promote reflection on care choices, the original sentence “We have given you a lot of information. If you agrees, we would continue the discussion tomorrow also with your mother and your sister” was changed into “You have received a lot of information. It would be important to come together with a decision that guarantees the best interest of your father and what he would have wanted without involving the judge supervises cases concerning guardianship”. To summarize the key points discussed, the original sentence “Could you try repeating in your own words what we told you?” was changed into “Before closing the encounter, if you agree, we would like to try to summarize the main points we discussed…”.An extensive rewording consistent with a person-centered approach has been performed to highlight that communication is a mutual dialogue and professionals have to bring out preferences, welcome, support and establish trusting relationships.Legal issues have been emphasized by clarifying the role of the patient, their family carers, and professionals in the decision-making process, as well as the need to document the process of communication beyond its outcomes.A sub-section listing other possible complex communication scenarios has been added to further stress that the booklet is focused on the most common ones. Some sentences regarding the role of patients and their family carers in taking care decisions have been reworded and the necessity to involve the patient in the decision-making process, if possible, has been highlighted. The order of communication scenarios in Carlo’s case ‐ unawareness of disease trajectory or prognosis, turbulent emotions and prolonged silences, preferences for aggressive care, family carers with different care preferences, choosing care treatments when the person’s preferences are unknown- was modified by shifting “choosing care treatments when the person’s preferences are unknown” between “turbulent emotions and prolonged silences” and “preferences for aggressive care” to reflect a temporal logic.Also, in the scenario “family carers with different care preferences”, it has been clarified the role of family carers in care decisions when the relative is no more cognitively competent according to the Italian law by adding the following sentence “This is a decision that awaits us healthcare professionals, you don’t have the responsibility for the choice. We ask you to help us understand what your father would have wanted”.Each check-list has been associated with a QR code which redirects to the desired information when smartphones and tablets frame the code.
Additional resources This section consists of four sub-sections: i) Running the family conferences; ii) Summary of the infographics for each communication scenario; iii) Flow charts summarizing the process of communication in presence and at distance; and iv) Communication protocols and communication methods. A person-centered approach has been adopted in the flow charts by reporting first extracts of dialogues from communication scenarios of the previous section and then referring to the communication protocol(s). Also, the concept of “documenting the process of communication” has been introduced in the flow charts.A further sub-section on times and manners of communication has been added. No further changes. No further changes.
Actionable resources Absent The summary of the infographics has been moved from the “Additional resources” section to a newly-added “Actionable resources” section. The check-lists appear as tear-away, ready-to-use worksheets. No further changes. No further changes.
Layout and design Different fonts and font size (range 10 to 14) for check lists, communication scripts, and narrative text.Abstract or stylized images.White background with leaves-based pattern. Increased font size across the entire booklet.Abstract and stylized images substituted with pictures highlighting the importance of human relationships.Light yellow background with removal of the leaves-based pattern.Page numbers added. Further increase in the font size and change of two images. Three members of the external review board highlighted some criticalities about font, colors, and layout therefore a graphic designer was involved to develop a better and user-friendly version of the booklet.

The final version of the booklet is freely available online and is downloadable from the website of the Italian Society of Palliative Care at https://www.sicp.it/aggiornamento/linee-guida-bp-procedures/2023/09/guida-alla-comunicazione-in-situazioni-complesse/

 Second step: Validation Content validity index

All external reviewers deemed the booklet complete with an overall mean (SD) clarity and coherence of 3.7 (0.5) and 3.9 (0.3), respectively. The final booklet resulted into 17 sub-sections and none was removed based on I-CVI always above 0.9. No differences emerged between professional raters and association representatives. The S-CVI/Ave was 0.97 and the S-CVI/UA was 0.53. The S5 Table shows the details for each sub-section.

 Product testing

The mean Gulpease index of all sub-sections was 60. This represents a language fairly easy to read and understand at a junior secondary school level. The ‘Introduction’ sub-section scored the lowest with 44. The ‘Actionable resources’ subsection scored the highest with 71, followed by ‘Basic communication skills’ with 70 (S5 Table).

The BALD score for the final version of the booklet was 25/32. The S6 Table shows the detailed design characteristics of the booklet.

 Third step: Preliminary evaluation Participants

Twenty-two professionals were contacted and all answered the evaluation questionnaire. The majority were females and the over 35 to under 35 years ratio was roughly equal. More than two thirds (n = 15) had more than 10 years of working experience with only three professionals with less than one year in the current service. Most were nurses, however, the profiles of physician, social worker, occupational therapist, and physiotherapist were also represented. The professionals worked in varying settings of care, including emergency, nursing home, medicine, surgery, primary care, and hospice. Only three professionals undertook specific education in clinical communication or counselling (Table 2).

10.1371/journal.pone.0304180.t002 Characteristics of professionals evaluating the educational booklet (n = 22).
Participants characteristics (n = 22) N
Female gender 18
Age, years
 ≤ 25 2
 26–35 8
 36–50 9
 > 50 3
Overall working experience, years
 1–4 3
 5–10 4
 11–15 5
 > 15 10
Experience in the current service, year
 < 1 3
 1–4 9
 5–10 4
 11–15 2
 > 15 4
Professional profile
 Nurse 17
 Physician 2
 Social worker 1
 Occupational therapist 1
 Physiotherapist 1
Setting of care
 Critical care/Emergency 6
 Nursing home 5
 Medicine 4
 Surgery 3
 Primary care 3
 Hospice 1
Work relationship
 Full-time 19
 Part-time 3
Education/training courses in clinical communication or counselling 3
Questionnaire

The evaluations for acceptability, usefulness, feasibility to use, and perceived acquired knowledge are presented in Table 3. In all, 18 (81.8%), 19 (86.4%) and 17 (77.3%) professionals deemed the booklet acceptable, moderate to highly useful, and moderate to highly feasible to use, respectively. Over 80% (n = 18) of the professionals perceived gain in knowledge after reading the booklet.

10.1371/journal.pone.0304180.t003 Acceptability, usefulness, feasibility to use and perceived acquired knowledge.
Evaluations of the booklet N (%)
N = 22
Acceptabilitya, mean (SD) 38 (7)
 ≥ 30 18 (81.8)
 ≥ 40 12 (54.5)
Usefulnessb, mean (SD) 7.6 (1.8)
 6–7 6 (27.3)
 ≥ 8 13 (59.1)
Feasibility to useb, mean (SD) 6.9 (1.6)
 6–7 9 (40.9)
 ≥ 8 8 (36.4)
Perceived acquired knowledgec
 Not at all 1 (4.5)
 Little 3 (13.6)
 Moderately 12 (54.5)
 Very 5 (22.7)
 Extremely 1 (4.5)

a Range 10 to 50. Themes addressed: usability, understandability, readability, guidance in providing information to patients and families, communication with patients/families, relationships with patients/families, and intention to use in practice. A score of 30 and higher was deemed “acceptable” and a score of 40 and higher “highly acceptable” (mean item scores 3 and 4, respectively).

b Scale from 1 (not at all useful/feasible to use) to 10 (extremely useful/feasible to use). A score between 6 and 7 indicated moderate usefulness/feasibility, while a score of 8 or above indicated a high degree of usefulness/feasibility.

c 5-point Likert scale from 1 (not at all) to 5 (extremely).

Table 4 shows the acceptability score at the item level: particularly, professionals perceived the booklet as consistent with their professional identity and standard by agreeing that it aligns with the way they think things should be done (15/22) and complements their usual approach to communication (14/22). All but three professionals perceived a high probability that using this booklet may result in more benefit than harm, and half believed that the booklet would improve communication and relationships between professionals and families. Around 60% (13/22) reported that they would continue to use the resource after the end of the project.

10.1371/journal.pone.0304180.t004 Acceptability of the booklet: Scoring at the item level by the 22 evaluating professionals.
Items ScoreMean (SD)* Number indicating 4 or 5 on five-point agreement scale*N (%)
1. The booklet is easy to use 3.7 (1.0) 11 (50)
2. The booklet is easy to understand 3.9 (0.8) 15 (68.2)
3. The booklet is easy to read 4.0 (0.9) 18 (81.8)
4. The booklet provides guidance that aligns with the way I think things should be done 3.8 (1.0) 15 (68.2)
5. The booklet is a reliable strategy to enhance communication with patients and their families 3.9 (0.9) 15 (68.2)
6. The booklet complement my usual approach to communicate with patients and their families 3.7 (0.9) 14 (63.6)
7. The booklet will enhance communication between professionals and patients/families 3.5 (0.9) 11 (50)
8. The booklet will improve the relationship between professionals and patients/families 3.5 (0.9) 11 (50)
9. There is a high probability that using this booklet may cause/result in more benefit than harm 4 (0.8) 19 (86.4)
10. I will continue to use this booklet even after the project is completed 3.6 (0.7) 13 (59.1)

*Each item scores from 1 (strongly disagree) to 5 (strongly agree)

All professionals would recommend the booklet to colleagues and 11 reported reasons: “provides practical and easily applicable advice” (n = 6), “provides a mental model and guidance in real-case scenarios” (n = 3), “helps start conversations” (n = 1), and “promotes reflection on professional competence and learning needs” (n = 1). Only one professional suggested enriching the bibliography to improve scientific soundness.

 Discussion

This study describes the development, validation, and preliminarily evaluation of an educational booklet aimed at supporting Italian professionals in serious illness conversations. The co-creation process led to an acceptable resource with valid content, high degree of perceived usefulness, feasibility to use, and acquired knowledge.

The co-creation process for developing the booklet relied on focus groups involving a purposive sample of stakeholders. Focus groups represent a person-centered research method that provides the opportunity to discuss own perspective in a collaborative, participatory setting and could be helpful in obtaining viewpoints that are representative of the majority [46]. To develop the educational booklet, the academics strictly worked with stakeholders from other sectors, including professionals with varying scope of practice, representatives of volunteering associations, and associations of patients and family carers with life-limiting illnesses. The iterative, user-centric approach with several rounds of revisions promoted the creation of a person-centered resource tailored on day-to-day practice challenges that is responsive to real life scenarios [35]. For example, in Carlo’s case to promote reflection on care choices, the original, professionals-centric sentence “we have given you a lot of information” was changed into the person-centric statement “you have received a lot of information”. Moreover, the user-centric approach allowed to improve realism by replacing didactic sentences with real-life expressions. Although several communication protocols suggest closing the encounter by making patients/family carers summarize the main key points discussed, all participants in our focus groups agreed that this is usually performed by the professionals to avoid further pressure. Therefore, the original sentence “could you try repeating [‥]” was changed into “we would like to summarize […]”.

As suggested by previous literature on participatory research [47, 48], the co-creation process ensured that the booklet was relevant to the users’ priorities and needs and resulted in all professionals recommending the resource to colleagues for its practicality and support in real-case scenarios. The involvement of meaningful stakeholders in quality improvement initiatives is recommended to develop innovative clinical resources aimed at improving practice [49, 50]. An active engagement of potential users in the design of the resources can affect their behavior change more positively, favour the integration of resources in daily practice and is finally more likely to improve health-related outcomes and public health [51].

The I-CVI always above 0.9 and the S-CVI/Ave of 0.97 indicate the high degree of content information validity. The target of a S-CVI/UA above 0.80 was not reached [38]; this was somewhat expected due to the large number of external validators. However, the involvement of 15 external validators despite the fact that more than ten are probably unnecessary [52] was justified by the need to include professionals with different scope of practice who work in a wide array of clinical settings. This approach wants to sustain the core message that serious illness conversations are ubiquitous across several care settings and require a multidisciplinary team [53–55]. Additionally, only two experts deemed some sections not relevant (one 6/17 and one 2/17) (S5 Table).

The booklet performed well in the product testing of readability and design that employed internationally accepted methods -Gulpease index and BALD criteria, respectively-. In general, an increase in readability leads to a shorter reading time, better memory of the content, and greater text comprehension [56]. Our booklet showed good readability (Gulpease index of 60), was perceived easy to read by over 80% (18/22) of the professionals and 15 (68%) agreed on high understandability. Gray and Leary identified four basic elements that influence the ease of reading: content, style, structure, and design [57]. Design should not be overlooked since sound design principles can help users to understand and use information [58, 59]. Consultation workshops arranged for the development of a resource to support person-centered conversations strongly recommend avoiding an arbitrary design that would reduce the usefulness and uptake among professionals [60]. Multiple adjustments were made to the design and layout of the booklet and a design agency was engaged to improve the production of quick-reading and visual-friendly solutions.

An ever-present barrier to serious illness conversations is the lack of time [34, 54, 61, 62] and professionals need to quickly identify information that answers specific communication challenges. A resource designed to support clinical practice must address this obstacle. In our booklet, each communication scenario is accompanied by practical hints, an infographic, and an example of good interaction. This reflects a time-based approach on conversations templates of 1, 2, and 5 minutes, respectively, according to the amount of time professionals have.

Most professionals perceived the booklet as useful (19/22) and gain in knowledge on how engaging in serious illness conversations (18/22), despite the generally long working experience. This suggests that though experience may be a valid aid in sustaining challenging conversations, it is not enough. An international study involving professionals who work in long-term settings across six European countries found that Italy scored the worst in self-efficacy regarding end-of-life communication [63]. These conversations are emotional and professionals need training and resources to acquire confidence; our booklet is the first resource in Italian language that offers professionals support.

The booklet strives for behaviour change and supports professionals overcoming individual-related barriers and timely initiating these conversations. Our resource is well-placed to promote changes in professional practice as suggested by addressing all the 12 domains identified to explain behavior change [27]. The booklet enhances knowledge to engage successfully in serious illness conversations, provides guidance for real-life difficult communication scenarios while ensuring readability and appealing design the positively influence the memory of the content, is consistent with professional identity and standard, and influences beliefs about capabilities and outcomes with most professionals reporting they would continue to use the resource and perceiving benefits in communication and relationships with families. Additionally, the booklet may stimulate reflection on professional competence and learning needs, enhance sensitivity on the topic of quality and timely communication, which triggers change in behavior, evoke emotions and behaviors related to previous communication experience while providing functional tools for their management, and propose desirable behaviors or instigate a process of social influence.

The booklet supports professionals to sustain serious illness conversations that mirror their most frequent challenges in daily practice and for each scenario offers 1-minute, 2-minute and 5-minute solutions. Communication scenarios are relevant to the professionals’ learning needs and priorities to prompt efforts in memorizing information and applying learning that is perceived relevant. Moreover, the opportunity to choose the template that best fits own routine increases the use of the booklet. The easily readable information and the user-friendly design promote the understandability of the booklet content. This helps professionals become more knowledgeable about how engaging in serious illness conversations and supports behavior change to improve practice and patient care [19]. The resource stimulates reflection on professional competence, learning needs, and motivation to enhance skills with reference to the professional standard. The high content information validity while employing communication scenarios set in multiple care settings (i.e., nursing home, oncology, and intensive care unit), as well as the involvement of professionals with different scope of practice and who work in different contexts across the development, validation, and evaluation of the booklet, suggests that the resource may be potentially useful irrespective of the professional profile and the setting of care. Finally, the collaboration between researchers, professionals, and lay stakeholders and the testing of the booklet using validated tools ensured the development of a resource of increased quality and utility.

The booklet presents a limited number of advanced chronic or terminal illnesses, but addresses the most frequent communication scenarios across different clinical settings and offers practical tips and examples of communication protocols that can be tailored to different chronic conditions. Moreover, issues related to communication with adults about an infant or child’s illness are not addressed. Parents are often scared, feel guilty, and may have difficulties retaining and processing information. When communicating to parents, professionals have to listen authentically, avoid judgement and act empathically to capture their history, values, perspectives and peculiarities that make one family different from another. Parents need to be supported and accompanied in the experience of their child’s illness, beyond receiving information that should be tailored with regard to content and modalities. Therefore, a specific resource to support communication in this setting is warranted. Secondly, the involvement of the design agency occurred late in the development process. Its integration from the outset of the project could have eased addressing the feedbacks and translating them into solutions. However, the design agency had received information on the objectives of the booklet and the main critiques emerged during its development, and the research team provided ongoing supervision. Thirdly, the evaluation of the booklet was mainly quantitative in nature, with professionals reporting ratings on acceptability, usefulness, feasibility of use, and acquire knowledge; despite professionals had the opportunity to report open comments and suggest changes, qualitative work could have contributed more deeply to identify nuances that benefit from improvements.

 Conclusions

The booklet developed helps address the lack of support tools on serious illness conversations for Italian professionals and could potentially improve the quality of serious illness conversations.

The co-creation process yielded an acceptable resource with good readability and excellent design, high content validity, and a high degree of perceived usefulness and acquired knowledge. Also, the time-based solutions favor the use of the resource in daily practice. All these features make the booklet ideally positioned to encourage changes in professional practice and a valid resource to enhance shared decision-making and care planning. In addition, the booklet has good potential for a wider distribution, as it is available on the website of a high-profile Italian society of palliative care and can be downloaded for free. Finally, translation to English and social media campaigns beyond congress presentations are planned to promote dissemination.

 Supporting information Standards for Quality Improvement Reporting Excellence for Education (SQUIRE-EDU).

(PDF)

 Review of the principles or content of the main communication protocols and methods.

(PDF)

 Characteristics of the participants to the focus groups (n = 15).

(PDF)

 Characteristics of the external review board (n = 15).

(PDF)

 Clarity, coherence, relevance, and readability for each sub-section of the booklet.

(PDF)

 Layout and design characteristics of the booklet based on the Baker Able Leaflet Design (BALD) criteria.

(PDF)

 <p>(TIF)</p> </caption> </supplementary-material> </sec> </body> <back> <ack> <p>We would like to express our gratitude to all professionals and associations engaged in the development, validation, and evaluation of the booklet for their cooperation. We thank Ylenia Arnone and Alessio Turrini for their contribution in developing the booklet content, Alexandra Antal and Valentino Bissacca for their help in validating the resource, and Luisa Goglio for revising its design. Finally, we thank Dr. Nicola Cornally for her methodological guidance across the development and validation processes.</p> </ack> <ref-list> <title>References Kelley AS, Bollens-Lund E. Identifying the Population with Serious Illness: The "Denominator" Challenge. J Palliat Med. 2018;21(S2):S7–s16. doi: 10.1089/jpm.2017.0548 29125784 Zoppi K, Epstein RM. Is communication a skill? Communication behaviors and being in relation. Fam Med. 2002;34(5):319–24. 12038712 Browning DM, Meyer EC, Truog RD, Solomon MZ. Difficult conversations in health care: cultivating relational learning to address the hidden curriculum. Acad Med. 2007;82(9):905–13. doi: 10.1097/ACM.0b013e31812f77b9 17726405 Daubman BR, Bernacki R, Stoltenberg M, Wilson E, Jacobsen J. Best practices for teaching clinicians to use a serious illness conversation guide. Palliat Med Rep. 2020;1(1):135–42. doi: 10.1089/pmr.2020.0066 34223467 Geerse OP, Lamas DJ, Sanders JJ, Paladino J, Kavanagh J, Henrich NJ, et al. A Qualitative Study of Serious Illness Conversations in Patients with Advanced Cancer. J Palliat Med. 2019;22(7):773–81. doi: 10.1089/jpm.2018.0487 30724693 Prineas S, Mosier K, Mirko C, Guicciardi S. Non-technical Skills in Healthcare. In: Donaldson L, Ricciardi W, Sheridan S, Tartaglia R, editors. Textbook of Patient Safety and Clinical Risk Management. Cham: Springer International Publishing; 2021. p. 413–34. Curtis JR, Downey L, Back AL, Nielsen EL, Paul S, Lahdya AZ, et al. Effect of a patient and clinician communication-priming intervention on patient-reported goals-of-care discussions between patients with serious illness and clinicians: a randomized clinical trial. JAMA Intern Med. 2018;178(7):930–40. doi: 10.1001/jamainternmed.2018.2317 29802770 Travers A, Taylor V. What are the barriers to initiating end-of-life conversations with patients in the last year of life? Int J Palliat Nurs. 2016;22(9):454–62. doi: 10.12968/ijpn.2016.22.9.454 27666307 Gonella S, Basso I, Clari M, Dimonte V, Di Giulio P. A qualitative study of nurses’ perspective about the impact of end-of-life communication on the goal of end-of-life care in nursing home. Scand J Caring Sci. 2021;35(2):502–11. doi: 10.1111/scs.12862 32343871 Stiefel F, Kiss A, Salmon P, Peters S, Razavi D, Cervantes A, et al. Training in communication of oncology clinicians: a position paper based on the third consensus meeting among European experts in 2018. Ann Oncol. 2018;29(10):2033–6. doi: 10.1093/annonc/mdy343 30412225 Beddard-Huber E, Fyles G, Tayler C, Barwich D, Hassan E, Ma SD. Implementation of the serious illness conversation program–lessons learned from a canadian provincial health system. 2020. Available from: https://www.bc-cpc.ca/wp-content/uploads/2021/05/BCCPC-Implementation-of-the-Serious-Illness-Conversation-Program.pdf Bernacki R, Paladino J, Neville BA, Hutchings M, Kavanagh J, Geerse OP, et al. Effect of the serious illness care program in outpatient oncology: a cluster randomized clinical trial. JAMA internal medicine. 2019;179(6):751–9. doi: 10.1001/jamainternmed.2019.0077 30870563 McGlinchey T, Mason S, Coackley A, Roberts A, Maguire M, Sanders J, et al. Serious illness care Programme UK: assessing the ‘face validity’, applicability and relevance of the serious illness conversation guide for use within the UK health care setting. BMC Health Services Research. 2019;19:1–9. doi: 10.1186/s12913-019-4209-8 31196072 White SJ, Condon B, Ditton-Phare P, Dodd N, Gilroy J, Hersh D, et al. Enhancing effective healthcare communication in Australia and Aotearoa New Zealand: Considerations for research, teaching, policy, and practice. PEC Innov. 2023;3:100221. doi: 10.1016/j.pecinn.2023.100221 37822775 Gonella S, Basso I, Dimonte V, Martin B, Berchialla P, Campagna S, et al. Association between end-of-life conversations in nursing homes and end-of-life care outcomes: a systematic review and meta-analysis. J Am Med Dir Assoc. 2019;20(3):249–61. doi: 10.1016/j.jamda.2018.10.001 30470575 Borghi L, Meyer EC, Vegni E, Oteri R, Almagioni P, Lamiani G. Twelve Years of the Italian Program to Enhance Relational and Communication Skills (PERCS). Int J Environ Res Public Health. 2021;18(2). doi: 10.3390/ijerph18020439 33429873 Paladino J, Kilpatrick L, O’Connor N, Prabhakar R, Kennedy A, Neal BJ, et al. Training clinicians in serious illness communication using a structured guide: evaluation of a training program in three health systems. J Palliat Med. 2020;23(3):337–45. doi: 10.1089/jpm.2019.0334 31503520 Shi S, Keeney T, Ritchie C. Advancing Communication and Decision-making for Older Adults Discharged to Skilled Nursing Facilities-Not Where but Why? JAMA Intern Med. 2022;182(11):1115–6. doi: 10.1001/jamainternmed.2022.4080 36121670 Giguère A, Zomahoun HTV, Carmichael PH, Uwizeye CB, Légaré F, Grimshaw JM, et al. Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2020;8(8):Cd004398. doi: 10.1002/14651858.CD004398.pub4 32748975 Paladino J, Koritsanszky L, Nisotel L, Neville BA, Miller K, Sanders J, et al. Patient and clinician experience of a serious illness conversation guide in oncology: a descriptive analysis. Cancer medicine. 2020;9(13):4550–60. doi: 10.1002/cam4.3102 32363775 Lakin JR, Koritsanszky LA, Cunningham R, Maloney FL, Neal BJ, Paladino J, et al. A systematic intervention to improve serious illness communication in primary care. Health Aff (Millwood). 2017;36(7):1258–64. doi: 10.1377/hlthaff.2017.0219 28679813 Lamas DJ, Owens RL, Nace RN, Massaro AF, Pertsch NJ, Moore ST, et al. Conversations about goals and values are feasible and acceptable in long-term acute care hospitals: a pilot study. J Palliat Med. 2017;20(7):710–5. doi: 10.1089/jpm.2016.0485 28437209 Aaronson EL, Greenwald JL, Krenzel LR, Rogers AM, LaPointe L, Jacobsen JC. Adapting the serious illness conversation guide for use in the emergency department by social workers. Palliat Support Care. 2021;19(6):681–5. doi: 10.1017/S1478951521000821 34140064 Merrill MD. A task-centered instructional strategy. Journal of research on Technology in education. 2007;40(1):5–22. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Cmaj. 1995;153(10):1423–31. 7585368 Wensing M, van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract. 1998;48(427):991–7. 9624774 Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33. doi: 10.1136/qshc.2004.011155 15692000 Gonella S, Di Giulio P, Berchialla P, Bo M, Cotogni P, Macchi G, et al. The Impact of Health and Social Care Professionals’ Education on the Quality of Serious Illness Conversations in Nursing Homes: Study Protocol of a Quality Improvement Project. Int J Environ Res Public Health. 2022;20(1). doi: 10.3390/ijerph20010725 36613049 Ariadne Labs: A Joint Center for Health Systems Innovation and Dana-Farber Cancer Institute. Serious Illness Conversation Guide. 2015, upadate May 2023. Available at https://www.ariadnelabs.org/wp-content/uploads/2023/05/Serious-Illness-Conversation-Guide.2023-05-18.pdf Tinetti M, Naik A, Dindo L. Conversation guide and manual for identifying patients’ health priorities. 2018. Available at https://patientprioritiescare.org/wp-content/uploads/2018/11/Conversation-Guide-and-Manual-for-Identifying-Patients27-Health-Priorities.pdf Keller JM, Litchfield BC. Motivation and performance. Trends and issues in instructional design and technology. 2002;2:89–92. Giannopoulos P, Mertens YJ, Secomandi LE, Olsder L, van Leeuwen BL. Development, validation and evaluation of a patient information booklet for rectal cancer survivors with a stoma: A three-step approach. Patient Educ Couns. 2021;104:2275–85. doi: 10.1016/j.pec.2021.02.045 33685763 Ogrinc G, Armstrong GE, Dolansky MA, Singh MK, Davies L. SQUIRE-EDU (Standards for QUality Improvement Reporting Excellence in Education): publication guidelines for educational improvement. Acad Med. 2019;94(10):1461–70. doi: 10.1097/ACM.0000000000002750 30998575 De Vleminck A, Pardon K, Beernaert K, Deschepper R, Houttekier D, Van Audenhove C, et al. Barriers to advance care planning in cancer, heart failure and dementia patients: a focus group study on general practitioners’ views and experiences. PLoS One. 2014;9(1):e84905. doi: 10.1371/journal.pone.0084905 24465450 Gonella S, Di Giulio P, Antal A, Cornally N, Martin P, Campagna S, et al. Challenges experienced by italian nursing home staff in end-of-life conversations with family caregivers during covid-19 pandemic: a qualitative descriptive study. Int J Environ Res Public Health. 2022;19(5). doi: 10.3390/ijerph19052504 35270195 Flynn R, Albrecht L, Scott SD. Two approaches to focus group data collection for qualitative health research: maximizing resources and data quality. Int J Qual Methods. 2018;17(1):1609406917750781. doi: 10.1177/1609406917750781 Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3(2):77–101. Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29(5):489–97. doi: 10.1002/nur.20147 16977646 Lucisano P, Piemontese ME. GULPEASE: una formula per la predizione della difficoltà dei testi in lingua italiana. Scuola e città. 1988;3(31):110–24. Tonelli S, Mahn KT, Pianta E. Making readability indices readable. In: Williams S, Siddharthan A, Nenkova A, editors. Proceedings of the first workshop on predicting and improving text readability for target reader populations. Canada: Association for computational linguistics; 2012, p. 40–8. Baker SJ. Who can read consumer product information? The Australian Journal of Hospital Pharmacy. 1997;27(2):126–31. O’Connor AM, Cranney A: Patient decision aids–Evaluation measures. User manual for acceptability. Ottawa, Canada: University of Ottawa, Ottawa Hospital Research Institute, 2002. https://decisionaid.ohri.ca/docs/develop/user_manuals/um_acceptability.pdf (Last accessed September 30, 2022). Graham ID, Logan J, Bennett CL, Presseau J, O’Connor AM, Mitchell SL, et al. Physicians’ intentions and use of three patient decision aids. BMC Med Inform Decis Mak. 2007;7:20. doi: 10.1186/1472-6947-7-20 17617908 Lopes MV, Silva VM, Araujo TL. Methods for establishing the accuracy of clinical indicators in predicting nursing diagnoses. Int J Nurs Knowl. 2012;23(3):134–9. doi: 10.1111/j.2047-3095.2012.01213.x 23043652 Stacey D, Lewis KB, Smith M, Carley M, Volk R, Douglas EE, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2024;1(1):Cd001431. doi: 10.1002/14651858.CD001431 Prior SJ, Mather C, Ford K, Bywaters D, Campbell S. Person-centred data collection methods to embed the authentic voice of people who experience health challenges. BMJ Open Qual. 2020;9(3). doi: 10.1136/bmjoq-2020-000912 32699082 Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving Research Impact Through Co-creation in Community-Based Health Services: Literature Review and Case Study. Milbank Q. 2016;94(2):392–429. doi: 10.1111/1468-0009.12197 27265562 Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. 2012;90(2):311–46. doi: 10.1111/j.1468-0009.2012.00665.x 22709390 Price A, Clarke M, Staniszewska S, Chu L, Tembo D, Kirkpatrick M, et al. Patient and Public Involvement in research: A journey to co-production. Patient Educ Couns. 2022;105(4):1041–7. doi: 10.1016/j.pec.2021.07.021 34334264 van Deventer C, McInerney P, Cooke R. Patients’ involvement in improvement initiatives: a qualitative systematic review. JBI Database System Rev Implement Rep. 2015;13(10):232–90. doi: 10.11124/jbisrir-2015-1452 26571293 Halvorsrud K, Kucharska J, Adlington K, Rüdell K, Brown Hajdukova E, Nazroo J, et al. Identifying evidence of effectiveness in the co-creation of research: a systematic review and meta-analysis of the international healthcare literature. J Public Health (Oxf). 2021;43(1):197–208. doi: 10.1093/pubmed/fdz126 31608396 Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382–5. 3640358 Anderson RJ, Bloch S, Armstrong M, Stone PC, Low JT. Communication between healthcare professionals and relatives of patients approaching the end-of-life: a systematic review of qualitative evidence. Palliat Med. 2019;33(8):926–41. doi: 10.1177/0269216319852007 31184529 Gonella S, Conti A, Albanesi B, Antal A, Dimonte V, Di Giulio P. Lived experiences of end-of-life communication among nursing home staff: An interpretive phenomenological study. J Adv Nurs. 2023;79(2):698–710. doi: 10.1111/jan.15489 36447384 Towsley GL, Hirschman KB, Madden C. Conversations about End of Life: Perspectives of Nursing Home Residents, Family, and Staff. J Palliat Med. 2015;18(5):421–8. doi: 10.1089/jpm.2014.0316 25658608 DuBay WH. The principles of readability. Impact Information, Costa Mesa;2004. Gray WS, Leary BE. What makes a book readable. University of Chicago Press;1935. Ancker JS, Kaufman D. Rethinking health numeracy: a multidisciplinary literature review. Journal of the American Medical Informatics Association. 2007;14(6):713–21. doi: 10.1197/jamia.M2464 17712082 Baxter M, Lonsdale MdS, Westland S. Utilising design principles to improve the perception and effectiveness of public health infographics. Information Design Journal. 2021;26(2):124–56. Reid H, Caterson J, Smith R, Baldock J, Jones N, Copeland R. What dohealthcare professionalswant from a resource to supportperson-centred conversations onphysical activity? A mixed-methods, user-centric approach to developing educational resources. BMJ Open Sport Exerc Med. 2022;8(2):e001280. doi: 10.1136/bmjsem-2021-001280 35813130 Evenblij K, Ten Koppel M, Smets T, Widdershoven GA, Onwuteaka-Philipsen BD, Pasman HRW. Are care staff equipped for end-of-life communication? A cross-sectional study in long-term care facilities to identify determinants of self-efficacy. BMC palliative care. 2019;18:1. doi: 10.1186/s12904-018-0388-z 30621703 Saretta M, Doñate-Martínez A, Alhambra-Borrás T. Barriers and facilitators for an effective palliative care communication with older people: A systematic review. Patient Educ Couns. 2022;105(8):2671–82. doi: 10.1016/j.pec.2022.04.003 35459530 Ten Koppel M, Onwuteaka-Philipsen BD, van der Steen JT, Kylänen M, Van den Block L, Smets T, et al. Care staff’s self-efficacy regarding end-of-life communication in the long-term care setting: Results of the PACE cross-sectional study in six European countries. Int J Nurs Stud. 2019;92:135–43. doi: 10.1016/j.ijnurstu.2018.09.019 30822706 10.1371/journal.pone.0304180.r001 Decision Letter 0 Wiyarta Elvan Academic Editor 2024 Elvan Wiyarta This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

17 Mar 2024

PONE-D-24-04935Sustaining health and social care professionals in serious illness conversations: Development, validation, and preliminary evaluation of an educational bookletPLOS ONE

Dear Dr. Gonella,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by May 01 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Elvan Wiyarta, M.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We noticed you have some minor occurrence of overlapping text with the following previous publication(s), which needs to be addressed:

- https://doi.org/10.3390/healthcare12030398

In your revision ensure you cite all your sources (including your own works), and quote or rephrase any duplicated text outside the methods section. Further consideration is dependent on these concerns being addressed.

3. We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service.

Whilst you may use any professional scientific editing service of your choice, PLOS has partnered with both American Journal Experts (AJE) and Editage to provide discounted services to PLOS authors. Both organizations have experience helping authors meet PLOS guidelines and can provide language editing, translation, manuscript formatting, and figure formatting to ensure your manuscript meets our submission guidelines. To take advantage of our partnership with AJE, visit the AJE website (http://learn.aje.com/plos/) for a 15% discount off AJE services. To take advantage of our partnership with Editage, visit the Editage website (www.editage.com) and enter referral code PLOSEDIT for a 15% discount off Editage services.  If the PLOS editorial team finds any language issues in text that either AJE or Editage has edited, the service provider will re-edit the text for free.

Upon resubmission, please provide the following:

The name of the colleague or the details of the professional service that edited your manuscript

A copy of your manuscript showing your changes by either highlighting them or using track changes (uploaded as a *supporting information* file)

A clean copy of the edited manuscript (uploaded as the new *manuscript* file)

4. Thank you for stating the following financial disclosure:

“This work was supported by Fondazione Assistenza e Ricerca Oncologica (F.A.R.O.) Onlus. The funding did not influence the study process either the study findings.”

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.

6. Please include a separate caption for each figure in your manuscript.

7. Please include a copy of Tables 1, 2, 3 and 4 which you refer to in your text on page 10 and 11.

8. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Additional Editor Comments:

Please revise according to the reviewer's input

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Review: Sustaining health and social care professionals in serious illness conversations: Development, validation, and preliminary evaluation of an educational booklet

Abstract

Clarity required about what constitutes a serious illness conversation (i.e. what illness, at what stage of illness trajectory) and whether the patient is an adult or a child. It is not clear from the abstract who (assume healthcare professionals) is aiming to communicate to who (the patient, patient’s family etc)

Introduction

Please make it clearer much earlier who is communicating to who (as per comments on abstract)

Communication skills training may not be a part of education for healthcare professionals in Italy, but is certainly a growing part of formal professional training for healthcare professionals in other countries. Please acknowledge these differences; the introduction could be strengthened by considering how other healthcare systems attempt to address challenges of communication.

The introduction helpfully identifies literature regarding the benefits of written communication guides; these are based on a wide range of different clinical contexts and the section could be strengthened by examination of where these have been used (eg with frail elderly vs acute clinical contexts) and the implications of these different environments for the nature and type of communication.

This section needs a much stronger rationale as to the limitations of the guides which exist already, and why something new is required.

The paper requires a rationale about why it was decided to use the format of a written communication booklet. Were other modalities considered eg video, audio, online learning etc.

Method

Please elaborate on the rationale for the clinical cases used and the communication challenges selected to be illustrated in these scenarios.

It would enhance the paper to include a brief review of the communication protocols referenced as examples of ‘good communication scripts’, to include their main principles or content.

Please clarify why representatives from patient associations, carers and volunteers were only drawn from the dementia field, how these representatives were identified and recruited.

Please add additional methodological information about the feedback from the focus group participants and how this was integrated into the second iteration of the booklet. It is essential that the methodology for analysing the results of the discussion is clearly described, how potential conflicts of opinion were resolved etc. Please add the ‘pre-defined set of questions’ as a table or to the body of the manuscript, with a rationale for how these were selected. If the focus groups were not recorded, transcribed and analysed using an established methodology, please state and explain why not.

Validation: please add further detail about the validation process. In particular, further information is required regarding the number of sections which did not achieve an I-CVI of >0.70 and what decisions were made regarding improving these sections or discarding them. Were there differences between the professional raters and those from patient organisations?

Further information is required about the sections which were adjusted, detailing what needed to be addressed to improve relevance, clarity or realism and how this was addressed.

The statistical analyses used a methodology based on decision aid development and evaluation. Please explore the similarities and differences between a decision aid and a communication booklet.

I suggest that presenting the method and then results for each of the phases of development would enhance the readability of the manuscript.

Results

The amendments to the initial draft of the guide is well documented in Table 1. However, this does not include adequate detail about why these adjustments were made (see comment re methodology used to identify and make decisions about what to amend).

Discussion

The authors highlight the importance of developing a resource to support person-centred conversations; please add in reflection about the decision not to use A Person-Centred Approach methodology. Elements of this methodology have been used by the team, but there is no mention of this within the paper.

A substantial section of the discussion is framed around the 12 domains of behaviour change. This is a useful framework, but it would be more logical and helpful to include this in the introduction; currently it feels like a ‘post hoc’ analysis rather than being core to the development of the booklet.

The decision to involve a design agency is not included in the method an needs earlier integration into the paper, with a clear rationale about the problem being addressed.

The discussion briefly mentions ‘the maternal-infant context’. I assume this refers to talking to adults when the patient is an infant or child? Please clarify the language around this. The issues relating to communication about a child’s illness are related but also include separate, additional challenges. This needs much more careful consideration than a single line in the discussion.

Please expand the section discussing the limitations of the booklet and its development. For example (but not exclusively), would any qualitative work have contributed to the evaluation of the booklet? What are the pros and cons of a physical booklet for wider distribution? How are the authors planning to disseminate the booklet? What impact do they anticipate it would have on patient care and how could this be evaluated? Limitations of the patient scenarios given the broader context of different types of chronic illness?

The booklet is aimed at discussions with patients, but particularly in the context of frail elderly or patients who are at end of life, what are the implications about communication with other members of the family or carers?

Reviewer #2: The introduction was comprehensive and read well.

Line 113 and contributes to know the patients’ valued priorities and align care with their preferences.

Methodology included creation of the resource, validation and evaluation.

Line 194

Fourthly, the booklet was further adjusted according to the emerging comments

Could the authors please provide the pre-set questions used in the focus group?

Results were provided with appropriate statistics.

A link to the booklet itself is provided. I would be interested to read an English version, if available.

This included discussion of the co-creation process.

The option of communication interventions appropriate for 1, 2 and 5 minutes was an excellent concept.

The paragraph on page 14 is long and could be divided into 2.

Suggest add 2 - 3 sentences to the conclusion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Dr Anthony Herbert

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0304180.r002 Author response to Decision Letter 0 Submission Version 1

5 Apr 2024

Please, see the file "Response to reviewers".

Submitted filename: Response to Reviewers.docx

 10.1371/journal.pone.0304180.r003 Decision Letter 1 Wiyarta Elvan Academic Editor 2024 Elvan Wiyarta This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

8 May 2024

Supporting health and social care professionals in serious illness conversations: Development, validation, and preliminary evaluation of an educational booklet

PONE-D-24-04935R1

Dear Dr. Gonella,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Elvan Wiyarta, M.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: I have reviewed this paper previously.

The authors have addressed all of my suggested comments.

This includes an expanded conclusion.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********

 10.1371/journal.pone.0304180.r004 Acceptance letter Wiyarta Elvan Academic Editor 2024 Elvan Wiyarta This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

22 May 2024

PONE-D-24-04935R1

PLOS ONE

Dear Dr. Gonella,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Mr. Elvan Wiyarta

Academic Editor

PLOS ONE