PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0312782 PONE-D-24-13780 Research Article Biology and life sciencesMicrobiologyMedical microbiologyMicrobial pathogensViral pathogensImmunodeficiency virusesHIV Medicine and health sciencesPathology and laboratory medicinePathogensMicrobial pathogensViral pathogensImmunodeficiency virusesHIV Biology and life sciencesOrganismsVirusesViral pathogensImmunodeficiency virusesHIV Biology and life sciencesOrganismsVirusesImmunodeficiency virusesHIV Biology and life sciencesOrganismsVirusesRNA virusesRetrovirusesLentivirusHIV Biology and life sciencesMicrobiologyMedical microbiologyMicrobial pathogensViral pathogensRetrovirusesLentivirusHIV Medicine and health sciencesPathology and laboratory medicinePathogensMicrobial pathogensViral pathogensRetrovirusesLentivirusHIV Biology and life sciencesOrganismsVirusesViral pathogensRetrovirusesLentivirusHIV Medicine and health sciencesMental health and psychiatry Medicine and health sciencesPublic and occupational healthHealth screening People and placesGeographical locationsNorth AmericaUnited StatesAlabama Medicine and health sciencesMental health and psychiatryMood disordersDepression Medicine and health sciencesMental health and psychiatryMental health therapies Medicine and health sciencesClinical medicineSigns and symptomsHIV clinical manifestations Biology and life sciencesPsychologyEmotionsAnxiety Social sciencesPsychologyEmotionsAnxiety Developing an implementation blueprint: Lessons learned from integrating electronic patient-reported outcomes in HIV clinics in Alabama Implementation and lessons learned from integrating electronic patient-reported outcomes in Alabama https://orcid.org/0000-0001-7996-8951 Gagnon Kelly W. Data curation Formal analysis Visualization Writing – original draft Writing – review & editing 1 * Burgan Kaylee Data curation Formal analysis Methodology Visualization Writing – original draft Writing – review & editing 1 Mulrain Morgan Data curation Project administration Writing – review & editing 1 Baral Stefan Conceptualization Supervision Writing – review & editing 2 Cropsey Karen Conceptualization Supervision Visualization Writing – review & editing 3 4 Mugavero Michael Conceptualization Supervision Writing – review & editing 1 Eaton Ellen Conceptualization Data curation Funding acquisition Investigation Methodology Project administration Writing – original draft Writing – review & editing 1 4 Division of Infectious Diseases, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, United States of America Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States of America Department of Psychiatry and Behavioral Neurobiology, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, United States of America Center for Addiction and Pain Prevention and Intervention, University of Alabama at Birmingham, Birmingham, AL, United States of America Jahun Ibrahim Editor University of Manitoba Faculty of Health Sciences, NIGERIA

EE reports honorarium from Gilead for participating in the HIV Re-engagement working group and from PRIME, DKBMed, and IAS-USA for developing HIV continuing education content. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

 * E-mail: kgagnon@uabmc.edu 30 10 2024 2024 19 10 e0312782 6 5 2024 12 10 2024 2024 Gagnon et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

People living with HIV are disproportionately affected by depression, anxiety, and substance use which impede engagement with HIV treatment services and can increase risks of HIV-related morbidity and mortality. Capturing timely, accurate patient data at point of care is recommended to inform clinical decision-making and retain patients on the HIV care continuum. Currently, there is limited use of validated screening tools for substance use and mental health at the point of care in HIV clinics, even though people with HIV (PWH) have a high prevalence of these comorbidities. Even fewer clinics screen in a manner that encourages disclosure of stigmatized substance use, depression, and anxiety. Electronic patient-reported outcomes (ePROs) are an evidence-based modality to overcome such limitations by eliciting responses directly from patients via tablet, smartphone, or computer. To date, there is limited consensus on how to implement ePROs into non-academic settings and enhance uptake. Our team sought to address this gap by examining the implementation of ePROs, previously implemented in an academic clinic, to enhance screening and treatment of mental health (MH) and substance use at five Ryan White HIV/AIDS Program-funded clinics in Alabama. The ePROs were delivered through a multi-component intervention titled HIV + Service delivery and Telemedicine through Effective Patient Reported Outcomes (+STEP), which also provides targeted training to frontline clinicians, and resources for MH and substance use treatment for PWH without access to care. The objective of this study is to provide an implementation blueprint that can be tested and utilized in other HIV clinics to integrate ePROs and increase evidence-based screening for depression, anxiety, and substance use among PWH, as well as outline lessons learned from implementation to date. The findings from this study provide practical steps and advice based on our experience in implementing electronic patient-reported outcomes in HIV clinics in the US Deep South.

 http://dx.doi.org/10.13039/100000025 National Institute of Mental Health 5R01MH124633-04 Eaton Ellen This study was funded by the National Institutes of Mental Health (PI: Eaton, 5R01MH124633-04). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability All relevant data are within the manuscript and its Supporting Information Files. Introduction

People with HIV (PWH) are disproportionately affected by depression, anxiety, and substance use which impede engagement with HIV treatment services and can increase risks of HIV-related morbidity and mortality [1–5]. Interruptions in HIV treatment and engagement in HIV care can have a negative impact on viral load (VL) suppression among people living with HIV, increasing the risk of HIV progression and transmission [6]. Since poor mental health and substance use are associated with interruptions to HIV treatment, routine screening during point of contact is recommended to identify these co-occurring conditions [7–9]. This is often achieved through the use of validated tools for psychosocial barriers to care such as depression, anxiety, and substance use [6, 10–12]. Routine mental health and substance use screening are recommended by the United States Preventive Services Task Force and the International Antiviral Society- USA for all adults [8, 9].

Despite being recommended for all adults, there is limited routine screening for substance use and mental health conditions beyond depression in HIV clinics, even though PWH have a high prevalence of these comorbidities [13]. Noted barriers to routine substance use and mental health screening are staff hesitation to initiate discussions about these sensitive subjects and limited time during clinical encounters [14, 15]. Electronic patient-reported outcomes (ePROs) are an evidence-based modality to overcome such limitations by eliciting responses directly from patients via tablet, smartphone, or computer [16]. Further, studies have shown that ePROs may mitigate stigma, medical mistrust, and social desirability bias that may impact patients’ comfort in reporting symptoms or behaviors as they are completed without a staff member [16, 17]. Lastly, integrating ePROs into routine HIV clinical care provides longitudinal data on symptom severity, care engagement, and patients’ psychological and physical well-being [16].

While there are documented processes for establishing ePROs in academic HIV settings, there remains limited consensus on how to implement ePROs into non-academic HIV clinical practice [6, 18]. Given the complexity and barriers to treatment for PWH in non-academic HIV clinical settings, including those in rural areas, our team sought to address this gap by examining the implementation of ePROS to enhance screening and treatment of mental health (MH) and substance use at five Ryan White HIV/AIDS Program (RWHAP)-funded clinics in Alabama (AL) [19]. The ePROs were delivered as part of a multi-component intervention titled HIV + Service delivery and Telemedicine through Effective PROs (+STEP), which also provides targeted training to frontline clinicians, and resources for MH and substance use treatment for PWH without access to care [19].

The objective of this study is to provide an implementation blueprint that can be tested and utilized in other HIV clinics to integrate ePROs and increase evidence-based screening for depression, anxiety, and substance use among PWH. This process, while straightforward in theory, proved quite difficult in practice when implemented in 2022 in under-resourced HIV clinics in the Southeastern US. In addition to sharing the preliminary blueprint and implementation outcomes of +STEP, this study aimed to provide a detailed description of the lessons learned from the implementation of ePROs in five RWHAP-funded clinics across Alabama. Specifically, we identified critical steps and components of integrating and scaling ePROs in HIV clinics that should be accounted for as part of any future efforts to implement ePROs in this clinical setting.

 Materials and methods

The protocol for this study is published in JMIR Research Protocols [19]. In brief, this study utilizes a stepped-wedge hybrid implementation-effectiveness type 1-design to simultaneously evaluate implementation strategies and the intervention [20]. The +STEP intervention consists of three components: ePROs, targeted staff training, and telemedicine services for MH and substance use. +STEP includes ePROs using validated scales for depression (Patient Health Questionnaire- 9) [21], anxiety (General Anxiety Disorder- 7) [22], alcohol consumption (Alcohol Use Disorders Identification Test-Concise) [23], substance use (Alcohol, Smoking, and Substance Involvement Screening Test) [11], and medication adherence (Self-Rating Adherence Scale-5) [24]. Targeted training is described in detail below. Although ePRO data is collected for the overall study, this manuscript is focused on the preliminary blueprint and lessons learned and does not feature any outcomes from the ePROs individually or in aggregate. The number of PWH receiving care at each of the five +STEP study sites can be found in Table 1. A figure showing the geographic distribution of sites has been previously published [19]. This study was approved by the University of Alabama at Birmingham (UAB) Institutional Review Board and received a waiver of informed consent as data were collected as part of routine clinical care.

10.1371/journal.pone.0312782.t001 The number of people living with HIV at each Ryan White Funded Clinic.
Ryan White Funded Clinics in Alabama PWH (n)
Clinic 1 264
Clinic 2 2,700
Clinic 3 346
Clinic 4 550
Clinic 5 956
TOTAL (Active PWH at Participating Sites) 4,816

Data from 2022

*Participating site includes a primary clinical site and one more satellite sites

After conducting the readiness, acceptability, and accessibility assessment and developing the aforementioned training modules, we began +STEP implementation at Clinics 1 and 2. Due to COVID-19 clinical precautions, implementation began after April 2022. In alignment with our stepped-wedge design, Clinics 3, 4, and 5 launched in October 2022, November 2022, and March 2023, respectively. Throughout the course of implementation, our team developed a preliminary implementation blueprint, a formal implementation plan that describes the specific goals, strategies, personnel, timeframe, and scope involved in the implementation change process [25, 26]. Measures from Proctor’s Conceptual Model of Implementation Research were used to assess early implementation outcomes [27]. Specifically, we evaluated adoption of ePROs and effectiveness of clinic workflows on adoption.

To identify lessons learned, our team discussed ongoing challenges and barriers at each step of the implementation process during weekly research team meetings and quarterly Research Team Meetings with experts in HIV and implementation science. The authors also reflected on this process and how issues could be prospectively addressed in future iterations of this work.

 Results Preliminary blueprint

We utilized a four-step process to implement +STEP, which serves as the preliminary blueprint to conduct similar work in other HIV clinics (Fig 1). To initiate this study, we identified partner sites by leveraging an existing consortium of all nine RWHAP Part C and D funded clinics across Alabama, the Alabama Quality Management Group (AQMG) [28]. Specifically, we presented the initial concept of +STEP to this consortium in the fall of 2019 and invited all sites to participate. Six sites expressed interest in participation; however, just after funding was granted, one site had to drop out due to competing priorities that arose as a result of COVID-19. We then worked with contacts at each of the five remaining sites to identify site champions. In addition to being primary contacts, champions were responsible for promoting +STEP, training new staff, and troubleshooting issues that arose.

10.1371/journal.pone.0312782.g001 Implementation blueprint utilized in +STEP.

Once sites and champions were introduced to +STEP objectives, we conducted a structured needs assessment with staff, providers, and patients to better ascertain: (1) their current workflow, (2) their current screening protocols, (3) their telehealth infrastructure, and (4) available resources. Detailed methods and the results of this needs assessment have been previously published [29]. Following the needs assessment, we ranked clinics from “most ready” to “least ready” to implement. We implemented in pairs starting with the clinic that ranked first in our readiness assessment as well as the clinic that ranked fifth, followed by the clinics that ranked second and fourth, and then the clinic that ranked third. This modified stepped-wedge design was utilized to identify barriers, challenges, and confusion surrounding the intervention and its implementation in stages.

The third step in our preliminary blueprint was to develop implementation materials prior to roll-out. Our team began by developing implementation materials that would act as a guide and reference for clinics throughout their launch. Specifically, we developed four asynchronous trainings that would provide targeted education to clinic staff. These trainings covered topics such as MH and substance use stigma, the importance of screening for and treatment of MH and substance use and the pros and cons of telehealth in this space. These trainings were designed for clinic staff, including providers, nurses, social workers, and front desk staff. We asked participating sites to complete trainings prior to their site’s launch in return for one hour of Continuing Medical Education (CME) credit and an additional payment installment to their clinic. The research team also compiled an implementation packet, which consisted of (1) contact information for the research team and implementation teams; (2) a pre-implementation checklist; (3) a suggested clinic workflow that details where ePROs may take place and which staff may be responsible for each stage; and (4) screenshots and step-by-step procedures for operating REDCap (REDCap, Vanderbilt University, Nashville, TN) for PRO data entry [30, 31].

Lastly, sites were onboarded to +STEP in alignment with the modified stepped-wedge study design. To onboard, the research team traveled to the sites to meet the implementation teams (champion, front desk staff, clinicians, social workers, case managers) and review onboarding materials. Onboarding materials included a reminder to complete the CME training and confirming that those implementing +STEP had access to REDCap. The research team also checked the iPads to ensure the REDCap shortcut was active and working. After confirming they had access to the REDCap, the implementation teams completed three mock patient encounters to ensure they understood the workflow and asked any questions of the study team. Virtual onboarding sessions were held for any implementation team members who were unable to join in-person meetings. Implementation teams were expected to begin ePROs after being onboarded to the study. A virtual follow-up meeting was scheduled for two weeks after onboarding to discuss progress and identify any challenges. These meetings continued on a monthly basis to ensure the research team could address questions or issues as they arose. The research team maintained open lines of communication with the implementation teams via e-mail between meetings.

 Implementation outcomes

To date, a total of 1,110 baseline ePROs have been completed across sites. As seen in Fig 2, this continuous increase is attributable to an additional 25–135 baseline ePROs each month. This demonstrates a steady roll-out of +STEP across sites and the initial adoption of ePROs among patients who had not yet received an ePRO.

10.1371/journal.pone.0312782.g002 Number of baseline ePROs each monthly and running total number of baseline ePROs.

As mentioned in the preliminary blueprint (Fig 1) our study team worked with implementation teams at each site prior to onboarding site visits to discuss where ePROs would be integrated into their workflow, including who was responsible for adding patients to the REDCap and handing them the iPad. In all workflows, patients completed the screeners on the iPads independently, unless they requested assistance. These workflows were modified as needed to align with the staff, resources, and structure of each clinic. Across the five clinics, two main workflows emerged: 1) ePROs are conducted with the patient during their medical visit with mental health staff as the responsible +STEP champion and 2) ePROs are conducted prior to a medical visit with a medical or mental health provider as responsible +STEP champion. A visual of these workflows can be found in Fig 3.

10.1371/journal.pone.0312782.g003 Workflows implemented by sites to integrate ePROs.

Specifically, two clinics utilize Workflow 1, and three clinics utilize Workflow 2. The effectiveness of these respective workflows is highly dependent on a variety of factors in each clinic, such as staffing and the clinic’s patient load. Table 2 presents the number of baseline ePROs collected at each site by workflow utilized. This suggests that Workflow 2 may be more consistent; however, it is worth noting that Clinic 2 had a change in clinic leadership and organizational restructuring that prevented them from participating for six months. The only other clinic currently utilizing Workflow 1 is still in the early stages of implementation and has not had the same time to achieve the level of adoption as other sites.

10.1371/journal.pone.0312782.t002 Workflow utilized by clinic and total number of baseline ePROs to date by clinic<xref ref-type="table-fn" rid="t002fn001">*</xref>.
Clinic 1 Clinic 2 Clinic 3 Clinic 4 Clinic 5
Workflow 1 X X
Workflow 2 X X X
Total Number of 1st ePROs to date 260 531 144 117 58
Proportion of baseline ePROs to Patient Population 98% 20% 42% 21% 6%

*Clinics are numbered according to date of onboarding with Clinic 1 implementing the intervention first and Clinic 5 being last.

 Lessons learned

We identified three primary areas of focus that should be considered during any future efforts to implement ePROs in similar clinical settings.

Clinic engagement

Before implementing a new intervention, it is critical to understand the clinics you are working with, including their resources, staffing, and primary points of contact who will help navigate on-the-ground project operations. We found that a productive strategy to initiate engagement was to reach out to existing community groups to present the proposed study and identify points of contact at interested sites. Specifically, during the planning stages of +STEP, our team leveraged AQMG to identify community partners that were willing and interested in implementing +STEP as the new standard of care at their clinic.

During this phase, we also identified a project champion from each site to provide input on +STEP, facilitate communication with other clinic staff, and participate in our needs assessment. These individuals typically held higher-level positions, such as Chief Operating Officer, Chief Executive Officer, or Lead Administrator. Though they could speak to the larger context of their clinics, their expertise was not best suited for understanding on-the-ground clinic operations and perspectives, nor communication of project activities and requirements between staff and the research team. Future studies should enter partnerships with a clearer understanding of which clinical staff could fulfill the role of a project champion, as well as an alternate staff that could fill in as needed.

Additionally, it would have improved our ability to support the sites if we had inquired about the funding landscape of each site, as well as other projects and programs that may cause competition for priorities, resources, and staff engagement. While our needs assessment did evaluate available resources and which staff should be included in the implementation of +STEP, we did not properly address the various priorities and demands associated with other long-term programs that may impede engagement in +STEP. Specifically, it would have been useful to better understand how many of these clinics were participating in research and programmatic grants. Each of our participating sites had a variety of sponsored projects with UAB, state, and federal funding agencies during the implementation of +STEP. While unknown to our research team, managing multiple studies and funded programs at once created frustrations and confusion regarding the differences between projects, including their goals, expected deliverables, and timelines. This was especially the case for our smaller clinics, which have limited staff capacity. For these sites in particular, the same staff were expected to fill multiple roles across projects resulting in confusion and competing priorities. This highlights how an intervention may be relatively easily adopted in an academic HIV clinic; whereas, implementation may be more challenging at understaffed clinics with limited capacity.

Another area of clinic engagement that arose as a lesson learned was the value of site visits. It was evident that site visits were critical to the onboarding and implementation process. Thus far, we have conducted two site visits to every site, one to launch the intervention (between April 2022 and March 2023) and second to check in, meet new staff, and discuss barriers to the intervention between May and August 2023. From an objective standpoint, we observed an uptick in the number of ePROs directly following site visits; however, we also noted subjective evidence of the value of these visits. Specifically, meeting in person helped the research and implementation teams build rapport and differentiate responsibilities across both groups. Because of site visits, it was clear that implementation teams had a better understanding of who to request support from as issues arose. Since these visits were longer than the monthly check-ins, the research team was also able to elicit more barriers to implementation and collect implementation outcomes data for continuous evaluation.

Further, the research team observed that these visits were often the only time all members of the implementation teams gathered to discuss the progress of conducting ePROs at their site. Thus, these meetings were internally beneficial to improve communication, both within the site and with the research team. Future studies should consider integrating more regular site visits in their protocol to help maintain rapport and relationships with partners and facilitate open communication. Research staff should budget time and resources to allow frequent site visits.

 Incentives

Understanding a clinic’s motivations for participation is imperative to the success of any implementation study. Prior to implementation, it is necessary to have clear and established guidelines for the delivery of incentives and what benchmarks or objectives need to be met in order for those incentives to be released. Incentives are necessary in order to ensure engagement in ongoing research activities, as well as to provide proper compensation for the time and energy dedicated to the work.

While funding for equipment and ongoing participation was provided to each clinic for their involvement in +STEP, we discovered that these clinics are involved in multiple research studies hosted by UAB. This complicated participation as incentive checks were distributed in bulk from all UAB projects and sites could not differentiate how much they were specifically receiving from +STEP. Further, incentives were dispensed to clinical leadership and not directly to staff implementing +STEP. While it was important to the study team to support the autonomy of the clinics in determining how money was allocated to research tasks, we observed that this incentive model did not act as a form of positive reinforcement for participation. Since staff did not personally receive incentives, there was a critical disconnect between participation and compensation. In future work, we plan to implement a dual incentive model to compensate both the clinic and staff to mitigate this issue.

 Communication

At the center of our lessons learned, we identified the importance of early, consistent communication with our study sites and research experts. This includes ensuring that appropriate representatives are identified, engaged, and at the table during discussions, ongoing review and modifications to implementation materials, and consideration for the dynamics of virtual and physical meeting spaces.

Primarily, it is critical for research team members, regardless of role on the project, to have training and/or experience with communicating with community organizations. A +STEP community liaison, who had established relationships and experience with these organizations, was engaged as part of the research team in an effort to avoid paternalistic behaviors from the research team. These organizations are the experts on their workflow and the needs of their patients, so utilizing an inquisitive approach rather than an instructional one was more productive in evaluating and addressing challenges to implementation.

Prior to visiting the sites for onboarding, the research team attempted to collect the names and roles of staff at the site who would be implementing +STEP at any point during the study period. We found that this was a difficult task, as sites did not fully understand the scope and expectations of the project. In addition, there was notable staff turnover such that staff who needed training evolved over the course of the study. We often collected additional names and contact information during site visits and conducted virtual onboarding for those staff members at a later date. It would have been beneficial to initiate the process with a virtual overview of the project followed by an inquiry as to what staff members would be involved.

Two communication-specific issues arose as it pertains to study materials: 1) specificity of implementation materials and 2) training needs and support. These issues were pronounced as study sites experienced staff turnover and new implementation team members joined the project. The development and use of implementation materials were vital for documenting the purpose and protocols for implementation of +STEP at each clinic; however, there were several opportunities for improvement to make these materials more useful to clinic staff. While our team developed materials that seemed most relevant to each clinic, it was apparent that more detail and nuance would have been beneficial to help guide staff through implementation. In addition to engaging an implementation expert to develop these materials, it was necessary to make ongoing modifications to the implementation materials and workflow documentation. This was of particular importance as new staff joined the teams to facilitate autonomy of each study site in managing both new staff and the implementation process.

As noted under clinic engagement, site visits were essential to implementation, however gathering space at sites was very limited and clinic schedules and patient demands served as barriers to engagement. This required critical evaluation of who needed to be present during site visits, both from the UAB +STEP research team and the sites’ implementation teams.

 Discussion

To address the limited consensus on the implementation of ePROs as part of routine HIV clinical practice in non-academic settings, +STEP seeks to establish a blueprint to integrate ePROs into five RWHAP clinics in AL. While this initiative is ongoing, findings from the first year of implementation demonstrate preliminary indicators of a successful blueprint. Specifically, we observed a relatively consistent rate of new ePRO participants over the course of the first year of implementation, accumulating to over one thousand enrolled participants. In addition to sharing the preliminary blueprint, this study reported important lessons learned during the implementation process. The findings from this study serve to begin addressing the literature gap pertaining to the implementation of ePROs to assess mental health and substance use within the context of HIV clinics.

To date, every site in +STEP has onboarded and initiated ePROs with their patients. Participating sites implemented one of two workflows, with initial indications that it is more effective to conduct ePROs prior to rooming the patient. Previous research focused on process guidance, such as choosing modalities for administering the ePROs and reporting [32–34]; however, these workflows provide a more granular understanding of how ePROs were integrated into HIV clinical care. The two workflow models reported in this study and the corresponding early uptake of ePROs at each site suggest that integrating ePROs prior to rooming the patient may be most achievable for HIV clinics. This corresponds with when screeners, regardless of modality, are administered in most clinical settings [35]. Previous literature suggests this may be because that is when patients have the most independent time and that it is difficult to intervene during other points in the visit, resulting in patients ultimately not completing the screeners [36].

Our preliminary blueprint includes four key steps to initiating and implementing ePROs into routine HIV care. Specifically, the blueprint is a sequential process that builds on itself to achieve implementation. Our four-step blueprint is similar to existing models for implementing ePROs [36, 37], which outline the importance of setting goals for collecting ePROs in clinical practice, understanding the existing context prior to implementation, and training implementation staff. Our preliminary blueprint differs in that we describe the process of identifying partners and leveraging existing community coalitions for intervention and the importance of developing implementation materials to inform the process at each site. As +STEP is an ongoing study, we anticipate that the preliminary blueprint may be modified, including adding further steps as we advance into later implementation and maintenance. Based on existing ePRO frameworks [38], we anticipate steps may include ongoing evaluation and process improvement.

Notably, we identified valuable lessons we learned over course of this process that would be beneficial to informing ongoing and future efforts to integrate ePROs into routine HIV care. Specifically, we shared lessons in three primary areas: clinic engagement, incentives, and communication. Within the context of clinic engagement, input and buy-in from leadership was vital; however, speaking directly with clinic staff and identifying an implementation champion during the first step of the blueprint would have helped us avoid later communication issues. While we did incentivize the clinics, it was a missed opportunity during the needs assessment phase to understand the scope and origin of each clinics’ compensation and how that may compete with our initiative. The presence of a community liaison on +STEP facilitated conversations and was overall a notable facilitator to interactions between the study team and clinics. Lastly, implementation materials were critical for each HIV clinic, but we identified the need to maintain these materials as living documents to facilitate implementation, particularly as turnover occurred at sites.

Conclusions

Our preliminary blueprint aligns with existing models for implementing novel interventions into routine clinical care. These efforts are the first to our knowledge that specify practices for implementing ePROs at RWHAP HIV clinics, particularly in the Deep South. This blueprint demonstrated early usefulness in facilitating implementation through the uptake of ePROs to date. Our lessons learned reflect effective strategies and areas for improvement to inform future efforts to introduce ePROs in HIV clinics. The practical information generated from +STEP thus far advances the field through early insight into the implementation process, outcomes, and lessons.

 Supporting information Manuscript data.

This table contains all the data presented in the manuscript in alignment with the data availability statement.

(XLSX)

 References Pence BW. The impact of mental health and traumatic life experiences on antiretroviral treatment outcomes for people living with HIV/AIDS. J Antimicrob Chemother. 2009;63(4):636–40. doi: 10.1093/jac/dkp006 19153077 Hoare J, Sevenoaks T, Mtukushe B, Williams T, Heany S, Phillips N. Global systematic review of common mental health disorders in adults living with HIV. Curr HIV/AIDS Rep. 2021:1–12. Klinkenberg Wa, Sacks S, for the Hiv/aids Treatment Adherence HO, Group CS. Mental disorders and drug abuse in persons living with HIV/AIDS. AIDS care. 2004;16(sup1):22–42. doi: 10.1080/09540120412331315303 15736820 Ickovics JR, Hamburger ME, Vlahov D, Schoenbaum EE, Schuman P, Boland RJ, et al. Mortality, CD4 Cell Count Decline, and Depressive Symptoms Among HIV-Seropositive WomenLongitudinal Analysis From the HIV Epidemiology Research Study. JAMA Netw Open. 2001;285(11):1466–74. doi: 10.1001/jama.285.11.1466 11255423 Gonzalez JS, Batchelder AW, Psaros C, Safren SA. Depression and HIV/AIDS treatment nonadherence: a review and meta-analysis. JAIDS (1999). 2011;58(2):181–7. doi: 10.1097/QAI.0b013e31822d490a 21857529 Kozak MS, Mugavero MJ, Ye J, Aban I, Lawrence ST, Nevin CR, et al. Patient Reported Outcomes in Routine Care: Advancing Data Capture for HIV Cohort Research. Clin Infect Dis. 2011;54(1):141–7. doi: 10.1093/cid/cir727 22042879 Substance Abuse and Mental Health Services Administration. Treating Substance Use Disorders Among People with HIV. 2021. Gandhi RT, Bedimo R, Hoy JF, Landovitz RJ, Smith DM, Eaton EF, et al. Antiretroviral drugs for treatment and prevention of HIV infection in adults: 2022 Recommendations of the International Antiviral Society–USA Panel. JAMA Netw Open. 2023;329(1):63–84. doi: 10.1001/jama.2022.22246 36454551 Patnode CD, Perdue LA, Rushkin M, Dana T, Blazina I, Bougatsos C, et al. Screening for unhealthy drug use: updated evidence report and systematic review for the US Preventive Services Task Force. JAMA Netw Open. 2020;323(22):2310–28. doi: 10.1001/jama.2019.21381 32515820 Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Int Med. 2001;16(9):606–13. doi: 10.1046/j.1525-1497.2001.016009606.x 11556941 Group WAW. The alcohol, smoking and substance involvement screening test (ASSIST): development, reliability and feasibility. Addiction. 2002;97(9):1183–94. doi: 10.1046/j.1360-0443.2002.00185.x 12199834 Chesney MA, Ickovics JR, Chambers DB, Gifford AL, Neidig J, Zwickl B, et al. Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (AACTG). AIDS care. 2000;12(3):255–66. Hitch AE, Gause NK, Brown JL. Substance use screening in HIV care settings: a review and critique of the literature. Curr HIV/AIDS Rep. 2019;16:7–16. doi: 10.1007/s11904-019-00434-9 30747409 Williams EC, Achtmeyer CE, Thomas RM, Grossbard JR, Lapham GT, Chavez LJ, et al. Factors underlying quality problems with alcohol screening prompted by a clinical reminder in primary care: a multi-site qualitative study. J Gen Int Med. 2015;30:1125–32. doi: 10.1007/s11606-015-3248-z 25731916 Johnson M, Jackson R, Guillaume L, Meier P, Goyder E. Barriers and facilitators to implementing screening and brief intervention for alcohol misuse: a systematic review of qualitative evidence. J Public Health. 2011;33(3):412–21. doi: 10.1093/pubmed/fdq095 21169370 Higgins J.P.T. G. Chapter 18: Patient-reported outcomes. 2011. In: Cochrane Handbook for Systematic Reviews of Interventions [Internet]. The Cochrane Collaboration. Available from: https://training.cochrane.org/handbook/current/chapter-18. Pritchett JC, Patt D, Thanarajasingam G, Schuster A, Snyder C. Patient-Reported Outcomes, Digital Health, and the Quest to Improve Health Equity. American Society of Clinical Oncology Educational Book. 2023;43:e390678. doi: 10.1200/EDBK_390678 37290027 Substance Abuse and Mental Health Service Association. Screening, Brief Intervention and Referral to Treatment (SBIRT) in Behavioral Healthcare [updated April 1, 2011January 9, 2020]. Available from: https://www.samhsa.gov/sites/default/files/sbirtwhitepaper_0.pdf. Gagnon KW, Baral S, Long D, Guzman AL, Johnson B, Burkholder G, et al. Delivery of the HIV Service and Telemedicine Through Effective Patient-Reported Outcomes (+ STEP) Intervention to Increase Screening and Treatment of Mental Health and Substance Use Disorders for People Living With HIV in Alabama: Protocol for an Effectiveness-Implementation Study. JMIR Res Protoc. 2023;12(1):e40470. doi: 10.2196/40470 37581919 Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. doi: 10.1097/MLR.0b013e3182408812 22310560 Kroenke K, Spitzer RL, Williams JB. The PHQ‐9: validity of a brief depression severity measure. J Gen Int Med. 2001;16(9):606–13. doi: 10.1046/j.1525-1497.2001.016009606.x 11556941 Williams N. The GAD-7 questionnaire. Occup Med. 2014;64(3):224–. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA, Project ACQI. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Arch Intern Med. 1998;158(16):1789–95. Bäck A, Sundell KA, Horne R, Landén M, Mårdby A-C. The Medication Adherence Report Scale (MARS-5) in a Swedish sample with bipolar disorder-a pilot study. Int J Pers Cent Med. 2012;2(2):263–70. Lewis CC, Scott K, Marriott BR. A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Imp Sci. 2018;13:1–13. doi: 10.1186/s13012-018-0761-6 29769096 Hyzak KA, Bunger AC, Herrmann SA, Kerlek A, Lauden S, Dudley S, et al. Development of an Implementation Blueprint to Scale-Up Contraception Care for Adolescents with Psychiatric Conditions in a Pediatric Hospital. Global Imp Res App. 2023;3(2):147–61. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76. doi: 10.1007/s10488-010-0319-7 20957426 Sohail M, Rastegar J, Long D, Rana A, Levitan EB, Reed-Pickens H, et al. Data for care (D4C) Alabama: clinic-wide risk stratification with enhanced personal contacts for retention in HIV care via the Alabama quality management group. JAIDS. 2019;82:S192–S8. doi: 10.1097/QAI.0000000000002205 31764254 Eaton EF, Burgan K, McCollum G, Levy S, Willig J, Mugavero MJ, et al. Expanding access to substance use services and mental health care for people with HIV in Alabama, a technology readiness assessment using a mixed methods approach. BMC Health Serv Res. 2022;22(1):919. doi: 10.1186/s12913-022-08280-z 35841096 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. doi: 10.1016/j.jbi.2008.08.010 18929686 Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi: 10.1016/j.jbi.2019.103208 31078660 Neve OM, Van Benthem PPG, Stiggelbout AM, Hensen EF. Response rate of patient reported outcomes: the delivery method matters. BMC Med Res Methodol. 2021;21:1–9. Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305–14. doi: 10.1007/s11136-011-0054-x 22048932 Crane HM, Lober W, Webster E, Harrington RD, Crane PK, Davis TE, et al. Routine collection of patient-reported outcomes in an HIV clinic setting: the first 100 patients. Curr HIV Res. 2007;5(1):109–18. doi: 10.2174/157016207779316369 17266562 Sandhu S, King Z, Wong M, Bissell S, Sperling J, Gray M, et al. Implementation of electronic patient-reported outcomes in routine cancer care at an academic center: identifying opportunities and challenges. J Oncol Pract. 2020;16(11):e1255–e63. doi: 10.1200/OP.20.00357 32926662 Fredericksen R, Harding B, Ruderman S, McReynolds J, Barnes G, Lober W, et al. Patient acceptability and usability of a self-administered electronic patient-reported outcome assessment in HIV care: relationship with health behaviors and outcomes. AIDS care. 2021;33(9):1167–77. doi: 10.1080/09540121.2020.1845288 33190523 Fredericksen R, Crane P, Tufano J, Ralston J, Schmidt S, Brown T, et al. Integrating a web-based, patient-administered assessment into primary care for HIV-infected adults. J AIDS HIV Res. (Online). 2012;4(2):47. doi: 10.5897/jahr11.046 26561537 Fredericksen RJ, Short D, Fitzsimmons E, McReynolds J, Karras S, Lober W, et al. PROgress Implementation Toolkit: Integrating Patient-Reported Outcomes. 2020. 10.1371/journal.pone.0312782.r001 Decision Letter 0 Jahun Ibrahim Academic Editor 2024 Ibrahim Jahun This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

26 Aug 2024

PONE-D-24-13780Developing an Implementation Blueprint: Lessons Learned from Integrating Electronic Patient-Reported Outcomes in HIV Clinics in AlabamaPLOS ONE

Dear Dr. Gagnon,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please address all concerns raised by the reviewers (see below) and in addition, ensure that references and tables adhere to plosone publication requirements. Please submit your revised manuscript by Oct 10 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Ibrahim Jahun, MD, MSc, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Thank you for stating the following financial disclosure: "This study was funded by the National Institutes of Mental Health (PI: Eaton, 5R01MH124633-04 )." Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."" If this statement is not correct you must amend it as needed. Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf. 3. Thank you for stating the following in the Competing Interests section: "E reports honorarium from Gilead for participating in the HIV Re-engagement working group and from PRIME, DKBMed, and IAS-USA for developing HIV continuing education content." Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.  Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf. 4. We note that your Data Availability Statement is currently as follows: All relevant data are within the manuscript and its Supporting Information Files. Please confirm at this time whether or not your submission contains all raw data required to replicate the results of your study. Authors must share the “minimal data set” for their submission. PLOS defines the minimal data set to consist of the data required to replicate all study findings reported in the article, as well as related metadata and methods (https://journals.plos.org/plosone/s/data-availability#loc-minimal-data-set-definition). For example, authors should submit the following data: - The values behind the means, standard deviations and other measures reported;- The values used to build graphs;- The points extracted from images for analysis. Authors do not need to submit their entire data set if only a portion of the data was used in the reported study. If your submission does not contain these data, please either upload them as Supporting Information files or deposit them to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories. If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. If data are owned by a third party, please indicate how others may request data access. 5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript is technically sound and pertinent to improving the quality of care of Person with HIV/AIDS (PWH) because I had previously worked with the author in trying to troubleshoot the best way of improving the health outcome of PWH. The author through this paper has highlighted the critical importance of introducing ePROs to specifically improve the health outcome of PWH particularly those susceptible to poor mental health and substance use.

Reviewer #2: The paper was well written, lesson learnt, and discussion were well aligned. However, the authors need to pay more attention on the use abbreviations. for examples: 1. RWHAP was sometimes written as RHWAP (page 6, 15), this needs to be corrected. 2. SUD - described as substance use instead of substance use disorders. 3. UAB was not described in the paper.

Reviewer #3: 1.Materials and Methods - Lines 103-104 speak of Qualitative Interviews and Structured Surveys. Please provide the respective interview guide and the survey questionnaire. They were not available under the data availability section.

2. Line 214, it will help if the authors could replace "Lessoned Learned" with RESULTS. They can include lessons learnt as a subtitle under RESULTS section.

3. An indication of many site visits was done, will be quite helpful as the authors recommended that there is a need to increase the site visits. So a measurement is needed to show from where this improvement ought to be done.

4. Discussion - It is advisable that the writers restrict their Discussions only to the study findings. They can only discuss what is stated in the results section, except where they are citing other literature in case of findings corroborations.

5. The definition of a Champion should come early, at the first instance of the word champion.

6. References: Please check all your references to ensure they are consistent with the Vancouver style.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Emeka Chrisian Madubuko

Reviewer #2: Yes: Mukhtar Liman Ahmed

Reviewer #3: Yes: Dr. Benson Ncube

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0312782.r002 Author response to Decision Letter 0 Submission Version 1

10 Oct 2024

Response to Reviewers

We sincerely thank the reviewer for their thoughtful and constructive suggestions, which have substantially strengthened our manuscript. Responses to the reviewer’s points are below. The reviewer’s original comments are listed followed by our response. Our responses correspond to changes highlighted in the revised article text and titled ‘Revised Manuscript with Track Changes'. A clean version has also been included, titled 'Manuscript'.

EDITOR COMMENTS TO AUTHOR

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

We have updated the manuscript to meet PLOS ONE’s style requirements.

2. Thank you for stating the following financial disclosure:

"This study was funded by the National Institutes of Mental Health (PI: Eaton, 5R01MH124633-04 )."

Please state what role the funders took in the study. If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.""

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. Thank you for stating the following in the Competing Interests section:

"E reports honorarium from Gilead for participating in the HIV Re-engagement working group and from PRIME, DKBMed, and IAS-USA for developing HIV continuing education content."

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

4. We note that your Data Availability Statement is currently as follows: All relevant data are within the manuscript and its Supporting Information Files.

Please confirm at this time whether or not your submission contains all raw data required to replicate the results of your study. Authors must share the “minimal data set” for their submission. PLOS defines the minimal data set to consist of the data required to replicate all study findings reported in the article, as well as related metadata and methods (https://journals.plos.org/plosone/s/data-availability#loc-minimal-data-set-definition).

For example, authors should submit the following data:

- The values behind the means, standard deviations and other measures reported;

- The values used to build graphs;

- The points extracted from images for analysis.

Authors do not need to submit their entire data set if only a portion of the data was used in the reported study.

If your submission does not contain these data, please either upload them as Supporting Information files or deposit them to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories.

If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. If data are owned by a third party, please indicate how others may request data access.

We have updated the data availability statement to reflect the added data file. The updated data availability statement is in the cover letter and we have uploaded the data file as Supporting Information.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

We have added the Supporting Information files to the end of the manuscript.

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

We have reviewed our reference list to ensure that it is complete, correct, and does not contain any retracted papers.

REVIEWER 1 COMMENTS TO AUTHOR

The manuscript is technically sound and pertinent to improving the quality of care of Person with HIV/AIDS (PWH) because I had previously worked with the author in trying to troubleshoot the best way of improving the health outcome of PWH. The author through this paper has highlighted the critical importance of introducing ePROs to specifically improve the health outcome of PWH particularly those susceptible to poor mental health and substance use.

We appreciate your comment.

REVIEWER 2 COMMENTS TO AUTHOR

The paper was well written, lesson learnt, and discussion were well aligned. However, the authors need to pay more attention on the use abbreviations. for examples: 1. RWHAP was sometimes written as RHWAP (page 6, 15), this needs to be corrected. 2. SUD - described as substance use instead of substance use disorders. 3. UAB was not described in the paper.

Thank you for your comment! We have corrected these abbreviations throughout.

REVIEWER 3 COMMENTS TO AUTHOR

1.Materials and Methods - Lines 103-104 speak of Qualitative Interviews and Structured Surveys. Please provide the respective interview guide and the survey questionnaire. They were not available under the data availability section.

Thank you for this comment. Lines 103-104 were intended to describe the overall intervention and not the methods of this manuscript specifically. However, we agree that this is confusing to the read so have removed the aims of the larger intervention from the methods section.

2. Line 214, it will help if the authors could replace "Lessoned Learned" with RESULTS. They can include lessons learnt as a subtitle under RESULTS section.

We have made this correction.

3. An indication of many site visits was done, will be quite helpful as the authors recommended that there is a need to increase the site visits. So a measurement is needed to show from where this improvement ought to be done.

Thank you for this comment. We have added this information on lines 271 to 274.

“Thus far, we have conducted two site visits to every site, one to launch the intervention (between April 2022 and March 2023) and second to check in, meet new staff, and discuss barriers to the intervention between May and August 2023.”

4. Discussion - It is advisable that the writers restrict their Discussions only to the study findings. They can only discuss what is stated in the results section, except where they are citing other literature in case of findings corroborations.

We agree with the reviewer’s sentiment regarding the scope of discussions but were not able to identify anywhere in our discussion that goes outside of the scope of our results or cited literature.

5. The definition of a Champion should come early, at the first instance of the word champion.

Thank you for this suggestion. We removed the initial first mention of champions so that the definition follows the first mention of champion on lines 159-161.

6. References: Please check all your references to ensure they are consistent with the Vancouver style.

Thank you again for the careful assessment. We have further reviewed the references and corrected the journal names to the abbreviated titles.

Submitted filename: Response to Reviewers.docx

 10.1371/journal.pone.0312782.r003 Decision Letter 1 Jahun Ibrahim Academic Editor 2024 Ibrahim Jahun This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

14 Oct 2024

Developing an Implementation Blueprint: Lessons Learned from Integrating Electronic Patient-Reported Outcomes in HIV Clinics in Alabama

PONE-D-24-13780R1

Dear Dr. Gagnon,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ibrahim Jahun, MD, MSC, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

 10.1371/journal.pone.0312782.r004 Acceptance letter Jahun Ibrahim Academic Editor 2024 Ibrahim Jahun This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

18 Oct 2024

PONE-D-24-13780R1

PLOS ONE

Dear Dr. Gagnon,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Ibrahim Jahun

Academic Editor

PLOS ONE