PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0308375 PONE-D-24-24617 Study Protocol Physical sciencesMathematicsStatisticsStatistical data Research and analysis methodsResearch designSurvey researchQuestionnaires Research and analysis methodsResearch designQualitative studies Medicine and health sciences Medicine and health sciencesEpidemiologyMedical risk factors Research and analysis methodsResearch designSurvey researchSurveys Science policyScience and technology workforceCareers in researchScientists People and placesPopulation groupingsProfessionsScientists Biology and life sciencesPsychologyPsychological attitudes Social sciencesPsychologyPsychological attitudes Exploring the potential of large language models for integration into an academic statistical consulting service–the EXPOLS study protocol The EXPOLS study protocol https://orcid.org/0000-0002-4190-8969 Fichtner Urs Alexander Conceptualization Data curation Formal analysis Investigation Methodology Project administration Software Validation Writing – original draft 1 2 * Knaus Jochen Conceptualization Data curation Project administration Resources Visualization Writing – review & editing 2 https://orcid.org/0000-0002-6831-0756 Graf Erika Conceptualization Supervision Writing – review & editing 2 Koch Georg Project administration Software 2 https://orcid.org/0000-0003-3023-3318 Sahlmann Jörg Conceptualization Writing – review & editing 2 https://orcid.org/0000-0001-7970-6388 Stelzer Dominikus Conceptualization Writing – review & editing 2 Wolkewitz Martin Supervision Writing – original draft 2 https://orcid.org/0000-0002-5666-8662 Binder Harald Funding acquisition Writing – review & editing 2 https://orcid.org/0000-0002-4418-3836 Weber Susanne Conceptualization Data curation Formal analysis Investigation Methodology Project administration Validation Writing – original draft 2 Faculty of Medicine and Medical Center, Institute of Medical Biometry and Statistics, Section for Healthcare Research and Rehabilitation Research (SEVERA), University of Freiburg, Freiburg, Germany Faculty of Medicine and Medical Center, Institute of Medical Biometry and Statistics, University of Freiburg, Freiburg, Germany Moges Bekalu Tadesse Editor Woldia University, ETHIOPIA

The authors have declared that no competing interests exist.

 * E-mail: Urs.Fichtner@uniklinik-freiburg.de 4 12 2024 2024 19 12 e0308375 18 6 2024 23 7 2024 2024 Fichtner et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Background

The advancement of Artificial Intelligence, particularly Large Language Models (LLMs), is rapidly progressing. LLMs, such as OpenAI’s GPT, are becoming vital in scientific and medical processes, including text production, knowledge synthesis, translation, patient communication and data analysis. However, the outcome quality needs to be evaluated to assess the full potential for usage in statistical applications. LLMs show potential for all research areas, including teaching. Integrating LLMs in research, education and medical care poses opportunities and challenges, depending on user competence, experience and attitudes.

 Objective

This project aims at exploring the use of LLMs in supporting statistical consulting by evaluating the utility, efficiency and satisfaction related to the use of LLMs in statistical consulting from both advisee and consultant perspective. Within this project, we will develop, execute and evaluate a training module for the use of LLMs in statistical consulting. In this context, we aim to identify the strengths, limitations and areas for potential improvement. Furthermore, we will explore experiences, attitudes, fears and current practices regarding the use of LLMs of the staff at the Medical Center and the University of Freiburg.

 Materials and methods

This multimodal study includes four study parts using qualitative and quantitative methods to gather data. Study part (I) is designed as mixed mode study to explore the use of LLMs in supporting statistical consulting and to evaluate the utility, efficiency and satisfaction related to the use of LLMs. Study part (II) uses a standardized online questionnaire to evaluate the training module. Study part (III) evaluates the consulting sessions using LLMs from advisee perspective. Study part (IV) explores experiences, attitudes, fears and current practices regarding the use of LLMs of the staff at the Medical Center and the University of Freiburg. This study is registered at the Freiburg Registry of Clinical Studies under the ID: FRKS004971.

 http://dx.doi.org/10.13039/501100002714 Albert-Ludwigs-Universität Freiburg This study is financed by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 499552394 – SFB 1597. The recipient of the funding award is Harald Binder. We acknowledge support by the Open Access Fund of the University of Freiburg. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability Data collection has not started yet. When data will have been collected, it will be published on a public repository. 1 Introduction

The progress of Artificial Intelligence (AI), especially in the area of Large Language Models (LLMs) is currently accelerating at an ever-faster pace [1]. LLMs are artificial neural networks trained by using self- and semi-supervised learning. Popular examples are OpenAI’s GPT models, Google’s PaLM, DeepMinds Gemini and Meta’s LLaMA and Microsoft’s KOSMOS [2].

LLMs become more and more attractive in various contexts of the scientific process, e.g. supporting the production of scientific texts, knowledge synthesis, translation–but also data analysis. In the medical context, the use of LLMs has especially been examined in order to support the communication with patients [3–5]. Not only in the context of medical research, the integration of LLMs is a cutting-edge advancement that is revolutionizing research across various fields. Garrel et al. examined the use of AI at a higher education landscape via a survey among students in Germany [6]. LLMs can take over "roles", e.g. those of programmers or data analysists using statistical software, e.g. Python or R, and can answer to questions that deal with problems of data analysis. It is notable that LLMs have the capability to perform tasks beyond their original intended purposes. Dell’Acqua et al. investigated how access to ChatGPT 4 impacts productivity and quality of consultants within a global management consulting company [7]. LLM tools can act as "statistical consultant" supporting to find answers and solutions for analysis strategies, programming issues and bug fixing. In order to evaluate the quality of statistical programming code generated by ChatGPT, Liu et al. conducted a systematic analysis of 4,066 ChatGPT-generated code snippets for 2,033 programming tasks in Java and Python [8]. They found that in every fourth case, the produced output was wrong. Their experiments suggest that ChatGPT showed potential for self-repairing of code, however they conclude that it still has limitations in its current state. Their study evaluated the potential of ChatGPT 3.5 which was trained with much less data than its successor ChatGPT 4. The newest GPT model of OpenAI was released in March 2023 and was trained with 175 billion training parameters. Furthermore, a new plugin, called "Advanced Data Analysis" (ADA) was released for ChatGPT 4 that provides specific features for data evaluation. After an update, this feature is now available as GPT called “Data Analyst”. With this feature, users can upload their data and let the system analyse them by using textual questions (called “prompts”). ADA introduces an extensive range of new application areas. For instance, Müller et al. investigated opportunities and limitations of this plugin for hydrological analysis [9]. However, both the use of LLMs as "consultant", as well as "data analyst" face several risks [10,11]. First, the quality of the outcomes of LLMs has to be verified and might depend on several aspects, as e.g. the quality of training data, the difficulty of the question and the wording of the prompts. Second, the quality of the outcomes needs to be validated and verified and it stands to question whether the outcome quality can be judged in a general manner. As long as the outcome quality and precision of the operations of the LLM is not guaranteed, it might produce errors of different levels of severity. Third, without deeper understanding and education, the advice-seeking person might not be able to correctly differentiate between invalid and proper answers or analysis results. Fourth, uploading data onto a server can be problematic for reasons of data protection and limits the use of ADA. Ignjatovic et. al investigated the use of ChatGPT 3.5 and 4 in order to solve biostatistical problems. They conclude that with ChatGPT 4 a good level of performance can be obtained, but the usage of such tools for solving biostatistical problems should be done with caution [12].

At the Medical Faculty of the University of Freiburg, statistical consulting is offered to every member of the faculty who searches supervision or support of questions related to data analysis. In some exceptional cases, even the whole analysis process is executed by the statisticians employed there. Since many medical researchers have limited education in data science, this consultation is offered to support good scientific practice using adequate analysis methods by well-trained field experts. Depending on the statistical expertise of the advisees and the complexity of the analysis questions, the consultation offer varies between a unique e-mail-based advice over two or three face-to-face or web-conference consulting sessions to possibly even programming meetings in presence of the advisee. In order to sustain high quality in statistical consulting, a standardized training and orientation concept has been developed for general quality improvement. This concept has been mandatory to be absolved for consultant beginners since 2019. Additionally, regular bi-weekly meetings are held for collegial exchange, where consultation cases are presented for joint discussion of consulting practices.

It is foreseeable that LLMs will play a role in the future handling of data analysis and thus, in the procedures that are aligned to statistical consultation. Both advice-seeking scientists and statistical field experts might make use of such tools to support their work. We anticipate that, in a best case scenario, the use of a LLM will enable consultants to significantly improve their efficiency in managing consulting tasks. Consultants could benefit from a shift of their working duties from programming tasks to more conceptual thinking tasks. Published studies point out this potential, however, successful support in terms of being more efficient when performing data analysis tasks using LLMs still stands to question [8,13]. Critical output assessment and learning in how to formulate prompts seems to be crucial.

The question of how to integrate LLMs in research is, however, not limited to the medical field. LLMs can be used for text and code production and thus, shows potential for all research areas, including teaching. Rahman & Watanobe addressed opportunities, threats, and strategies of ChatGPT for education and research [14]. However, many contextual factors might play a role in order to determine a qualified use of LLMs with high efficiency, utility and user satisfaction. Those factors might lie in the persons interacting with the systems, e.g. level of education in statistics, competence and experience in using LLMs, attitudes and fears towards LLMs and personal preferences. Additionally, also factors within the technology used might play an important role, e.g. the outcome quality of the LLM, its interoperability, its flexibility, its transparency, the repeatability of the outcomes as well as its comparability and ease of use. Gärtner et al. investigated the use, benefits, and limitations of ChatGPT and Large Language Models (LLMs) in teaching at Bavarian universities of applied sciences [15]. They found that the usage of LLMs in teaching, studying and daily professional life is still low due to the lack of application possibilities, poor quality of results and legal concerns of the potential users. On the other side, respondents viewed the use of LLMs for teaching as an opportunity and expressed the need for support by their employers.

1.1 Statistical consulting

Many statistical research organizations offer methodological support to clinical and life science researchers. This may happen in individual consultations without further involvement, in project-specific co-operations or in long-term collaborations among domain experts and biostatisticians [16]. Consultation can be provided on aspects of study design and methodological evaluation, and the advice given can refer to basic research or to clinical research projects.

At the Institute of Medical Biometry and Statistics (IMBI), a free consulting service is operated by a cross-section of all employed statisticians and mathematicians, with more than 20 IMBI scientists providing consulting in addition to their research activities. Over 250 requests for advice are processed each year. Those seeking advice are members of the Medical Faculty of the University of Freiburg or the Freiburg University Medical Center. The advisees range from students with questions about doctoral projects and postdocs to experienced research group leaders and principal investigators. The contents addressed range from study planning including sample size calculation and planning of the statistical analysis in a study protocol through application of statistical methodology in a given data set.

At the IMBI, and hence within the EXPOLS project, consulting services are defined as follows:

"Advising in a single or a few advising session(s) (sometimes called consulting service), where the advisee poses questions on statistical aspects in a particular project and the adviser discusses possible solutions but does not further engage in the project." [16]

In particular, no further engagement means that no data analysis is performed by the consultant. As there is a wide range in the research areas, research questions and statistical expertise of advisee, there is a huge heterogeneity between the consulting sessions.

 1.2 The integration of a LLM into statistical consulting

At the IMBI, statistical consulting is partly standardized. Every advisee fills out a template providing information on the project and their requests. Then, the administrative team of the statistical consulting group assigns the case to a consultant based on the best matching expertise. How the consultation is operated is not prescribed and varies from case to case and from consultant to consultant. Usually the consultants get in touch with the advisees and, if necessary, meet with them (online). To enhance the advice provided by consultants, retrieving information from literature or utilizing other informative materials from the internet can be instrumental. Depending on the request and the expertise of the consultant, more or less preparation is necessary. In some cases, written feedback is also sufficient. In the context of this study, each statistical consultant has received access to ChatGPT 4 and is advised to integrate the LLM in the consultations. Due to the considerable heterogeneity among consulting sessions, the tasks for which the LLM can be used will also vary. Potential uses of the LLM include pre- or post-session preparation, real-time assistance, data analysis guidance, drafting documentation, serving as an educational tool, and many more. Consultants are not instructed to use ChatGPT in a specific manner or for specific tasks and possible scenarios could be that the consultant integrates the LLM as a third person during the consulting session, that the consultant uses it for producing analysis code, that the consultant uses it for summarizing scientific texts, and so on. One aim of this project is to assess how this integration works and how it can support statistical consulting.

 2 Materials and methods 2.1 Research questions

Within this project, we aim to answer the following research questions:

How can LLMs be used to support statistical consulting?

1.1 How do advisees and consultants perceive and rate the use of LLMs in statistical consulting?

1.2 What are the strengths, limitations and areas of improvement in the use of LLMs for statistical consulting?

1.3 What factors play a role in determining the use of LLMs with high efficiency, utility and user satisfaction?

Is the developed training module for statistical consultants and (future) scientists efficient and sustainable, as measured by the quality and relevance feedback from the participants?

What are the experiences, attitudes, fears, beliefs and current practices regarding the use of LLMs?

 2.2 Primary and secondary outcomes

This study is divided into four sub-studies using different methods to gather data. Each sub-study addresses one or more research questions and thus has its own measurement goals und study population (see section Study population). However, as this is an exploratory study, no hypotheses will be formulated in advance and thus, no clearly prioritized outcome measures can be derived. The outcome measures are displayed by study part in subsequent Table 1.

10.1371/journal.pone.0308375.t001 Study outcomes.
Study part Population Outcomes Research question
I: Mixed mode study using qualitative and quantitative methods (online, standardized and online, semi-structured) Statistical consultants(Institute of Medical Biometry and Statistics) Quantitative:Result expectationResult achievementSatisfaction with use of LLMQuality AssessmentCommunication AssessmentEfficiency AssessmentFuture potentialPerceived risks and limitationsQualitative:Result expectationPerceived barriersPerceived potentialPossible pitfalls and solutionsExperiences 1, 1.1, 1.2, 1.3
II: Quantitative standardized online questionnaire Statistical consultants(Institute of Medical Biometry and Statistics) Appropriateness of the time frame of the training moduleStructure of the content of the coursePerceived value of contentPerceived learning successQuality in communicationMotivation of the coursePreparation quality of the courseWillingness to recommend the courseUse of training session for future consulting tasksStrengths and limitations, improvement potentialOverall quality rating of course 2
III: Quantitative standardized online questionnaire Advisees(Clinicians and domain experts) Evaluation of the absolved consulting session (captured via regular evaluation form as a score and categorised)Satisfaction with the use of LLM in consultationQuality assessmentCommunication assessmentEfficiency assessment 1.1
IV: Quantitative standardized online questionnaire Scientific employees(researchers and clinician scientists at the University Medical Center and the Albert-Ludwigs-University Freiburg) Attitudes and beliefs towards the use of LLMExperiences with the use of LLMCurrent usage of LLM and contextsPerceived benefits in using LLMPerceived barriers in using LLMKnowledge of or participation in development of guidelines / rules towards the use of LLMChanges in practice in line with the introduction of LLMDifferences across faculties and research fields 3

The schedule of enrolment, interventions and measurements covering all measurement time points for the study samples according to the SPIRIT guideline is displayed in Fig 1.

10.1371/journal.pone.0308375.g001 Schedule of enrolment, interventions, and measurements. 2.3 Study design

This monocentric study will utilize qualitative and quantitative methods to explore the use of LLM in an academic statistical consulting service from both consultant and advice-searching staff. Each of the four sub-studies addresses different research questions and is organized in working packages (WP) (see Table 2).

10.1371/journal.pone.0308375.t002 Working packages and time table.
Working package (WP) Month
  I II III IV V VI VII
WP 1: Development of training—course conceptualization              
WP 2: Training session with all statistical consultants              
WP 3: Development of questionnaires, instruments, interview guideline              
WP 4: Pre-training survey among consenting consultants              
WP 5: Semi-structured qualitative interviews with consultants              
WP 6: Analysis of qualitative interviews and synthesis of results              
WP 7: Post-training survey among consultants (T1: one week after training, T2:three months, T3: six months)              
WP 8: Continuous post-consulting survey among advisees              
WP 9: Survey among all members of the Medical Faculty / Medical Center              
WP 10: Analysis and preparation of findings, knowledge synthesis              
WP 11: Dissemination of results, data and evaluation of need for further research (Study part II, IV)              
WP 12: Analysis and preparation of findings for follow-up (Study part I, III)              

WP 1: To develop the training module, existing teaching material from online resources, prior course visits and own preparatory works will be consolidated. The course consists of a lecture part and a hands-on part. The whole teaching session is estimated to take approximately 2–3 hours. Sustainable reuse potential of the course will guide the development of the training module.

WP 2: The course will be offered in a synchronous hybrid format to enable participants to take part from home office. We offer up to three different dates to ensure that every statistical consultant can take part in the training session. Participation will be mandatory as part of consultants’ work-related duties. The course will be offered to all consultants employed at the IMBI and is not restricted to those who participate in the study. The course will be conducted by SW, the principle investigator of this project.

WP 3: For all study parts, we will make use of standardized online questionnaires implemented via REDCap. Those questionnaires will be constructed based on existing literature and earlier used items and scales in other, comparable contexts. The questionnaires will be offered in bilingual form (GER, ENG). The interview guideline applied in study part I will be developed within a consortium of field-experts from Sociology, Empirical Social Science, Statistics, Health Services Research, Medicine and Mathematics.

WP 4: Before the training session, we will ask consenting statistics consultants to fill out a short questionnaire provided via REDcap on their expectations about the use of LLM in statistical consulting. Furthermore, we will collect data on potential predictors that might have influence on the adaptation and use of LLM in statistical consulting.

WP 5: With a purposive sampling approach, we will select consultants and will conduct semi-structured guideline-based qualitative interviews to gather in-depth insights into their perceptions regarding the use of LLM in statistical consulting, the associated barriers, limitations and potential benefits. The interviews will be conducted via bilateral Webex-Meetings. The interviews will be digitally recorded.

WP 6: All interviews of WP 5 will be transcribed verbatim and analyzed qualitatively by SW and UF following a parallel coding approach according to Kuckartz [17]. The results will be used to refine survey instruments that will be used in WP 7 and 9.

WP 7: After all consultants participated in the training session, they are asked to actively involve LLM in their subsequent consultation sessions. At t1 (one week after training session), t2 (two months after training session) and t3 (6 months after after training session), the consultants will receive a standardized online questionnaire provided via REDcap.

WP 8: All advisees participating in the study will be asked to fill out an evaluation questionnaire after their consultation session. This questionnaire is attached as a link to the regular evaluation of the statistical consulting session and is executed as a separate REDCap form. The questionnaire also captures whether or not a LLM was used in the consulting session and what the reasons for non-use were. The survey will also contain questions regarding the advisee’s perception towards the use of LLM during the session.

WP 9: Scientific employees at the University Medical Center and the Albert-Ludwigs-University Freiburg will be asked to complete a standardized online questionnaire provided via REDCap on a voluntary basis. For recruiting, we will use several different methods to ensure reaching all staff involved in research. The questionnaire will be online for 6 weeks.

WP 10: The baseline data of study parts I to IV will be analyzed exploratory and in a descriptive manner. Potential correlations or group-differences will be tested bivariately.

WP 11: The quantitative data and the results will be published in adequate scientific journals (see data publication) and needs for further research will be evaluated.

WP 12: The follow-up data of study parts I and III will be analyzed exploratory and in a descriptive manner. Results will be published in adequate scientific journals.

A detailed overview of the study recruitment workflow is illustrated in the Section "Data management plan".

 2.4 Study population

The study population varies and thus is described for each study part.

Study part I (Mixed mode study using qualitative and quantitative methods (online, standardized, semi-structured)):

Inclusion criteria: All potential study participants must be members of the statistical consulting team at the time of data collection and have signed a declaration of informed consent. The statistical consulting team consists of 24 consultants. Assuming a participation rate of 80–90% the estimated sample size for the quantitative part is n = 20; estimated sample size for the qualitative part: n = 6.

Due to the interest already expressed among the consultants, the assumed participation rate seems realistic. We expect a high level of intrinsic motivation based on professional involvement. Participation can take place during regular working hours so that no additional private time resources need to be invested. In order to ensure a high participation rate we implement repeated reminders.

Study part II (Quantitative standardized online questionnaire):

Inclusion criteria: All potential study participants must be members of the statistical consulting team at the time point of data collection and have given electronic informed consent. Similar to study part I, the estimated sample size is n = 20.

Study part III (Quantitative standardized online questionnaire):

Inclusion criteria: All potential study participants must have absolved at least one statistical consulting between the date of the teaching of the consultants and the end of the observation period. Electronic informed consent needs to be obtained in advance. There are at least 15 consultations per month. With a data collection period of 6 months this results in at least 90 consultations. Assuming a participation rate of 33% the estimated sample size is n = 30.

Study part IV (Quantitative standardized online questionnaire):

Inclusion criteria: All potential study participants must be employed as scientific or medical staff at the University of Freiburg or the University Medical Center or must work closely with science. The declaration of consent is integrated into the start of the online survey. The source population consists of approximately 5,800 scientific employees. In addition, employees working closely to science, but not officially declared as scientists, are also invited to participate. Assuming a response rate of 10–20% the estimated sample size is 600–1,200.

Exclusion criteria: All members that do not fulfil the inclusion criteria will be excluded. Furthermore, potential participants will be excluded if they do not have sufficient language skills in either German or English.

 2.5 Recruitment

Study participants for study part I and II are employed in our institute and will be contacted via their associated company e-mail address. Project members are part of the study population and can participate both in the qualitative and the quantitative part. The principal investigators have no authority to issue directives within the consulting team. There are no conflicts of interest, as participation in the study is voluntary and no supervising dependencies are apparent. Participation or non-participation in the study has no impact on the employment. To ensure this, the questionnaires are treated confidential.

For the qualitative study, a purposive sampling approach will be chosen and participants will be recruited by personal contact.

Study participants for study part III will be recruited via the statistical consultant.

Study participants for study part IV will be recruited via multiple media, e.g. newsletters, as well as newsfeeds, e.g. of the University Medical Center.

 2.6 Instruments

The following data / instruments will be collected / used (Table 3).

10.1371/journal.pone.0308375.t003 Instruments used for EXPOLS.
Study Part Data / Instrument
I Experience as consultantSexTypical consultation formatReferences used for consultationExperience with use of AI for statisticsUsability of AI (UMUX; [18]) covering effectiveness, efficiency and satisfactionTechnology Readiness (TRI 2.0; [19])• Optimism (4 items)• Innovation (4 items)• Discomfort (4 items)• Insecurity (4 items)Result expectation [20]• Performance Outcome Expectation (POE) (3 Items)• Self-Evaluative Outcome Expectation (SEOE) (3 Items)• Social Outcome Expectation (SE) (3 Items)Perceived benefits and disadvantages in using AIUsage of ChatGPT after training sessionReasons for non-usage
II Learning successCompetence gainGeneral teaching competencePotential use of ChatGPTPreferred teaching modeOverall ratingRecommendation potential: Net Promoter Score [21]
III Use of AI during consultationSatisfaction with use of AI during consultationUsability of AI (UMUX; [18])Perceived benefits and disadvantages in using AI
IV Status groupExperience in scienceFaculty membershipSexExperience using AI: areasPotential use of AI in the future: areasDifficulties using AITechnology Readiness (TRI 2.0; [19])• Optimism (4 items)• Innovation (4 items)• Discomfort (4 items)• Insecurity (4 items)Result expectation [20]• Performance Outcome Expectation (POE) (3 Items)• Self-Evaluative Outcome Expectation (SEOE) (3 Items)• Social Outcome Expectation (SE) (3 Items)Experience and potential for training sessions with the topic AIExperience in using AI for teachingFears about the role of AI in the futureKnowledge, use and need for guidelines regarding AIUse of AI in the patient-provider relationship (only medical staff)
2.7 Risk-benefit assessment

Individual benefit associated with study participation:

Study part I, II: The study participants are trained in the use of LLMs and can therefore develop a differentiated opinion on integrating modern AI tools for their work.

Study part III: Study participants learn about potential benefits and risks for their own methodological data analysis under expert guidance.

Study part IV: There is no individual benefit to participating in the anonymous survey.

Burdens and risks associated with study participation:

Study part I, II: Familiarization with new tools is always necessary for scientific staff. The additional workload is minimal due to potentially longer counselling sessions.

Study part III: there is a risk of potential minimal increased time effort in statistical consultation.

Study part IV: no risk.

 2.8 Sample size

Since this study is of explorative nature and we aim to conduct a survey with the full basic population, sample size calculation is not applicable.

Study part I, II: statisticians (consultants): the sample size planning is based on the current consultants (N = 24). Assuming a participation rate of 80–90%, we expect data of n = 20 statisticians.

Study part III: domain experts (advisees): considering January until May 2023 as reference over 15 consultations per month can be expected. With a duration of 6 months for data collection of this subpopulation and an expected response rate of 33%, we expect data of n = 30 domain experts (advisees).

Study part IV: scientific employees: 2,623 scientists were employed 2022 at the University and 3,189 persons were employed in 2022 with association to the Medical Faculty working in science. In addition, employees working closely to science, but are not officially declared as scientists, are also invited to participate. Assuming a response rate of 10–20% the estimated sample size is 600–1,200.

 2.9 Statistical methods

Analyses will be explorative.

Qualitative analysis:

Qualitative data from study part I will be transcribed verbatim and then will be analyzed with the structuring content analysis approach [17].

Free-text responses from study part II-IV will be thematically analyzed to identify common themes and impressions.

Quantitative analysis:

Data collected through Likert-scale or multiple-choice questions will be described using descriptive statistics. Absolute frequencies will be provided for the number of consultations and consultants. The number of evaluations will be presented for both consultations and consultants (absolute and relative). Ratings of consultations through the evaluation form will be described using arithmetic mean with a 95% confidence interval, standard deviation, minimum, 25th percentile, median, 75th percentile, maximum, and the count of complete and missing values.

For the pre-post comparison of ratings, Mann-Whitney U tests will be employed, since we expect a small sample size. In this case, the distributive assumptions for parametric approaches will probably be violated and non-parametric procedures are favorable [22,23].

 2.10 Data management plan

The study recruitment workflow is illustrated in the Figs 2–4.

10.1371/journal.pone.0308375.g002 Recruitment scheme study parts I & II. 10.1371/journal.pone.0308375.g003 Recruitment scheme part III. 10.1371/journal.pone.0308375.g004 Recruitment scheme part IV.

General data in statistical consulting and use of the "ChatGPT" service (OpenAI):

As a matter of principle, the consultants do not accept transmission of any research data as part of the statistical counselling at the IMBI; for the majority of cases the activity is purely consultative. However, the consultants are free to enter into a cooperation project with the advisees if they are interested in methodological issues or for other scientific reasons. For data protection reasons, this cooperation is not covered by the general regulations of the statistical consulting and therefore must be evaluated and documented separately as a normal scientific project.

This means that the data on which the consulting activities are based are not available at the IMBI and are not transferred to the Institute.

In the context of the present study, a LLM supports the consulting activities and is subject to the restrictions of statistical consulting, so no data is transferred to the IMBI, stored or processed at the IMBI. There are certain LLMs that can be used in local data centers (e.g. Llama), but these do not yet offer the range of functions of ChatGPT and the Advanced Data Analyses (ADA) function in particular can currently only be used here. The ChatGPT language model used in the study can only be used as a commercial service from a non-European company (OpenAI) in an American data center. The establishment of third-country processing with the conclusion of an order data processing contract with correspondingly extended standard contractual clauses is not intended. This means that the transfer of data relevant to data protection law, i.e. generally clinical study or routine data, to the ChatGPT web service is not permitted within the scope of this study.

The creation of simulation data, which may be transferred for the use of ADA, should be generated under the guidance of the consultant–if ChatGPT is used for simulation data creation, no original patient data must be used. All statisticians are obliged to comply with the standards of the University Medical Center and try to sensitize the advisees so that data is not imported into ChatGPT or other services under data protection regulations outside the counselling process.

 Survey data (quantitative data)

Detailed information and procedures are described in the attached data protection concept.

Study part I-IV: The questionnaires are collected via the REDCap data capture report form system. It is provided by the Clinical Trials Unit (ZKS Freiburg). Technically, the public instance is used, which allows the questionnaires to be used outside the University Medical Center network (essential for parts III and IV with potential participants from pre-clinical institutes and the University). Analysis data sets (cleaned questionnaires) are transferred as CSV files to a network drive of the University Medical Center. Access is restricted to the study evaluators.

Study part I, II: Due to the small number of cases and the known participants (statistical advisory team of the IMBI), there is an increased risk of combinatorial re-identification of using the quantitative data. Here, specific information such as "years of professional experience" is already sufficient for re-identification. This means that the quantitative data sets are excluded from publication and the questionnaire is designed in such a way that it can be used without any consequential risk.

 Survey data–pseudonymization/anonymization

Study part I: participants will be pseudonymized to allow for contact for interviews.

Study part II: participants will be anonymous in the survey (not in the qualitative part).

Study part III: surveys will be pseudonymized in order to combine the study response with the evaluation response of the overall consultation. Only study leaders have access to the pseudonymization list.

Study part IV: participants are anonymous in the survey.

Free text fields will be cleaned before evaluation: any potentially identifying information will be stripped and information will be categorized where specific terms could lead to additional risk of re-identification.

 Consent

This study is registered at the Freiburg Registry of Clinical Studies under the ID: FRKS004971. The ethics committee of the University of Freiburg has approved the study on 11 June 2024 (24-1148-S1).

Electronic informed consents will be obtained for all parts of a multi-part study. For part III, verbal consent will be given for contact during consultations, and questionnaire-related consents are strictly managed, with applications to statistical consulting anonymized one year post-consultation and retained for two years post-study.

 Audio files

Study part I: interviews will be recorded for later transcription. This transcription will be done during the study period. The interview data will be stored on a University Medical Center network drive and is only accessible to the study evaluators.

 Data publication

In line with good scientific practice, the data will be published in public repositories as far as possible.

Study part I-IV: all free text fields will be cleared and categorized. Sensible information in free text will be removed at all.

Study part I, II: no data will be published.

Study part III: study data will be published, but without the connected information on the statistical consulting case.

Study part IV: study data will be published for reuse of the scientific community.

 Status and timeline

In line with the working packages displayed in Table 2, a timeline of 7 months is expected. At the time point of submission of this study protocol, the digitalization of questionnaires for study part I and IV is executed. Furthermore, a concept of the training session is finalized and ready for application. No data was collected so far.

 3 Discussion

With this study, we expect to gather insights into the potential integration of LLMs into statistical consulting using the example of ChatGPT. By this, we are able to investigate the strengths and limitations of using LLMs for statistical consulting in an academic environment and can identify potential usage patterns of our study population. To our knowledge, this is the first study dealing with this topic using different types of data. Furthermore, we aim to explore experiences, attitudes, fears and current practices regarding the use of LLMs of the staff at the Medical Center and the University of Freiburg. The strength of this study concept is that it combines qualitative and quantitative methods and implements them from different perspectives (statistical consultants, advisees and staff of the University and Medical Center). This enables a triangulation of data and might contribute to a better understanding of the studied field. This approach offers the possibility to compensate for the shortcomings of using only either qualitative or quantitative methods and is in line with the Mixed-Methods principle. By approaching our research questions using different methods, we expect to reach for the broad purposes of breadth and depth of understanding and corroboration [24].

However, the study design is limited in its generalisability. It is unclear whether the setting of statistical consulting at other college and university organizations is similar to the IMBI. Depending on the response rates, we can at least draw conclusions based on surveying the full basic population at Freiburg. Furthermore, this small basic population limits potentially applicable statistical methods, especially if response rates will be lower than expected. Our high expected response rates lie within the anticipated importance of the topic and feedback we received when planning the study. However, it is debatable how realistic these assumptions are. We will take all non-monetary measures, e.g. reminders and broad communication of the survey parts via various media, to increase the response rate. Especially study part IV might benefit in its representativeness from rolling out the survey across different Universities on national or even international level. Within the given timeframe, this was not executable. The conceptualization of a cross-institutional study would require a longer preparation period with external funding. However, we will publish all material used for potential replications in the future. By using administrative data on professions and numbers of employees, we can compare our collected data with the real distribution at the University and Medical Center of Freiburg and are able to draw conclusions about coverage.

 Supporting information SPIRIT 2013 checklist: Recommended items to address in a clinical trial protocol and related documents*.

(DOC)

 Original study protocol.

(PDF)

 References De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11: 1166120. doi: 10.3389/fpubh.2023.1166120 37181697 Naveed H, Khan AU, Qiu S, Saqib M, Anwar S, Usman M, et al. A Comprehensive Overview of Large Language Models. 2023 [cited 13 Dec 2023]. doi: 10.48550/ARXIV.2307.06435 Montastruc F, Storck W, de Canecaude C, Victor L, Li J, Cesbron C, et al. Will artificial intelligence chatbots replace clinical pharmacologists? An exploratory study in clinical practice. Eur J Clin Pharmacol. 2023;79: 1375–1384. doi: 10.1007/s00228-023-03547-8 37566133 Witzsch UKF, Borkowetz A, Enzmann T, Rodler S, Leyh-Bannurah S-R, Loch T, et al. Digitalisierung in der Urologie–Herausforderung und Chance. Urologie (Heidelberg, Germany). 2023;62: 913–928. doi: 10.1007/s00120-023-02154-w 37606658 Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023;183: 589. doi: 10.1001/jamainternmed.2023.1838 37115527 Garrel J, Mayer J, Mühlfeld M. Künstliche Intelligenz im Studium Eine quantitative Befragung von Studierenden zur Nutzung von ChatGPT & Co. 2023; 84. doi: 10.48444/H_DOCS-PUB-395 Dell’Acqua F, McFowland E, Mollick ER, Lifshitz-Assaf H, Kellogg K, Rajendran S, et al. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. SSRN Journal. 2023 [cited 5 Jan 2024]. doi: 10.2139/ssrn.4573321 Liu Y, Le-Cong T, Widyasari R, Tantithamthavorn C, Li L, Le X-BD, et al. Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues. ACM Trans Softw Eng Methodol. 2024;33: 1–26. doi: 10.1145/3643674 Irvine DJ, Halloran LJS, Brunner P. Opportunities and limitations of the ChatGPT Advanced Data Analysis plugin for hydrological analyses. Hydrological Processes. 2023;37: e15015. doi: 10.1002/hyp.15015 Yu P, Xu H, Hu X, Deng C. Leveraging Generative AI and Large Language Models: A Comprehensive Roadmap for Healthcare Integration. Healthcare (Basel, Switzerland). 2023;11. doi: 10.3390/healthcare11202776 37893850 Guo Z, Jin R, Liu C, Huang Y, Shi D, Supryadi, et al. Evaluating Large Language Models: A Comprehensive Survey. 2023 [cited 13 Dec 2023]. doi: 10.48550/ARXIV.2310.19736 Ignjatović A, Stevanović L. Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. J Educ Eval Health Prof. 2023;20: 28. doi: 10.3352/jeehp.2023.20.28 37840252 Nejjar M, Zacharias L, Stiehle F, Weber I. LLMs for Science: Usage for Code Generation and Data Analysis. arXiv; 2024. Available: http://arxiv.org/abs/2311.16733. Rahman MdM, Watanobe Y. ChatGPT for Education and Research: Opportunities, Threats, and Strategies. Applied Sciences. 2023;13: 5783. doi: 10.3390/app13095783 Gärtner C, Moraß A, Koss S, Garbusa S, Matern S, Innermann I, et al. Einsatz, Nutzen und Grenzen von ChatGPT und anderen Large Language Modellen an den bayerischen HAWs. Die Studien- und Schriftenreihe des Forschungs- und Innovationslabors Digitale Lehre–FIDL; 5. FIDL–Forschungs- und Innovationslabor Digitale Lehre; 2024 p. 4301 KB, 111 pages. doi: 10.34646/THN/OHMDOK-1466 LeBlanc M, Rueegg CS, Bekiroğlu N, Esterhuizen TM, Fagerland MW, Falk RS, et al. Statistical advising: Professional development opportunities for the biostatistician. Statistics in Medicine. 2022;41: 847–859. doi: 10.1002/sim.9290 35194815 Kuckartz U. Qualitative Inhaltsanalyse: Methoden, Praxis, Computerunterstützung. 4. Auflage. Weinheim Basel: Beltz Juventa; 2018. Finstad K. The Usability Metric for User Experience. Interacting with Computers. 2010;22: 323–327. doi: 10.1016/j.intcom.2010.04.004 Parasuraman A, Colby CL. An Updated and Streamlined Technology Readiness Index: TRI 2.0. Journal of Service Research. 2015;18: 59–74. doi: 10.1177/1094670514539730 Niederhauser DS, Perkmen S. Beyond self-efficacy: Measuring pre-service teachers’ Instructional Technology Outcome Expectations. Computers in Human Behavior. 2010;26: 436–442. doi: 10.1016/j.chb.2009.12.002 Mandal PC. Net promoter score: a conceptual analysis. IJMCP. 2014;8: 209. doi: 10.1504/IJMCP.2014.066899 Okeh UM. Statistical analysis of the application of Wilcoxon and Mann-Whitney U test in medical research studies. Biotechnology and Molecular Biology Reviews. 2009;4: 128–131. Gibbons JD, Chakraborti S. Comparisons of the Mann-Whitney, Student’s t, and Alternate t Tests for Means of Normal Distributions. The Journal of Experimental Education. 1991;59: 258–267. doi: 10.1080/00220973.1991.10806565 Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a Definition of Mixed Methods Research. Journal of Mixed Methods Research. 2007;1: 112–133. doi: 10.1177/1558689806298224 10.1371/journal.pone.0308375.r001 Decision Letter 0 Moges Bekalu Tadesse Academic Editor 2024 Bekalu Tadesse Moges This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

16 Jul 2024

PONE-D-24-24617Exploring the Potential of Large language models for integration into an academic Statistical Consulting Service – The EXPOLS Study ProtocolPLOS ONE

Dear Dr. Fichtner,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 30 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Bekalu Tadesse Moges

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that funding information should not appear in any section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript.

3. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript. 

4. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 3 in your text; if accepted, production will need this reference to link the reader to the Table.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

Dear Author(s), You need to consider improving the quality of the protocol, as suggested by the reviewers.

1. It is rationalized that training material to integrate LLMs into statistical results is needed. However, the second research question related to the development of such material is formulated. I think the material needs to be evaluated, based on the data collected, in terms of its quality and relevance for the advisee and consulting jobs. Thus, I suggest this very question needs revision.

2. if the project needs mixed-methods study, it is better to choose or develop a mixed methods design as appropraite, that fits into your study conceptualization. the mixed methods should allow genuine integration of both qualitative and quantitative data (Teddlie and Tashakorri, 2009, Creswell, 2020)

3. Why is the Mann-Whitney U test selected? it is a non parametric equivalent to the independent samples t-test, used when data doe not meet the assumptions of parametric test. please look into it. (Note that, If you argue otherwise to the comments, they should be convincing, and you are welcome)

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Major Points of Review

1. The title is interesting and ideal, make it summarize the specific

Part 1: Introduction

1. The introduction does not clearly articulate the problems of data analysis or the rationale for this specific study regarding the Potential of Large language models for integration into an academic Statistical Consulting Service. For example, it should explain the problem of data quality, the problem of statistical methods, and the gap of statistical software or another specific purpose. The novelty of large language models should be articulated, highlighting any advancements.

Part 2: The integration of an LLM into statistical consulting.

1. The integration of LLM into statistical consulting is not well explained for the pre-session preparation, data analysis guidance, and real-time assistance as well as assessing integration.

2. Some of the research questions are not measurable. For instance, how to measure, efficiency, utility, and user satisfaction?

Study Design

1. Why incorporate qualitative methods, b/c more of such types of scenarios are more appropriate with quantitative methods if it is possible, please clearly articulate the reason for why.

2. The qualitative and quantitative methods are different aspects, so how do design at a time?

3. Is the given study population representative?

Part 3: Statistical methods

1. Qualitative and quantitative analysis is much more limited, regarding data type, purpose focuses, and others will not provide.

Part 4: Novelty and Contribution

The paper claims to contribute to improving the LLM. The novelty of this contribution is not fully established. The manuscript should better situate its work within the existing literature on LLM, explaining how it advances the state-of-the-art.

Minor Points of Review:

1. Clarity and Readability: the manuscript contains some typographical errors that need corrections for better readability. The structure of the manuscript could be improved by clearly delineating the sections on introductions, methods, and discussions.

Subheadings and bullet points can help in organizing the content more logically.

Conclusion: The study addresses an important issue for LLMs in supporting statistical consulting. Therefore, the manuscript requires substantial revisions. Furthermore, the novelty and contribution of the work are not established. Addressing the points raised above will be essential for improving the quality and impact of the manuscript.

Reviewer #2: The authors did an excellent job, and I have great appreciation for their dedication and hard work. Their commitment and effort are truly commendable

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Wudneh Ketema Moges

Reviewer #2: Yes: Teklebirhan Abraha Gebrehiwot

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Commented.docx

 10.1371/journal.pone.0308375.r002 Author response to Decision Letter 0 Submission Version 1

19 Jul 2024

Please find our detailed response in the uploaded rebuttal letter.

 10.1371/journal.pone.0308375.r003 Decision Letter 1 Moges Bekalu Tadesse Academic Editor 2024 Bekalu Tadesse Moges This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

23 Jul 2024

Exploring the Potential of Large language models for integration into an academic Statistical Consulting Service: The EXPOLS Study Protocol

PONE-D-24-24617R1

Dear Dr. Fichtner,

We’re pleased to inform you that your protocol has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter, and your protocol will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible—no later than 48 hours after receiving the formal acceptance. Your protocol will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Bekalu Tadesse Moges

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

 10.1371/journal.pone.0308375.r004 Acceptance letter Moges Bekalu Tadesse Academic Editor 2024 Bekalu Tadesse Moges This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

21 Aug 2024

PONE-D-24-24617R1

PLOS ONE

Dear Dr. Fichtner,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Bekalu Tadesse Moges

Academic Editor

PLOS ONE