A cornerstone of modern medicine, imaging has revolutionized diagnosis, treatment planning, and disease monitoring with radiology reports an indispensable component of patient medical records. Traditionally, these reports have been accessible only to radiologists and the referring providers, who then interpret findings for patients [1]. However, the rise of electronic portals has led to an increasing number of patients directly accessing their medical information, with the 21st Century Cures Act mandating access to all parts of the electronic health record (EHR) [2]. While this shift empowers patients to play a more active role in their care, the complexity of medical jargon and acronyms in radiology reports often leads to significant confusion, anxiety, and potential misinterpretation [3,4]. Simply looking up individual terms often fails to provide a cohesive understanding of the report’s narrative and its implications. This underscores a critical need for tools that can translate the entire report findings into plain language, thereby enhancing patient comprehension and facilitating more meaningful patient-provider communication Moreover, advanced imaging modalities continue to evolve rapidly, reinforcing the need for clear, transparent patient-facing reports.

Correspondingly, there has been growing attention toward health literacy and its associations with patient engagement, treatment adherence, and care disparities [5]. Advanced natural language processing (NLP), in turn, has emerged as one promising tool to help bridge the gap between complex medical information and ease of understanding [6–8]. In radiology in particular, NLP techniques have been used in everything from observation detection for diagnostic surveillance to quality assessment to clinical support services [9,10]. A few studies have sought to use NLP to simplify radiology reports, namely through linking terms to the consumer health vocabulary system [11] and the French lexical network [12].

With the rise of large language models (LLMs), OpenAI’s ChatGPT, Google Bard, and Microsoft Bing have also been explored for simplifying radiology reports [13,14]. The main advantage of these tools is their accessibility and comprehensiveness — freely available to anyone with an internet connection and able to simplify an entire radiology report, rather than simply appending a summary, linking to a glossary, or altering the structure. However, there are limitations. While general-purpose LLMs like ChatGPT are accessible, their effectiveness for specialized tasks like simplifying radiology reports heavily depends on user-crafted prompts, which can lead to variable quality and reliability [15,16]. Patients may not possess the expertise to formulate optimal prompts. Rads-Lit addresses this by embedding a systematically optimized prompt within a user-friendly interface, specifically designed for radiology reports. This aims to provide more consistent, reliable, and appropriately simplified outputs compared to ad-hoc use of general LLMs, thereby offering a more dependable solution for enhancing patient health literacy in this domain.

We have used a variety of prompts to assess these LLMs for accuracy and fidelity in simplifying radiology reports [15,16]; however, we have this data only for specific prompts and remain unsure if the LLMs would be inaccurate or inadequate if asked different prompts. Given the infinite possibility of prompts and the corresponding variable quality of responses, patients may not be able to take full advantage of these chatbots to improve their own health literacy. While general LLMs show promise, their direct application by patients for simplifying complex medical texts like radiology reports is fraught with challenges, including prompt variability and inconsistent output quality [13–16]. This highlights a critical gap: the need for specialized, optimized tools that can reliably simplify radiology reports to an appropriate health literacy level while maintaining clinical accuracy. Therefore, this study aimed to address the following research questions:

1.) Can a systematic prompt engineering process identify an optimal LLM prompt to simplify radiology reports to a target 5th-7th grade reading level?

2.) How does a specialized tool (Rads-Lit), utilizing an optimized prompt, perform in terms of readability improvement across various imaging modalities compared to original reports and a basic simplification prompt?

3.) What is the accuracy, completeness, and perceived safety (by radiologists) of the simplified reports generated by such a tool?

We present the development, optimization, and preliminary evaluation of the Rads-Lit Tool, an AI interface designed to address these questions. This work includes a novel methodology for LLM prompt assessment for health literacy, aiming to empower patients with understandable medical information, thereby fostering improved patient-provider communication and informed decision-making.“

The prompt optimization process is summarized in Fig 1 and Table 1.

After determining the 5 best-performing prompts in the pilot analysis across Stages 1–3 (Table 1), we re-calculated the average readability scores of these prompts across 750 radiology reports (Table 2). These prompts, namely Prompts A-E, demonstrated significantly improved readability scores compared to the original radiology report and the common prompt, “Simplify:” across all four readability indexes tested (p < 0.0001) (Table 1, S2 Fig). The final five prompts had comparable, reading score differences, with aRGLs of 6.1 to 6.3, and were not statistically significant.

Given that were no statistical differences between the five prompts, we utilized the Delphi method to decide our prompt. 2 reviewers blindly assessed 5 outputs from each of the 5 prompt on fidelity and accuracy and chose prompt D, “You are a health literacy tool. Interpret this radiology finding at the 5^th grade level:”. Despite not having the lowest readability score, we chose this prompt because it best retained fidelity while clearly defined the role of the LLM and provided a goal grade level (Table 2).

In simplifying 750 radiology reports, our selected prompt for our tool (utilizing Prompt D) shows that all four readability indices tested showed statistically significant improvement in readability scores of findings compared to raw radiologist reports (N = 750, p < 0.0001) (Table 1, Fig 2, S3 Fig). The chosen prompt for our interface produced output with a median of 55 [38–72.5] words.

Across all imaging types, our tool (utilizing prompt D) simplified radiology reports from an aRGL of 13.7 to an aRGL of 6.2 (Fig 2). CT radiology reports had an aRGL of 13.9, and our tool provided simplified outputs with an aRGL of 6.3. This simplification held across different examination types, including US (13.6 to 6.2), Mammogram (13.2 to 5.9), MRI (13.7 to 6.4), and X-ray (14.1 to 6.3; Table 1).

With the optimized prompt, we assessed our tool for accuracy, completeness of information, inclusion of beneficial supplementary information not found, appropriate urgency, and comfort providing output without supervision using a 5-level likert-type evaluation (Fig 3). For accuracy of the information, both radiologists predominantly found most simplified reports to be free of inaccuracies or misleading details, with combined agreement or strong agreement recorded 255 times out of 300 outputs (85%). However, the radiologists indicated disagreement or strong disagreement with this sentiment for 24 outputs (8%). Regarding the inclusion of all pertinent or actionable details from the original reports in the simplified versions, 249 outputs (83%) conveyed agreement or strong agreement. They expressed disagreement or strong disagreement with 18 (6%) outputs. The provision of beneficial supplementary information in the simplified reports not found in the original impression was found in 81 outputs (27%), while 189 outputs (63%) disagreed or strongly disagreed. On the matter of comfortably sharing the simplified reports directly with patients without additional oversight, the radiologists felt expressed agreement or strong agreement to comfortably sharing reports without additional supervision in 230 outputs (76.7%), while they expressed reservations for 39 outputs (13%). The radiologists felt, in 261 of the outputs (87%), that the simplified reports adeptly communicated the required urgency. Disagreement or strong disagreement was recorded in only 8 instances (2.7%).

Our proof-of-concept study presents the development and evaluation of the Rads-Lit patient interface. Through an iterative comparative assessment, we evaluated 79 distinct prompts to determine which five best simplified radiologist reports, achieving the lowest RGLs, relative to the common prompt (“Simplify”). Ultimately, we selected the prompt: “You are a health literacy tool. Interpret this radiology finding at the 5^th grade level” for further testing based on our team’s previous study on the importance of context for OpenAI’s LLM [25]. Evaluations by 2 radiologists confirmed that the output from this prompt was accurate 85% of the time with little to no extraneous content relative to the original radiology report. While our findings suggest there is room for model refinement, they also underscore the importance of balancing the clarity of simplified reports with preserving nuanced clinical details integral to patient care. Importantly, simplified language does not equate to clinical guidance. Without proper framing, patients may misinterpret these outputs as standalone diagnostic conclusions.

Given the critical need to promote radiological literacy, approaches such as summary statements, [26] language glossaries, [27] structured templates with standardized lexicon [28,29], video radiology reports [30,31], and listing the radiologist’s phone number have been proposed [32]. This study highlights the utility of using NLP and LLMs to simplify radiology reports, as well as several areas of further research. For example, radiologists’ opinions diverged regarding the inclusion of supplementary information, with only 27% of the outputs found beneficial and a notable 63% disagreement, suggesting that a one-size-fits-all approach may not be ideal. This variability reinforces the need for clear communication that simplified reports are meant to enhance, rather than replace clinical consultation. Some patients might benefit from additional context, while others might find this supplementary information extraneous or confusing. Additionally, the collective comfort of the radiologists in sharing 76.7% of the simplified reports directly with patients indicates the tool’s potential in promoting patient autonomy and comprehension. Still, given the novelty of this technology and risk of hallucinations [33,34], it is unlikely that radiologists would trust the Rads-Lit patient interface to operate autonomously. Indeed, in a previous single-center survey, 76.9% of radiologists reported that they would not support AI-generated simplifications of reports without a manual check [35]. A safety net, possibly in the form of a reviewing radiologist, will likely still be necessary. Hospitals and health systems must consider embedding controls within such tools, such as requiring clinician review before release, clear disclaimers on patient-facing outputs, and integration with the electronic health record to ensure traceability. These institutional guardrails are critical to prevent misuse, including inappropriate self-management or misinformed clinical decision-making by patients.

Patients are already using LLMs like OpenAI’s ChatGPT, Google Bard, and Microsoft Bing to better understand their medical care; however, given the infinite possibility of prompts, most people are not providing additional context that they are a patient, that the chatbot should act as a health literacy tool, or that simplification should happen at a specific grade level [36–38]. Indeed, our previous work has shown that the prompt “Simplify” on ChatGPT-3.5 consistently produces excessively complicated outputs for the average American’s 7^th grade reading level [13,17,18]. Other research from our group suggests that the level of simplification differs based on racial context: Open AI’s ChatGPT-3.5 and ChatGPT-4 simplified radiology reports at a higher reading level for those who self-identified as White or Asian, when compared to those who self-identified as Black or American Indian/Alaska Native [39]. These findings raise important concerns about equity and implicit bias in LLM-generated outputs. While the reasons for these disparities remain unclear, they may reflect systemic inequities embedded in the training data or variation in how different demographic identifiers interact with the model [40].

Some small studies have explored the accuracy of LLM-simplified radiology reports with variable results [41]. For example, using 3 fictitious radiology reports, Jeblick et al. created 15 simplified reports and found that one-third of these reports had incorrect statements, missing key medical information, or potentially misleading passages [42]. Tepe et al. analyzed 30 simplified radiology reports and found that their readability and understandability was significantly improved, although their accuracy in assessing the urgency of medical conditions was inadequate [43]. For contrast, analysis of simplified reports from 20 cardiovascular MRIs and 60 shoulder, knee, and lumbar spine MRIs found that GPT-4 produced highly accurate reports with minimal confusing or inaccurate output [44,45]. Our group has similarly shown that, in analysis of 150 mammography, X-ray, CT, MRI, and ultrasound scans, radiologists found that 83–86% of radiology reports simplified by GPT 3.5 and 4 had no errors and all essential information [16]. However, to our knowledge, there has been no previous work that has comprehensively assessed the accuracy, completeness of information, appropriate urgency, and comfort providing output without supervision, especially in such a large sample size with 750 radiology reports across diverse examination types. Our results suggest the import of implementing standard prompts and guidelines for LLM-based patient education in order to maximize the utility of these tools and to improve equity in health communication.

Our proof-of-concept tool explores the capabilities of LLMs and suggests that these tools can be safely incorporated into radiology practice. With a median count of 55 words in this study, a radiologist should be able to review the simplified output quickly, but the benefits to patients must be weighed against these disruptions to providers, as RVUs may not accommodate for this additional review time and a separate billing code is unlikely. An example of an implementation of such a technology is in a speech recognition and reporting platform, where LLMs and NLP could readily generate a simplified summary, which the radiologist can proof, and if needed, edit. Or, this technology could be implemented in EHRs allowing a radiologist to review simplified output before signing a report. There is evidence a simplified report or summary may benefit patients and improve patient satisfaction scores, but adoption by providers and impact on providers must also be assessed. Importantly, we must address concerns about the unintended consequences. What if, for instance, these simplified reports lead to heightened anxiety? And does it truly offer time-saving benefits to referring physicians when they discuss findings with their patients? Although our work is foundational, it’s essential to be proactive in our discussion and think about the potential implications [46].

This study is not without its limitations. For one, we relied primarily on readability metrics to guide our prompt engineering. Although we utilized the Delphi method to look at the outputs of the 5 prompts with the lowest readability scores to identify the best prompt and ensure that the prompts retained fidelity and accuracy, we did not use accuracy to identify our prompt until that point. Second, the readability metrics used in this study are language and structure-focused, so these measures may not necessarily capture comprehensibility from a medical perspective. Additionally, readability metrics may not adequately capture patient literacy needs. This highlights the need for further research to refine the interface and ensure its applicability across diverse patient populations and radiology subspecialties, as well as explore the potential for personalized approaches to simplifying radiology reports, considering individual patient characteristics, such as age, education, and prior medical knowledge. Moreover, future iterations must prioritize clear guardrails and user education to ensure that patients understand simplified reports are adjunctive, not directive, and that clinical follow-up remains essential. Finally, we have utilized 2 attendings and a resident with the likert scale to assess these prompts when in reality the incorporation of patient feedback and physician input on the simplified reports is crucial to evaluate the tool’s real-world usability and its potential for enhancing patient-provider communication. Future multi-site studies would validate these readability metrics in real-world settings, ensuring their reliability and relevance to patient outcomes.

It is crucial to reiterate that the Rads-Lit Tool is designed as a health literacy aid to simplify existing radiology report text, not as a diagnostic or clinical decision support system. As rightly noted by challenges in the broader AI field, these models, including the one underpinning Rads-Lit, currently lack the ‘Gestalt’ understanding required to interpret findings within the full, complex clinical context of an individual patient, especially concerning comorbidities or the relative intensity of multiple pathologies. While the tool improves access to comprehension, it does not obviate the need for follow-up with a referring clinician. Misinterpretation, such as assuming a non-urgent finding requires no action or self-treating based on the simplified language, poses real risks if safeguards are not in place. Our tool does not attempt this; its purpose is solely to make the concluded report more comprehensible to patients after it has been finalized by a radiologist.

We believe that this proof-of-concept tool can help begin a discussion of how to utilize such LLMs for patient-centered care in radiology. In our tool, we also showcase preliminary features (that remain untested) to allow patients to press “explain more” if they don’t understand or want to learn more about a particular sentence of the generated output and allow physicians to edit responses if they are using the tool to provide patients a better understanding of their report findings (Fig 4). As LLM solutions become more common, discussion regarding the implementation of such tools is of utmost importance.

In conclusion, this study outlines the development and preliminary evaluation of Rads-Lit, demonstrating that an AI tool with optimized prompting can significantly improve the readability of radiology reports. Our findings suggest its potential as a specialty-specific health literacy aid, shifting the onus of prompt engineering from the patient to a systematically designed interface (Fig 4). While improved patient-provider communication and patient-centered care are key goals, this study represents an initial step. Extensive further research, including direct patient feedback, validation across diverse populations and clinical settings, and assessment of real-world impact on patient understanding and outcomes, is crucial before such tools can be broadly incorporated into clinical practice [47]. Most importantly, any implementation of such tools must prioritize patient safety by incorporating robust safeguards to prevent misinterpretation and inappropriate self-management, ensuring that simplified reports enhance rather than replace essential clinical relationships. Continued collaboration among patients, clinicians, developers, and policymakers will be essential to responsibly harness this technology to support, rather than replace, human-centered care.