A Positive Spin: Large Language Models Can Help Directors Evaluate Programs Through Their Patients' Own Words

Thursday, October 30, 2025

10:30 AM - 10:45 AM

Location: ROOM: Conference Room 4G >>> DIRECTIONS Fourth Floor >>> From the Main Elevator Bank, proceed left. Turn left at the Pullman Boardroom. Room 4G is at the end of the corridor.

Presenting Author(s)

Leah R. Flaherty, PsyD

Clinical Director Empowered Relief Program- AHN
Allegheny Health Network
Pittsburgh, PA, United States

Slides Slides

Non-presenting Author(s)

Kendra H. Oliver, Ph.D, M.P.S. (she/her/hers)

Managing Director for Data Science
Responsible Data Science, Office of the Provost, University of Pittsburgh
Pittsburgh, PA, United States

Quantitative data provides valuable metrics, qualitative feedback offers a richer understanding of the patient experience. But analyzing large volumes of qualitative data can be time-consuming and resource-intensive. This session presents a novel approach: using large language models (LLMs) like ChatGPT to efficiently analyze patient responses. We'll share a case study of using LLMs to evaluate a skill-based pain education program, demonstrating how LLMs can identify key themes, uncover unexpected insights, and empower program directors to make data-driven decisions. Learn how to harness the power of LLMs to amplify the patient voice and improve program outcomes, ethically and efficiently.
Key Takeaways:

Gain insights into the value of qualitative feedback in program evaluation.

Understand how LLMs can be used to analyze qualitative data efficiently.

Explore a case study of LLM-assisted analysis in a pain management program.

Learn about the benefits, limitations, and ethical considerations of using LLMs in research.

Discover practical strategies for implementing LLM-assisted analysis in your own work.

Learning Objectives:

Upon completion, participant will be able to describe the potential benefits of using generative AI (GenAI) in thematic analysis of qualitative patient feedback for program evaluation.
Upon completion, participant will be able to identify key considerations and limitations for the responsible implementation of large language models (LLMs) in healthcare research and program administration.
Upon completion, participant will be able to list three ways that insights gleaned from GenAI-assisted qualitative analysis can complement and enrich quantitative data in assessing program effectiveness.