A New Path to Understanding the Physician’s Decision Journey Using Simulated Patients

A New Path to Understanding the Physician’s Decision Journey Using Simulated Patients
Chandra Chaterji, Consultant, and Greg Chu, Chief Operating Officer, InTask, Inc.
Abstract
Simulations built around the management of virtual patients offer healthcare market research and marketing science professionals an innovative and powerful approach to deeper understanding of the behaviors and decision-making processes of physicians. Rather than asking physicians what they do and why, as we might in a questionnaire, simulation allows us to observe physician behavior within an engaging environment in which they treat virtual patients. This data collection method using simulated patients is scalable, and can be used in both large and small sample studies. In this article, we report on early findings from pilot tests of this novel simulation software platform developed for use in market research.

Keywords: Simulation, Decision journey, Big data, Segmentation, Behavioral economics, Virtual patient
Introduction
There are two broad approaches to collecting market research data. One is observation—starting with participant observation and ethnography, and extending all the way to think-aloud methods, focus groups, IDIs and even journalism approaches. These methods have traditionally been associated with qualitative research. The other broad approach is structured questioning, as exemplified by the market research questionnaires utilized in quantitative surveys. Both approaches have strengths and weaknesses. Observational methods can provide deep insights into what respondents do and why, but do not readily scale to large samples. Structured questioning is amenable to statistical design principles, and scales easily, but does not necessarily illuminate the respondents’ thought processes. Moreover, we are increasingly sensitive to problems of respondent engagement with quantitative research procedures since structured questionnaires offer little intrinsic value to the respondent. In the realm of healthcare, the questionnaire data often does not derive from ongoing medical or business processes that healthcare professionals need to engage with as part of their work. Nor does it resemble or invoke the tasks they actually undertake on a daily basis. As a result, lengthy or tedious surveys may not inspire respondents to provide their best thinking in answering survey questions.
 
Simulation of virtual patients who are “treated” by physicians offers an intriguing opportunity to combine the best elements of observation and questionnaire-based surveys.  Within digital environments, physicians can be engaged in actual tasks, such as the diagnosis and treatment of virtual patients.  Rather than asking physicians what they do, simulation allows researchers to observe what physicians do under a range of environmental situations and with different patient types tailored to address specific areas of research interest.  At the same time, digital simulations can provide researchers with insights into the physician’s decision-making process as reflected in the order, frequency and duration of access of patient information, as well as through the lab tests and diagnostics ordered to support treatment decisions for specific patients.  Moreover, simulation allows us to observe how all this unfolds in virtual time, as patients can be programmed to return for multiple visits with evolved presentations.
 
Simulation also generates large streams of data which, relative to survey data, can reasonably be called “big” -- created with no more investment of time or effort on the part of the respondent than that demanded by traditional market research questionnaires. This last point hints at additional ways in which big data can be leveraged in healthcare research. We are largely familiar with both the promise and challenge of capturing, managing and making sense of big data in healthcare. Significant value has already been generated from EMR systems, social media, and even the internet of things in supporting improved commercial decision-making.  Considerable room for further progress exists, but it is interesting to note that most efforts are focused on analytic approaches rather than the sourcing and production of the data itself. Most approaches to working with big data today seem to assume that the data itself is a given—that it arises from natural processes that are separate from analytic endeavors.  Accordingly, efforts to derive value from “naturally occurring” big data necessarily devote considerable effort to the cleaning and organizing of data so that it fits with the researcher’s analytic goals.  Given the fact that real world data can be extremely messy, this is no small task. Simulation offers the possibility of creating relatively clean, ready-to-use big data that provide insight into both what respondents do and why they do it–in essence combining some of the best traits of both observational and survey research techniques.
 
In this paper, we provide an overview of our experience in pilot testing a simulation platform developed for use as a market research tool.  We will focus on two points: 1) The respondent interaction with the simulation platform, and 2) The types of data generated by the simulation.
Method
In order to test simulation as an alternative or complementary approach to current market research and secondary data analytics, InTask, Inc. developed an EMR-like simulation interface built around the diagnosis and treatment of virtual patients.  The simulation, which is browser-based and can run on tablet, laptop or desktop devices, is structured around three tasks:  the examination of the patient, treatment decisions, and scheduling of follow-up visits.  The content of the simulation platform can be customized to address a wide range of therapy areas and issues.  Figure 1 contains a screenshot showing the patient examination screen from a beta version utilized in a study on schizophrenia. 
 
Figure 1: Screenshot of Beta Version of Schizophrenia Simulation Platform



The simulation records every action taken by the physician when engaging with the virtual patients.  While this ensures the capture of basic information such as treatment decisions made, lab tests ordered or even referrals, it also provides detailed information on what the physician chose to view in making those decisions.  Thus, referring to the patient examination screen shown above, if the physician clicks on one of the information category buttons on the left side of the screen, this is recorded—as is the frequency with which he clicked on this piece of information, the order of access relative to other options and the time spent before clicking on the next information category.  It is important to note that the physician is not forced to examine all groups of information; the simulation provides more potentially relevant information than the physician will actually utilize in making a treatment decision.  Inasmuch as the physician pays more attention to some information items than others, the simulation helps reveal his decision-making process.  In this sense, the behavioral economics concept of “bounded rationality” is “hard wired” into the simulation design. 
 
Critical to the success of the simulation is the design of the virtual patients which populate it.  The simulation can accommodate a wide range of patient parameters, including static descriptions of the patient, such as age and gender, as well as dynamic measures such as symptoms, lab values, concomitant conditions and current treatments.  These parameters can be conveyed in various formats including text, image and video.  Patients are designed not only to be realistic, but also representative of the pool of patients relevant to the research issue.  For example, if a marketer is interested in second line therapy in type 2 diabetes, the virtual patients might be designed to fail on first line therapy with metformin.  In this case, the progress of the virtual patient will be determined through a pre-determined, branched logic largely independent of the physician’s treatment decisions.  In other cases, the patient progress will be determined by a Markov process, informed by outcome probabilities derived from real-world data. 
 
Our experience in conducting multiple pilot tests and commercial projects across a range of therapeutic areas is that a good grasp of the patient presentation is critical. We always work closely with client teams—marketers and physicians—when designing the virtual patients, and have found these inputs to be indispensable. Designers of virtual patients for specific applications must know what clinical parameters are relevant, as well as the outcomes that are reasonable to include within the virtual timeframe of the simulation. They must also know what diagnostic and treatment options to include.  In crafting virtual patients, input from the client’s medical affairs group is often useful, as is prior market research—in particular, chart audit data.  As with questionnaire development, pre-testing of the simulation is essential.  Comfort with experimental design is also important, as the virtual patients may be created to tease out the impact of specific patient attributes in driving diagnostic or treatment behaviors.  In our work, we have created virtual clones—pairs of patients who are essentially identical except for one or two key traits hypothesized to be determinants of behavior.  This experimental approach, akin to A/B testing, has proved useful in quantifying the impact of patient parameters on prescribing decisions. But the most vital input to the design of effective simulations is a thorough understanding of the research issue.  In this respect, designing a good simulation is no different from designing a good questionnaire. 
 
A second critical design consideration is the format of the patient information presented in the simulation.  Overly zealous efforts to emulate reality can cloud interpretation of simulation results. A photo of the patient, for example, may convey information that impacts physician assessments and treatment decisions.  Without validated and standardized portrait portfolios, the simulation designer may introduce unnecessary noise into the analysis of quantitative simulation data by using stock photos.  Inclusion of other potentially ambiguous diagnostic input, such as an actual radiograph of a joint rather than a succinct and clear radiologist report, may similarly open the door to analytic challenges.  Fortunately both these concerns can be easily addressed. Engagement with the simulation platform does not demand patient photos, and straightforward diagnostic information is usually sufficient to maintain a sense of realism. It is worthwhile noting, however, that inclusion of ambiguous stimuli can be extremely valuable in qualitative research applications of the simulation. In our experience, inclusion of these stimuli have invariably enriched discussions with physicians and generated insights that were missed in more traditional qualitative projects.
 
The remainder of this article focuses on actual findings and data analysis derived from two pilot tests of the simulation platform conducted in the United States in 2016 with a total of 178 primary care physicians and psychiatrists.  In each of these pilots, physicians were asked to treat eight virtual patients suffering from either schizophrenia or Alzheimer’s disease over multiple visits in virtual time.  Respondents were recruited from a fieldwork agency panel via an on-line screener.  Their reactions to the simulation were captured using a post-simulation questionnaire.
 
Physician Reactions to the Simulation
The promise of simulation, as well as the theoretical efficacy of the approach, is premised in large part on the ability of simulations to better engage respondents and present data gathering tasks which more closely replicate real-world situations than do surveys.  Early user experience testing with the simulation platform across a number of therapy areas suggested that simulations do in fact engage physicians to a greater extent than do typical surveys.  Post-simulation surveys from our two quantitative pilots confirmed this.  In both pilot tests, 9 out of 10 respondent physicians agreed that the simulation was more interesting than the online surveys they typically complete.  (Figure 2) 
 
Figure 2: Combined Results From Two Pilot Studies (n=178) Conducted in 2016 in the United States with Primary Care Physicians and Psychiatrists


Equally important is the fact that respondents viewed the simulation exercise as reflective of reality.  More than 80% of respondents in the two pilot studies agreed that the “treatment decisions they made within the simulation reflect how they would treat similar patients in the real world” and that the “patient profiles and the way they changed in response to my treatment decisions were realistic”.  (Figure 3)
 
Figure 3: Combined Results From Two Pilot Studies (n=178) Conducted in 2016 in the United States with Primary Care Physicians and Psychiatrists. Percentages Reflect Top 3 Box Agreement on a 7 Point Scale.


Physician Behavior: Time Spent
If physicians claim that the simulation is more interesting and also realistic, is this reflected in how they actually interacted with the simulation?  Since everything they actually do in the simulation is recorded, we have the means to provide at least a glimpse into how respondents engaged with the digital environment.  One measure of interest is the amount of time physicians spent treating each virtual patient over the course of the simulation.  In both pilot studies included in this analysis, physicians were asked to treat eight virtual patients over three visits each for a total of 24 virtual patient interactions.  Figure 4 shows the average time in seconds spent with each patient over each visit.  As is evident from the graph, the average time spent with each patient declines quickly after the first few visits.  This was anticipated and reflects a rapid learning curve as physicians became familiar with the simulation interface.  Beyond this, two important points are evident in the time stamp data from which this data is derived.  First, decline in time spent per virtual patient visit levels off rather than continuing to decline.  This suggests that, at least within the context the patient volume presented in this simulation, respondents were not growing bored with the simulation.  Second, we note that the spikes in time spent all correspond with the appearance of a new virtual patient.  This is reassuring, as it reflects the real world dynamic of clinical presentations in which initial visits are longer and more extensive than follow-up visits. This analysis of time spent on virtual patient interaction supports the self-assessment of respondents that they were engaged in the research process and that the process itself reflects, at least in part, their behavior in the real world.

Figure 4: Combined Results From Two Pilot Studies (n=178) Conducted in 2016 in the United States with Primary Care Physicians and Psychiatrists. Average Virtual Patient “Visit” Time in Seconds.


Physician Behavior: Information Seeking And Prescribing
At this point, we can move beyond physician interaction with the simulation and consider the actual data generated from it.  In broad terms, the simulation produces two categories of data.  The first relates to diagnostic and treatment decisions selected for specific virtual patients.  When considered over multiple visits in virtual time, the resulting data is analogous to longitudinal patient data—with the exception that each individual virtual patient is seen by a sample of respondents.  Hence the unit of statistical analysis is primarily the physician rather than the virtual patient. As seen in Figure 5, we can readily observe how a sample of physicians will treat a specific mild-moderate Alzheimer’s disease patient over three visits, each separated by three months of virtual time.
 
Figure 5: Data from Alzheimer’s Disease Study (n=105), Conducted with Primary Care Physicians and Psychiatrists in the United States (Nov 2016)


While this example shows longitudinal results at the drug class level, the simulation can easily capture information that enables analysis of drug choice by molecule, strength, form and dosing.  The nature of the simulation interface allows for very rapid collection of this detailed treatment information, a design characteristic of particular importance for areas like oncology, where customized cocktails of treatments may be utilized which are extremely difficult to pre-code in standard questionnaires.
 
The second major type of data generated by the simulation relates to the decision-making process of the physician, as reflected in the virtual patient information accessed.  Unlike traditional surveys in which the survey design attempts to force respondents to view all the information provided, the virtual patient simulation as implemented here takes a radically different approach.  Physicians are free to view whatever patient information they choose—multiple times or not at all, and in any particular order they wish.  As in the real world, more potentially relevant information is provided than physicians will actually absorb in making their treatment decisions.  What information they choose to pay attention to should correlate with information that is most likely to drive their prescribing decisions.  Moreover, given that physicians are not homogeneous in their perspectives on patient treatment, information access can also provide the means to segment physicians based on decision-making style.
 
Figure 6 shows the average number of times that physicians accessed different categories of information in a pilot study on schizophrenia.  Each category corresponds with a category of information presented in the simulation interface and viewable with the click of a button.  As is evident from the graph, on average, physicians viewed most categories of information at least once on the first visit of the virtual patient.  The average frequency of access then declined on the second and third patient visits.  For some categories like patient overview, which contained static information such as age and gender, it is not surprising that frequency of access declined precipitously in visits 2 and 3.  Respondents quickly learned that there was no new information in these categories and therefore tended to ignore them as the simulation went on.  But what about categories such as symptom set 1, symptom set 2 and patient interaction which contain dynamic information—information which changed with each visit?  Frequency of access also declined here on second and third visits, although not as dramatically as for static categories. 
 
Data From Schizophrenia Study (n=73), Conducted with Psychiatrists in the United States in Feb 2016 ; Average Number of Times Category Accessed per Patient, by Visit Number


In order to better understand the dynamics of information access observed in the simulation, we conducted a simple segmentation of respondents (SPSS K-means procedure) based on the categories of information physicians accessed on an individual patient basis across all three visits.  Included in this analysis were the three “dynamic” categories of information: symptom set 1, symptom set 2 and patient interaction, as well as two “static” categories: social support and insurance. 
 
Our analysis produced a three cluster solution for each of the eight virtual patients included in the simulation.  Across all virtual patients, the two largest of these segments accounted for at least 85% of respondents.  Given the limited sample size (73 responses per patient), these two segments became the focus of our analysis.  Figure 7 displays the dynamic categories of information accessed by these two segments of respondents for visit 1 for each of the eight virtual patients.  As is evident from the table, virtually all physicians, regardless of segment, viewed the two dynamic information categories related to symptom set 1 and symptom set 2.  Segment One, however, extended this thoroughness to the patient interaction category, which included brief narrative descriptions of the patient or caregiver’s subjective assessment of the patient’s situation.  In contrast, Segment Two was generally less likely to view this category of information.

Figure 7: Results of Physician Segmentations Produced Through K-Means Procedure for Each Virtual Patient. Shaded Percentages are Significant at >95%.
Segment Size % Viewing Symptom Set 1 % Viewing Symptom Set 2 % Viewing Interaction
Patient 1
Segment 1 44% 100% 100% 100%
Segment 2 48% 100% 100% 94%
Patient 2
Segment 1 58% 100% 100% 98%
Segment 2 27% 100% 100% 100%
Patient 3
Segment 1 47% 100% 96% 100%
Segment 2 38% 100% 96% 68%
Patient 4
Segment 1 79% 100% 98% 100%
Segment 2 21% 73% 73% 27%
Patient 5
Segment 1 45% 97% 100% 100%
Segment 2 44% 100% 100% 63%
Patient 6
Segment 1 55% 100% 100% 100%
Segment 2 29% 100% 100% 76%
Patient 7
Segment 1 42% 97% 97% 97%
Segment 2 44% 100% 97% 69%
Patient 8
Segment 1 60% 100% 98% 100%
Segment 2 29% 91% 86% 24%


As is evident in the table, both the size of segment two and the percentage of segment physicians viewing the patient interaction information varied by virtual patient.  This suggests that physician decision-making style—as measured by information access—is dependent, at least in part, on the type of patient seen.  In this specific case, one possible interpretation is that certain combinations of patient presentation and physician decision-making style increase the value of patient and caregiver input relative to the better defined categories of symptom sets 1 and 2. 
 
Further work in this area needs to be conducted to develop improved analytic approaches to understanding physician decision-making within simulations. These efforts should proceed along several paths. First, information access order, frequency of access and time spent on each item should be considered alongside our current binary metric of access (viewed or did not view specific information). Second, while information access varies by virtual patient, a meta-analysis of information access across patients should be pursued to determine if there are decision-making styles which are relatively stable regardless of patient. Third, the technique gives us a unique ability to understand physician decision journeys, which is of great practical importance. Finally, simulations such as this should be linked to real world behavioral data, such as prescribing, to determine how decision-making style is related to actual behavior.
 
Big Picture: What Have We Learned And What’s Next?
  • Our early experience demonstrates that physicians find simulation to be more engaging than questionnaires and suggests that they interact with simulation in a way that reflects their real-world practice, demonstrating face validity. 
  • Physician behavior during the task varies across patients and across visits with the same patient, demonstrating the validity of the procedure. As such, these simulations can provide insight into what information physicians attend to in making treatment decisions, adding to our understanding of physician decision making.
  • In quantitative applications of the simulation platform, we have been able to employ simple A/B testing with virtual patient clones to quantify the impact of specific patient characteristics on prescribing. 
  • In qualitative applications of the simulation, we have been able to use the realism of the simulation to drive deeper insights into physician decision-making. Most notably, we have found the simulation to be extremely valuable in conducting cognitive interviews with physicians. In one cognitive interview study, the insights we developed around the treatment algorithm radically altered the client’s in-going perceptions of what was originally thought to be a simple, sequential algorithm.
  • In general, client teams have said that the technique provides insights and learning that they did not uncover in previous research. This has encouraged us to develop simulation as a tool for both quantitative and qualitative research applications.
  • Going forward, we intend to employ more sophisticated experimental design manipulations in patient symptoms and patient interaction in conjunction with environmental parameters (insurance, competitive activity, etc.) to understand the effects of these variables on physician prescribing behavior.
At this early stage, we have only scratched the surface of the analytic possibilities of such simulation data.  Much work remains to be done, but initial results from these pilot tests in simulation suggest that the effort promises significant rewards.

About the Authors
Chandra Chaterji is an independent consultant in analytics and strategic marketing. He earned his stripes in marketing science, market research and brand strategy, in companies such as Visa, J. Walter Thompson and BBDO. He has presented papers at the INFORMS Marketing Science and the AMA’s Advanced Research Techniques conferences. An ex-academic, he has taught in the graduate business programs at Loyola of Chicago, Rutgers and UC Davis.
 
Greg Chu is Chief Operating Officer at InTask, Inc. where he oversees the development of simulation platforms for use in healthcare market research. His career in healthcare includes leadership positions in commercial research and analytics at Merck & Co., Forest Labs, Novartis, Synovate Healthcare and Ipsos Healthcare.