Abstract
Introduction: Automated self-scheduling may benefit healthcare organizations, yet uptake has been slow. The aim of this study was to develop a consensus statement regarding the organizational-level determinants of implementation success based on the collective knowledge of experts. A three-stage modified Delphi method was used to reach consensus on the top determinants of implementation of self-scheduling solutions by healthcare organizations. A panel of 53 experts representing 41 academic health systems identified barriers and facilitators involving the organization’s inner and outing settings, as well as the characteristics of the intervention and the individuals engaged in the solution. Offering convenience for patients is the leading enabler for organizations to implement the technology. The consensus may aid healthcare organizations and suppliers engaged in adopting and developing self-scheduling technology to improve implementation success. Further research is recommended to diagnose and examine each barrier and facilitator and how these factors interact.
Objective: The aim of this study was to develop a consensus statement regarding the determinants of implementation success based on the collective knowledge of experts working in the field.
Methods: A Delphi panel was constructed based on selected participants employed by academic health systems and experienced with self-scheduling implementation. Panelists were recruited based on participation in an educational event that featured the topic. Purposive and snowball sampling were used. Panelists participated in surveys collected over three rounds. An 80 percent agreement among panelists and interquartile range (IQR) <1 determined the barriers and facilitators. The top-10 determinants were presented in rank order.
Results: Between January 6, 2021, and May 26, 2021, 53 panelists representing 41 academic health systems participated in three rounds of surveys to reach consensus on the barriers and facilitators to implementation of self-scheduling by healthcare organizations in the United States. In round one, panelists documented 530 determinants. In round two, the determinants were grouped into 72 barriers and 85 facilitators, each of which participants rated on a five-point Likert scale. Fifteen determinants met the 80 percent threshold and 1.0 IQR. The final round concluded with a top-10, rank-ordered listing of determinants (seven facilitators and three barriers) that also incorporated a median rating score using five-point Likert scale.
Conclusion: A three-stage modified Delphi method was used to reach consensus on the top determinants of implementation of self-scheduling solutions by academic health systems. The consensus may aid healthcare organizations and suppliers engaged in adopting and developing self-scheduling technology to improve implementation success. Further research is recommended to diagnose and examine each barrier and facilitator and how these factors interact.
Keywords: Delphi panel, automated self-scheduling, barriers and facilitators, implementation
Introduction
Appointment management in the ambulatory setting is important for healthcare organizations as waits and delays lead to poor management outcomes: dissatisfied patients, scheduling disruptions, and wasted appointment slots. For example, according to the Patient Access Collaborative,1 the median new patient lag time for a scheduled ambulatory appointment in the United States is 16.35 days, while the median utilization rate of appointment slots is only 73.6 percent.
Automated self-scheduling may benefit healthcare organizations in their efforts to manage staffing costs,2-7 patient satisfaction,8-10 appointment attendance,11-15 patient accountability,16 and information transparency.17,18 Self-scheduling may offer the convenience patients seek.19-20 Automated, self-service reservation systems have helped other industries striving for improvements in customer loyalty,21 operations,22,23 profitability,24 and customer wait times.25
Despite evidence to support the value of the technology to healthcare organizations, the uptake of self-scheduling in healthcare has been minimal to date.
Providers have expressed reluctance about self-scheduling based on cost, flexibility, safety, and integrity.26 As the technology emerged in the early 2000s, physician and software developer Dr. Jonathan Teich elucidated a critical challenge related to complexity: “Before you can successfully implement self-scheduling, you have to implement ‘Mabel.’ Mabel is the generic scheduling administrator who has been working for Dr. Smith for 35 years, and knows a thousand nuances and idiosyncrasies and preferences that have been silently established over the years … Unfortunately for the computer world, it’s extremely difficult to find out what Mabel really knows, let alone try and put it into an algorithm.”27 In addition to the Mabel factor, physicians have conveyed a fear of losing control.28-30
Researchers have also documented patients’ hesitancy about self-scheduling based on concerns about accuracy, security, and a lack of empathy as compared to a human interaction.31 Further, patients’ prior experience with technology, as well as communication preferences, have been recognized as barriers.32
What remains unknown is the potential influence of the current diffusion of self-service technology, the changes in patients’ access to virtual delivery platforms, and the heightened expectations for convenience that have resulted from the COVID-19 pandemic as it relates to healthcare organizations’ uptake of self-scheduling. By identifying barriers and facilitators to automated self-scheduling, the research will assist healthcare organizations seeking solutions to the management of the ambulatory enterprise, ultimately benefiting patients through improved service, reduced disruptions, and enhanced utilization of providers’ time.
Background
Despite the acknowledged benefits of administrative technology in healthcare, adoption has been slow, with implementation barriers cited as evidence of the limited diffusion.33
The goal of the study was to derive a consensus statement regarding the organizational-level barriers and facilitators to implementation of automated patient self-scheduling by healthcare organizations in the United States. The primary research question posed was: “What is the consensus regarding the barriers and facilitators as identified by professionals employed by academic health systems engaged in the implementation of patient self-scheduling?” The research aimed to inform healthcare organizations considering the implementation of self-scheduling. More broadly, the study may enlighten suppliers in the creation and maintenance of the technology for healthcare organizations.
Methods
Design
The consensus process was conducted using a three-stage modified Delphi technique to solicit, identify, and synthesize determinants of the implementation of self-scheduling technology by healthcare organizations. The modified Delphi technique is a structured, participatory qualitative research method.34 Named for the Oracle at Delphi in Ancient Greece, the Delphi technique, which was originally developed by the RAND Corporation in 1948,35,36 involves an iterative process until consensus is obtained. Due to the anonymity of the process, the risk of domination by one individual or coalition is avoided.37 The Delphi method has become a popular technique in health sciences research38 and technology foresight.39 The research method was selected, as the literature lacked evidence of the determinants of implementation of the technology under study.40 As self-scheduling represents an emerging technology for healthcare organizations, the opinions of stakeholders engaged as practitioners of the intervention are important.41,42
The Delphi method can account for key informants who are geographically and professionally diverse.43,44 Given the workload of the panelists during the COVID-19 pandemic, the Delphi technique was selected, as it does not require a specific meeting time, thereby allowing a thoughtful response at a convenient time for participants.45 This research was conducted electronically and was considered to yield the same results as a traditional paper-based survey.46,47
Participants
Participants were identified based on attendance at an educational event held in September 2020 that featured the implementation of automated self-scheduling solutions by academic health systems. Snowball sampling was subsequently applied to identify additional key informants with knowledge of the research subject. Panelists with direct expertise in the implementation of the technology were sought to ensure validity of the consensus statement.48 The author sent communication to 74 potential participants between December 16, 2020, and January 6, 2021, inviting them to participate in the study. The goal was participation from 40 to 60 participants based on other research studies that developed consensus about a complex subject involving different stakeholders.49 Fifty-three agreed to participate; 41 academic health systems were represented. Panelists were from all US Census Bureau-designated regions. (Table 1 presents the count of Delphi panelists by region.) The outpatient enterprises of the academic health systems represented by the panelists ranged from 500,000 to more than 4 million patient encounters per annum.
Delphi Technique
The first Delphi survey was distributed between January 6, 2021 and February 21, 2021, via an online survey tool (SurveyMonkey®) to participants’ email addresses. In the initial round, data on participant demographics were collected to include role, training, and geography. The first round featured an open-ended response to avoid introducing bias in the study.50,51 Participants were asked: “Describe six factors that negatively shape the implementation of self-scheduling at your organization” and “Describe six factors that positively shape the implementation of self-scheduling at your organization.” The responses were documented as barriers or facilitators and mapped in alignment with the Consolidated Framework for Implementation Research (CFIR).52 CFIR enabled the research to be presented in a standard, evidence-based framework, thereby facilitating the opportunity for industry adoption of the research findings.53,54 The CFIR Domains and Constructs are presented in Figure 1.
The second survey was distributed between March 2, 2021, and April 4, 2021. Participants scored agreement or disagreement with statements on a five-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree): “To what extent do you agree with this statement.” The survey was self-administered, thus allowing participants to respond without risk of influencing one another’s answers.55 To describe the relative importance of each item, the median and interquartile range were calculated.56 A consensus point of 80 percent was determined to prioritize the determinants.57-59 During the second round of the online survey, four barriers and 11 facilitators received equal or greater than 80 percent of participants’ votes. The 15 factors were compiled for the next round of the survey.
The third and final step of the Delphi was initiated with the panel on April 27, 2021. Responses were collected through May 26, 2021. Three survey rounds were employed to reach consensus.60 The third and final survey included the 15 factors that received greater than 80 percent of participants’ agreement during the second round. From these, participants were asked to rate each determinant using a Likert scale of 1 to 5 ranging from 1 (strongly disagree) to 5 (strongly agree): “To what extent do you agree with this statement.” The median and interquartile range were calculated.61 Participants were also asked to place in rank order the most important determinants of self-scheduling by healthcare organizations. The percent of the expert panel ranking the factor in the top 10 was also calculated to support the informants’ consensus.62 This iterative process permitted participants to reassess their views considering the aggregated results.63 (See Figure 2.)
The Johns Hopkins Bloomberg School of Public Health’s Institutional Review Board (IRB) determined that this research study does not qualify as human subjects research as defined by DHHS regulations 45 CFR 46.102, and therefore does not require IRB oversight. All participants provided informed consent to take part at the beginning of the process as part of the online survey.
Results
Of the 74 informants identified to participate, 53 agreed to contribute. Of the 53 who agreed to participate, 52 responded to the first round, 53 in the second round, and 52 in the third round. Up to four reminders were sent to encourage participation for each round beginning on the survey due date. The 53 participants represented 41 academic health systems and all regions of the United States. The participants were recruited from three categories: technology professionals (n=9), management professionals (n=41), and other stakeholders in self-scheduling (n=4) (e.g., clinicians who were engaged in the implementation). Management professionals included roles such as executive director of ambulatory operations, chief access officer, and vice president of ambulatory services. Participants could select more than one role. Eight participants were clinicians by training; 44 were not; one was unknown. All participants were individuals employed by healthcare organizations and experienced with a past, current, or future implementation of automated self-scheduling.
In the first round, a total of 530 factors that contributed to the implementation of self-scheduling by healthcare organizations were identified. Fifty-two participants cited a total of 277 factors that negatively shape the implementation of self-scheduling at their organization (barriers), and 253 positively shape the implementation of self-scheduling at their organization (facilitators). Participants submitted an average of 10.2 barriers and facilitators.
Fifty-three key informants responded to the second survey. The author categorized responses from Round 1 into 72 barriers and 85 facilitators based on CFIR. (See Table 2.) The key informants were asked to rate the 157 determinants using a five-point Likert scale. Fifteen factors were identified based on a consensus of 80 percent and higher than 1.0 interquartile range.
The 15 factors were presented to the expert panel in the third and final round. Fifty-two participants rank ordered the 15 factors between 1 (most important) and 15 (least important). The participants were also asked to rate each factor using a five-point Likert scale. The consensus of the panel is presented in Table 3. Three barriers and seven facilitators were identified.
Panel members gave the highest ranking to the enabler that reflected the patients’ needs: “Convenience for patients to schedule appointments via our self-scheduling solution.” The determinant also had the highest consensus, median, and percentage of experts who agreed or strongly agreed. Two other facilitators topped the list: the organizations’ culture to support access and the relative advantage of self-scheduling as compared to the call center. Other facilitators were identified as peer pressure from competitors, the engagement of the academic health systems’ executives, and the buy-in of leaders. Complexity was the primary barrier, as well as providers’ resistance based on specialization and the variability of scheduling protocols.
In total, seven facilitators and three barriers were identified by the expert panelists. Four of the five domains of the CFIR were incorporated in the 10 determinants considered most important to panelists, providing evidence of the broad array of components that influence the implementation of automated self-scheduling. The CFIR domain of “process” was the only one not included in the consensus of key factors. The results of the Delphi panel confirmed myriad determinants of an effective implementation of technology by healthcare organizations.64
Discussion
With this study, stakeholders rated the determinants of implementation for automated self-scheduling by academic health systems. The use of the modified Delphi technique successfully yielded a consensus of the top determinants of implementation to automated self-scheduling as offered by academic health systems. Development of an evidence-based consensus of implementation determinants can be used to further the diffusion of the technology. To the best of the author’s knowledge, this was the first study of its kind for this technology. The discussion of the determinants is presented in the framework of the eight CFIR constructs represented in the panelists’ top-10 list.
Patient Needs and Resources
The most-cited determinant by the Delphi panel was an enabler based on the users’ needs: “Convenience for patients to schedule appointments via our self-scheduling solution.” Recognition of the need for innovation is the initial stage of Rogers Diffusion of Innovation Theory.65 Awareness of the users’ interest was also evident in another top-10 determinant cited by the expert panel: ease of use for patients. Implementation may have been hindered historically by a lack of attention to patient needs. A perceived benefit for patients has led to implementation success by healthcare organizations.66,67 Proactive, clear communication about the benefits of the technology for patients facilitates implementation.68
Culture
Panelists documented and prioritized an organizational culture to promote access to care. There is evidence that culture impacts the success of technology implementation.69-71 The determinant tracks closely with the awareness of the need for the technology to facilitate access for patients, yet it establishes the panel’s perceived priority of the organization’s culture to achieve it.
Relative Advantage
Automated self-scheduling can effectively replace the same transaction over the telephone. The ranking of the relative advantage may reflect the panelists’ belief that automated self-scheduling offers a reduction of personnel costs, access outside of normal operating hours,72,73 improved staff utilization,74-80 and patient time savings.81-83 Regardless of the source of the advantage, the perception that such exists is an essential condition for successful technology implementation.84
Complexity
Acknowledging and analyzing complexity to avoid inadvertent consequences is crucial to the effectiveness of an implementation.85 The ranking of this determinant as the highest barrier may reflect the panel’s perception that the complexity may not be diagnosed or addressed by current solutions. There is an adverse association between the perception of complexity and the success of an intervention.86,87 Automated self-scheduling is a technology purchased, built, and deployed by a healthcare organization. Unlike other well-studied technology solutions like electronic health record systems, however, the primary user is the patient, not the organization, provider, or employee. The implementation of a patient-facing solution adds to the complexity and may increase the challenges of implementation.88
Knowledge and Beliefs about the Intervention
Providers’ resistance has been demonstrated in other studies about novel healthcare technology.89 Factors include fear and dissatisfaction with roles and responsibilities,90 a lack of trust,91 resistance to change,92 and uncertainty.93 Studies regarding physician receptivity, however, have centered on the implementation of electronic health record systems or their components. Similar barriers may exist for an administrative technology. The rationale regarding specialization may reveal the source of resistance, one that tracks closely with the previous barrier related to complexity.
Peer Pressure
The need to be competitive was revealed as the sixth facilitator to self-scheduling implementation. Panelists may consider self-scheduling to be a requirement rather than a luxury. This may reflect a mimetic response by healthcare organizations as it relates to competitors, considered to be highly influential for adoption of technology.94 The competitive environment for healthcare organizations is significant, with mergers and acquisitions predicted to increase in the future based on various policy changes and financial positions.95 Reacting to peers has been demonstrated to be particularly influential for organizations that are late adopters.96
Readiness for Implementation – Leadership Engagement
The expert panel concluded the involvement of leaders as important facilitators. Engagement of leaders has been determined to be of significance in all facets of technology implementation in healthcare.97-99 The inclusion of two determinants related to the involvement of leaders in the top-10 list promotes its import as a facilitator of implementation. The ranking of executive leadership engagement may reveal that direct management support is not sufficient for implementation success. As self-scheduling involves stakeholders both internal and external to the organization, executive leaders may be a crucial facilitator for automated self-scheduling.
Adaptability
The final top-10 determinant, “variability about scheduling protocols across providers or specialties within a department,” reflects an intervention characteristic. Adaptability is recognized as a critical factor as an intervention is disseminated more broadly within an organization.100 Ease of modification is positively correlated with an effective implementation.101-103
The Delphi panel’s key barriers and facilitators for self-scheduling offered insight into experts’ perceptions of determinants of implementation success. The factors that are absent from the list may be of equal import. “Process” was the only CFIR domain that was not represented in the consensus of determinants. According to CFIR,104 the domain, which incorporates engaging, executing, planning, and reflecting and evaluating, is the “single most difficult domain to define, measure, or evaluate in implementation research.” The lack of the domain being considered as a barrier or facilitator may confirm the placement of automated self-scheduling at the beginning of the technology’s life cycle.
The absence of cost (a construct within the “intervention characteristics” domain) and available resources (a construct within “inner setting”) may indicate that financial outlay for the technological solution is not a barrier. Time, effort, and resources, however, may be needed for healthcare organizations to address barriers to patients’ technology acceptance, a journey that has been determined to be present, complex, and nonlinear.105,106 The presence of a digital divide has been well documented for other technologies,107,108 and its absence as a barrier may also reflect the stage of the technology’s life cycle. As the technology is diffused, additional research regarding the digital divide is warranted.
Opportunities for Research
Further research is warranted to identify actions that may address the barriers and facilitators to implementation of self-scheduling technology. The research ascertained the determinants. Healthcare organizations may now proactively tackle the barriers and seek facilitators to increase diffusion of the technology. For example, organizations may survey patients regarding their expectations for a digital access experience, using reports that feature the voice of the customer to draw the organizations’ attention to the most important facilitator, the delivery of convenience. An inventory of competitors’ capabilities may be shared with leadership to address peer pressure. Known barriers such as providers’ resistance may be addressed proactively by open dialogue with providers about the technology, a step that may have otherwise been overlooked in the belief that the technological solution was solely administrative. Table 4 lists actions for healthcare organizations to consider to remove barriers and promote facilitators based on the determinants identified by the expert panelists. Further research is warranted to identify effective actions to address each determinant.
Limitations
The Delphi technique has been criticized for the potential for bias in participant selection and engagement.109 This study strived to overcome the bias through the variety of participants as it relates to geography, professional roles, and training.110 The value of the Delphi technique is determined by the quality and stability of the panel of participating experts and the time between rounds, which were proactively managed by the author.111 Participants represented various roles in academic health systems; however, they may not have represented persons from all areas of responsibility for implementation. The panel did not contain the opinions of suppliers (persons creating the technology) or patients (persons using the technology). As the research study aimed to develop a consensus for the implementation of the technology by healthcare organizations, these stakeholders were purposely excluded. This may have introduced bias in the results. The panels of experts represented healthcare organizations that were academic health systems; the ambulatory clinics associated with these healthcare organizations are large and complex. Gathering consensus from experts who represented academic health systems may limit generalizability of the results.
Conclusion
The purpose of this study was to provide consensus from a panel of experts engaged in automated self-scheduling about the barriers and facilitators to this novel technology. The Delphi method was effective in identifying 10, rank-ordered determinants of implementation success. The research may inform stakeholders about current priorities to consider the deployment and dissemination of this technology within healthcare organizations, thus contributing to the adoption of evidence-based practices to promote improvement efforts in managing service, access, and utilization of the ambulatory enterprise.
Acknowledgements
The author would like to thank the expert panelists for their participation in this research and Drs. Doug Hough, Kathryn McDonald, Michael Rosen, Aditi Sen, Jonathan Weiner, and Christina Yuan of Johns Hopkins Bloomberg School of Public Health for their guidance and support.
Notes
1. Collaborative, Patient Access. 2021 Benchmark Report. Atlanta: Patient Access Collaborative, 2021.
2. Friedman, J. “Internet Patient Scheduling in Real-life Practice.” The Journal of Medical Practice Management. 2004. 20(1), pp. 13-15.
3. Jones R, Menon-Johansson A, Waters AM, Sullivan AK. “eTriage - a novel, web-based triage and booking service: enabling timely access to sexual health clinics. Int J STD AIDS. 2010. 21(1), pp. 30-33.
4. Idowu, P., Adeosun, O., and Williams, K. “Dependable Online Appointment Booking System for NHIS Outpatient in Nigerian Teaching Hospitals.” International Journal of Computer Science and Information Technology. 2014. 6, pp. 59-73.
5. Kamo, N, Bender, J, Blackmore, C. “Meaningful use of the electronic patient portal – Virginia Mason's journey to create the perfect online patient experience” 2017. Healthcare.
6. Zhao, P., Yoo, I., Lavoie, J., Lavoie, B. J., & Simoes, E. “Web-Based Medical Appointment Systems: A Systematic Review.” Journal of Medical Internet Research. 2017. 19(4), e134. https://doi.org/10.2196/jmir.6747.
7. Lee, Y. P., Tsai, H. Y., and Ruangkanjanases, A. “The determinants for food safety push notifications on continuance intention in an e-appointment system for public health medical services: The perspectives of utaut and information system quality.” International Journal of Environmental Research and Public Health. 2020. 17(21), pp. 1-15.
8. Gupta, D., and Denton, B. “Appointment scheduling in health care: Challenges and opportunities.” 2008. IIE Transactions, 40(9), pp. 800–819.
9. Chen, S, Liu S, Li S, Yen DC. “Understanding the mediating effects of relationship quality on technology acceptance: an empirical study of e-appointment system.” 2013. J Med Syst, 37(6), p. 9981.
10. Volk AS, Davis MJ, Abu-Ghname A, Warfield RG, Ibrahim R, Karon G, Hollier LH Jr. “Ambulatory Access: Improving Scheduling Increases Patient Satisfaction and Revenue.” 2020. Plast Reconstr Surg, 146(4). doi: 10.1097/PRS.0000000000007195, pp. 913-919.
11. Craig, A. “Self-scheduling appointments.” 2007. Advanced Nurse Practitioner, 15(10), pp. 24-25.
12. Parmar V, Large A, Madden D, Das V. “The online outpatient booking system ‘Choose and Book’ improves attendance rates at an audiology clinic: a comparative audit.” 2009. Informatics in Primary Care, 17, pp. 183–6.
13. Siddiqui, Z. Rashid, R. “Cancellations and patient access to physicians: Zocdoc and the evolution of e-medicine.” 2013. Dermatology Online Journal, 19(4), p. 14.
14. Paré, G., Trudel, M.C., and Forget, P. “Adoption, use, and impact of e-booking in private medical practices: mixed-methods evaluation of a two-year showcase project in Canada.” 2014. JMIR Medical Informatics, 2(2). Doi: 10.2196/medinform.3669., p. e24.
15. Yanovsky RL, Das, S. “Patient-initiated online appointment scheduling: Pilot program at an urban academic dermatology practice.” 2020. J Am Acad Dermatol, 83(5). doi: 10.1016/j.jaad.2020.03.035., pp. 1479-1481.
16. Xie, H., Prybutok, G., Peng, X., and Prybutok, V. “Determinants of Trust in Health Information Technology: An Empirical Investigation in the Context of an Online Clinic Appointment System.” 2020. International Journal of Human-Computer Interaction, 36(12).
17. Chen, S, Liu S, Li S, Yen DC. “Understanding the mediating effects of relationship quality on technology acceptance: an empirical study of e-appointment system.” 2013. J Med Syst, 37(6), p. 9981.
18. Zhang, M., Zhang, C., Sun, Q. Cai, Q, Yang, H and Zhang, Y. “Questionnaire survey about use of an online appointment booking system in one large tertiary public hospital outpatient service center in China.” 2014. BMC Medical Informatics and Decision Making.
19. Chang HH, Chang CS. “An assessment of technology-based service encounters & network security on the e-health care systems of medical centers in Taiwan.” 2008. BMC Health Serv Res. 17(8). doi: 10.1186/1472-6963-8-87., p. 87.
20. Kurtzman, GW, Keshav, MA, Satish, NP, Patel, MS. “Scheduling primary care appointments online: Differences in availability based on health insurance.” 2018. Healthcare, 6(3). doi: 10.1016/j.hjdsi.2017.07.002., pp. 186-190.
21. Chiu, C.-K. “Understanding relationship quality and online purchase intention in e-tourism: A qualitative application.” 2009. Qual. Quan. 43(4), pp. 669–675.
22. Jannsson, B. “Choosing a Good Appointment System—A Study of Queues of the Type.” 1966. Operations Research, 14(2).
23. Mak, HY, Rong Y, Zhang J. “Sequencing appointments for service systems using inventory approximations.” 2014. Manufacturing & Service Operations Management, 16., pp. 251-262.
24. Shugan, S. and Zie, J. “Advance Pricing of Services and Other Implications of Separating Purchase and Consumption.” 2000. Journal of Service Research, 2(3). doi: 10.1177/109467050023001, pp. 227-239.
25. Robinson LW, Chen RR. “Estimating the implied value of the customer’s waiting time.” 2011. Manufacturing & Service Operations Management, 13., pp. 53-57.
26. Zhao, P., Yoo, I., Lavoie, J., Lavoie, B. J., & Simoes, E. “Web-Based Medical Appointment Systems: A Systematic Review.” 2017. Journal of Medical Internet Research, 19(4), e134. https://doi.org/10.2196/jmir.6747.
27. Versel, N. “Online Reservations: Letting Patients Make Their Own Appointments.” Providers Edge. [Online] March 22/29, 2004. http://www.providersedge.com/ehdocs/ehr_articles/online_reservations-letting_patients_make_their_own_appointments.pdf..
28. Lowes, R. “Let patients book their own appointments?” 2006. Med Econ, 83(11)., pp. 27-28.
29. Riddell, S. “Technology: the race to get the NHS online.” 2012. Health Serv J, 122(6319), p. 32.
30. Farr, C. “Dentistry takes the cybercure: scheduling, consultations, records move to the net.” 2000. Dent Today, 19(5), pp. 106-113.
31. Nadarzynski, T., Miles, O., Cowie, A., and Ridge, D. “Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A mixed-methods study.” 2019. Digital Health, 5. doi: 10.1177/2055207619871808.
32. Zhao, P., Yoo, I., Lavoie, J., Lavoie, B.J. and Simoes, E. “Web-Based Medical Appointment Systems: A Systematic Review.” 2017. Journal of Medical Internet Research, p. e134.
33. Wachter, R. “Making IT work: harnessing the power of health information technology to improve care in England.” Report of the National Advisory Group on Health Information Technology in England. London, UK: Department of Health. [Online] September 2016. [Cited: 2016.] https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/550866/Wachter_Review_Accessible.pdf.
34. Dalkey, N and Helmer, O. “An Experimental Application of the Delphi Method to the Use of Experts.” Massachusetts: Reading (MA): Addison-Wesley. 1963. Management Science 9(3), pp. 458-467.
35. Helmer, Olaf, and Nicholas Rescher. “On the epistemology of the inexact sciences.” 1959. Management Science, Vol. 6, pp. 25-52.
36. Dalkey, N and Helmer, O. “An Experimental Application of the Delphi Method to the Use of Experts.” Massachusetts: Reading (MA): Addison-Wesley. 1963. Management Science 9(3), pp. 458-467.
37. Jairath, N., and J. Weinstein. “The Delphi methodology (Part one): A useful administrative approach.” 1994. Canadian Journal of Nursing Administration, Vol. 7, pp. 29-42.
38. De Villiers, Marietjie R., Pierre JT De Villiers, and Athol P. Kent. “The Delphi technique in health sciences education research.” 7, 2005. Vol. 27, pp. 639-643.
39. Birko, Stanislav, Edward S. Dove, and Vural Özdemir. “Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation.” 8, 2015. PloS one, Vol. 10, p. e0135162.
40. Jones, J and Hunter, D. “Consensus methods for medical and health services research.” 1995. British Medical Journal, Vol. 311, pp. 376-380.
41. Meshkat, Babak, Seamus Cowman, Georgina Gethin, Kiaran Ryan, Miriam Wiley, Aoife Brick, Eric Clarke, and Eadhbhard Mulligan. “Using an e-Delphi technique in achieving consensus across disciplines for developing best practice in day surgery in Ireland.” 2014. Journal of Hospital Administration, Vol. 3.
42. Trevelyan, EGR, Robison, N. “Delphi Methodology in Health Research: How to Do It?” 2015. European Journal of Integr. Medicine, 7(4), p. 428.
43. Jairath, N., and J. Weinstein. “The Delphi methodology (Part one): A useful administrative approach.” J3, 1994. Canadian Journal of Nursing Administration, Vol. 7, pp. 29-42.
44. Linstone, Harold A., and Murray Turoff. “The Delphi Method.” Massachusetts : Reading, MA: Addison-Wesley, 1975. Vol. 29.
45. Schmidt, Roy C. “Managing Delphi surveys using nonparametric statistical techniques.” 3, 1997. Decision Sciences, Vol. 28, pp. 763-774.
46. Boulkedid, R, Abdoul, H, Loustau, M, et al. “Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.” 2011. Plos One, p. 6:e20476.
47. Gill, Fenella J., Gavin D. Leslie, Carol Grech, and Jos M. Latour. “Using a web-based survey tool to undertake a Delphi study: Application for nurse education research.” 11, 2013. Nurse Education Today, Vol. 33, pp. 1322-1328.
48. Hasson, Felicity, and Sinead Keeney. “Enhancing rigour in the Delphi technique research.” 9, 2011. Technological Forecasting and Social Change, Vol. 78, pp. 1695-1704.
49. Santaguida, Pasqualina, Lisa Dolovich, Doug Oliver, Larkin Lamarche, Anne Gilsing, Lauren E. Griffith, Julie Richardson, Dee Mangin, Monika Kastner, and Parminder Raina. “Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care.” 1, 2018. BMC family practice, Vol. 19, pp. 1-14.
50. Sinha, Ian P., Rosalind L. Smyth, and Paula R. Williamson. “Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies.” 1, 2011. PLOS Medicine, Vol. 8, p. e1000393.
51. Custer, Rodney L., Joseph A. Scarcella, and Bob R. Stewart. “The modified Delphi technique: A rotational modification.” 2, 1999. Journal of Vocational and Technical Education, Vol. 15, pp. 1-10.
52. CFIR Guide. [Online] 2021. https://cfirguide.org/constructs/process/.
53. Damschroder, L, Arron, D., Keith, R., et al. “Fostering Implementation of Health Services Research Findings into Practice: a Consolidated Framework for Advancing Implementation Science.” 2009. Implementation Science, 4, pp. 4-50.
54. Birken, SA, Powell, BJ, Shea, CM, et al. “Criteria for selecting implementation science theories and frameworks: results from an international survey.” 2017. Implementation Science, 12(1), p. 124.
55. Jairath, N., and J. Weinstein. “The Delphi methodology (Part one): A useful administrative approach.” 3, 1994. Canadian Journal of Nursing Administration, Vol. 7, pp. 29-42.
56. Jones, J and Hunter, D. “Consensus methods for medical and health services research.” 7001, 1995. British Medical Journal, Vol. 311, pp. 376-380.
57. Fink, Arlene, Jacqueline Kosecoff, Mark Chassin, and Robert H. Brook. “Consensus methods: characteristics and guidelines for use.” 9, 1984. American Journal of Public Health, Vol. 74, pp. 979-983.
58. Powell, Catherine. “Methodological issues in nursing research.” 4, s.l. Journal of Advanced Nursing. 2003. “The Delphi technique: myths and realities,” Vol. 41, pp. 376-382.
59. Lynn, Mary R. “Determination and quantification of content validity.” 6, 1986. Nursing Research, Vol. 35, pp. 382–385.
60. Iqbal, S. “The Delphi method.” 7, 2009. Psychologist, Vol. 22, pp. 598-601.
61. Jones, J and Hunter, D. “Consensus methods for medical and health services research.” 7001, 1995. British Medical Journal, Vol. 311, pp. 376-380.
62. Katcher, M. L., A. N. Meister, C. A. Sorkness, A. G. Staresinic, S. E. Pierce, B. M. Goodman, N. M. Peterson, P. M. Hatfield, and J. A. Schirmer. “Use of the modified Delphi technique to identify and rate home injury hazard risks and prevention methods for young children.” 3, 2006. Injury Prevention, Vol. 12, pp. 189-194.
63. Khayatzadeh-Mahani, Akram, Krystle Wittevrongel, David B. Nicholas, and Jennifer D. Zwicker. “Prioritizing barriers and solutions to improve employment for persons with developmental disabilities.” 18, 2020. Disability and Rehabilitation, Vol. 42, pp. 2696-2706.
64. Durlak, Joseph A., and Emily P. DuPre. “Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation.” 3, 2008. American Journal of Community Psychology, Vol. 41, pp. 327-350.
65. Rogers, Everett M. “Diffusion of innovations.” New York: Free Press : s.n., 2003.
66. Shortell, Stephen M., Jill A. Marsteller, Michael Lin, Marjorie L. Pearson, Shin-Yi Wu, Peter Mendel, Shan Cretin, and Mayde Rosen. “The role of perceived team effectiveness in improving chronic illness care.” 11, 2004. Medical Care, Vol. 42, pp. 1040-1048.
67. Waneka, Renae, and Joanne Spetz. “Hospital information technology systems' impact on nurses and nursing care.” 12, 2010. The Journal of Nursing Administration, Vol. 40, pp. 509-514.
68. Studer, Melanie. “The effect of organizational factors on the effectiveness of EMR system implementation--What have we learned?” 4, 2005. Healthcare Quarterly (Toronto, Ont.), Vol. 8, pp. 92-98.
69. Ostroff C, Kinicki AJ, Tamkins MM. Ostroff C, Kinicki AJ, Tamkins MM: Organizational culture and climate. “Handbook of Psychology: Volume 12, Industrial and Organizational Psychology.” [book auth.] Kinicki AJ, Tamkins MM: “Organizational Culture and Climate.” Handbook of Psychology: Volume 12, IndustEdited by: Borman WC, Ilgen DR, Klimoski RJ. 2 Ostroff C. New York : Wiley, 2003, pp. 565-587.
70. Harper, G. and Utley, D. “Organizational Culture and Successful Information Technology Implementation.” 2001. Engineering Management Journal, 13:2, pp. 11-15.
71. Police, Rachel, Talia Foster, and Ken Wong. “Adoption and use of health information technology in physician practice organisations: systematic review.” 4, 2010. Journal of Innovation in Health Informatics, Vol. 18, pp. 245-258.
72. Ryan, M., Waller, J., Marlow, LAV. “Could Changing Invitation and Booking Processes Help Women Translate their Cervical Screening Intentions into Action? A Population-based Survey of Women’s Preferences in Great Britain.” 2019. BMJ Open, p. 9:e028134.
73. Ryan, M., Marlow, L., Forster, A., Ruwende, J., and Waller, J. “Offering an app to book cervical screening appointments: A service evaluation.” 2020. Journal of Medical Screening, 27(2), pp. 85-89.
74. Friedman, J. “Internet Patient scheduling in Real-life Practice.” 2004. The Journal of Medical Practice Management, 20(1), pp. 13-15.
75. Volk, AS, Davis, MJ, Abu-Ghname, A, Warfield, RG, Ibrahim, R, Karon, G, Hollier, LH Jr. “Ambulatory Access: Improving Scheduling Increases Patient Satisfaction and Revenue.” 2020. Plastic Reconstructive Surgery, 146(4), pp. 913-919.
76. Yanovsky, RL, Das, S. “Patient-initiated online appointment scheduling: Pilot program at an urban academic dermatology practice.” 2020. J Am Acad Dermatol, pp. 1479-1481.
77. Zhang, X, Yu, P, Yan, J. “Patients' adoption of the e-appointment scheduling service: A case study in primary healthcare.” 2014. Stud Health Technol Inform. 204, pp. 176-81.
78. Zhang, X., Yu, P, Yan, J, Ton, AM, Spil, I. “Using diffusion of innovation theory to understand the factors impacting patient acceptance and use of consumer e-health innovations: a case study in a primary care clinic.” 2015. BMC Health Serv Res. 21, pp. 15-71.
79. Ganguli, I., Orav, EJ., Lupo, C., Metlay, JP, Sequist, TD. “Patient and Visit Characteristics Associated With Use of Direct Scheduling in Primary Care Practices.” 2020. JAMA Netw Open 3(8), p. e209637.
80. Chen, P, Xiao L, Gou Z, Xiang L, Zhang X, Feng P. “Telehealth attitudes and use among medical professionals, medical students and patients in China: A cross-sectional survey.” 2017. Int J Med Inform, 108, pp. 13-21.
81. Judson, TJ, Odisho AY, Neinstein AB, Chao J, Williams A, Miller C, Moriarty T, Gleason N, Intinarelli G, Gonzales R. “Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19.” 2020. J Am Med Infor, pp. 860-866.
82. Jones, R, Menon-Johansson, A, Waters, AM, Sullivan, AK. “eTriage - a novel, web-based triage and booking service: enabling timely access to sexual health clinics.” 2010. Int J STD AIDS, pp. 30-33.
83. Denizard-Thompson, N.M., Feiereisel, K.B., Stevens, S.F. et al. “The Digital Divide at an Urban Community Health Center: Implications for Quality Improvement and Health Care Access.” 2011. J Community Health, 36, pp. 456-460.
84. Greenhalgh, Trisha, Glenn Robert, Fraser Macfarlane, Paul Bate, and Olivia Kyriakidou. “Diffusion of innovations in service organizations: systematic review and recommendations.” 4, 2014. The Milbank Quarterly, Vol. 82, pp. 581-629.
85. Kochevar, Laura K., and Elizabeth M. Yano. “Understanding health care organization needs and context.” 2, 2006. Journal of General Internal Medicine, Vol. 21, p. S25.
86. Greenhalgh, Trisha, Glenn Robert, Fraser Macfarlane, Paul Bate, and Olivia Kyriakidou. “Diffusion of innovations in service organizations: systematic review and recommendations.” 4, 2014. The Milbank Quarterly, Vol. 82, pp. 581-629.
87. Gustafson, David H., François Sainfort, Mary Eichler, Laura Adams, Maureen Bisognano, and Harold Steudel. “Developing and testing a model to predict outcomes of organizational change.” 2, 2003. Health Services Research, Vol. 38, pp. 751-776.
88. Lewy, Hadas. “Wearable technologies–future challenges for implementation in healthcare services.” 1, 2015. Healthcare Technology Letters, Vol. 2, pp. 2-5.
89. Rathert, Cheryl, Tracy H. Porter, Jessica N. Mittler, and Michelle Fleig-Palmer. “Seven years after Meaningful Use: Physicians’ and nurses’ experiences with electronic health records.” 1, 2019, Health Care Management Review, Vol. 44, pp. 30-40.
90. Boonstra, Albert, and Manda Broekhuis. “Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions.” 10, 2013. BMC Health Services Research, p. 23.
91. Schreiweis, Björn, Monika Pobiruchin, Veronika Strotbaum, Julian Suleder, Martin Wiesner, and Björn Bergh. “Barriers and facilitators to the implementation of ehealth services: systematic literature analysis.” 11, 2019. Journal of Medical Internet Research, Vol. 21, p. e14197.
92. Kruse, CS and Kristof, C, Jones, B, Mitchell, E, Martinez, A. “Barriers to electronic health record adoption: a systematic literature review.” 12, 2016, Journal of Medical Systems, Vol. 40, pp. 1-7.
93. Moxey, Annette, Jane Robertson, David Newby, Isla Hains, Margaret Williamson, and Sallie-Anne Pearson. “Computerized clinical decision support for prescribing: provision does not guarantee uptake.” 1, 2010. Journal of the American Medical Informatics Association, Vol. 17, pp. 25-33.
94. Greenhalgh, Trisha, Glenn Robert, Fraser Macfarlane, Paul Bate, and Olivia Kyriakidou. “Diffusion of innovations in service organizations: systematic review and recommendations.” 4, 2014. The Milbank Quarterly, Vol. 82, pp. 581-629.
95. Carroll, John D. The State of Competition in the U.S. Healthcare Industry. The National Law Review. [Online] March 31, 2021. [Cited: June 6, 2021.] https://www.natlawreview.com/article/state-competition-us-healthcare-industry.
96. Walston, Stephen Lee, John R. Kimberly, and Lawton Robert Burns. “Institutional and economic influences on the adoption and extensiveness of managerial innovation in hospitals: The case of reengineering.” 2, 2011. Medical Care Research and Review, Vol. 58, pp. 194-228.
97. Moxey, Annette, Jane Robertson, David Newby, Isla Hains, Margaret Williamson, and Sallie-Anne Pearson. “Computerized clinical decision support for prescribing: provision does not guarantee uptake.” 1, 2010. Journal of the American Medical Informatics Association, Vol. 17, pp. 25-33.
98. Yusof, Maryati Mohd, Lampros Stergioulas, and Jasmina Zugic. “Health information systems adoption: findings from a systematic review.” 1, 2007. Studies in Health Technology and Informatics, Vol. 129, p. 262.
99. Studer, Melanie. “The effect of organizational factors on the effectiveness of EMR system implementation--What have we learned?” 4, 2005. Healthcare Quarterly (Toronto, Ont.), Vol. 8, pp. 92-98.
100. Mendel, Peter, Lisa S. Meredith, Michael Schoenbaum, Cathy D. Sherbourne, and Kenneth B. Wells. “Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research.” 1-2, 2008. Administration and Policy in Mental Health and Mental Health Services Research, Vol. 35, pp. 21-37.
101. Gustafson, David H., François Sainfort, Mary Eichler, Laura Adams, Maureen Bisognano, and Harold Steudel. “Developing and testing a model to predict outcomes of organizational change.” 2, 2003. Health Services Research, Vol. 38, pp. 751-776.
102. Rogers, Everett M. Diffusion of Innovations. New York: Free Press : s.n., 2003.
103. Leeman, Jennifer, Marianne Baernholdt, and Margarete Sandelowski. “Developing a theory‐based taxonomy of methods for implementing change in practice.” 2, 2007. Journal of Advanced Nursing, Vol. 58, pp. 191-200.
104. CFIR Guide. [Online] 2021. https://cfirguide.org/constructs/process/.
105. Nadal, Camille, Corina Sas, and Gavin Doherty. “Technology acceptance in mobile health: scoping review of definitions, models, and measurement.” 7, 2020. Journal of Medical Internet Research, Vol. 22, p. e17256.
106. Loncar-Turukalo, Tatjana, Eftim Zdravevski, José Machado da Silva, Ioanna Chouvarda, and Vladimir Trajkovik. “Literature on wearable technology for connected health: scoping review of research trends, advances, and barriers.” 9, 2019. Journal of Medical Internet research, Vol. 21, p. e14017.
107. Chang, J, Yuanhony Lai, A, Gupta, A, Nguyen, A, Berry, C, Shelley, D. “Rapid Transition to Telehealth and the Digital Divide: Implications for Primary Care Access and Equity in a Post-COVID Era.” 2021, Milbank Quarterly, p. Online ahead of print.
108. Hochmuth, A, Wrona, K, Exner, A-K, Dockweiler, C. “Digitization and health inequality and equity in nursing.” 2021, Pflege, pp. 151-158.
109. Sackman, H. Delphi Critique: Expert Opinion, Forecasting and Group Process. s.l. : Lexington Books, 1974.
110. Khayatzadeh-Mahani, Akram, Krystle Wittevrongel, David B. Nicholas, and Jennifer D. Zwicker. “Prioritizing barriers and solutions to improve employment for persons with developmental disabilities.” 18, 2020. Disability and Rehabilitation, Vol. 42, pp. 2696-2706.
111. Rathert, Cheryl, Tracy H. Porter, Jessica N. Mittler, and Michelle Fleig-Palmer. “Seven years after Meaningful Use: Physicians’ and nurses’ experiences with electronic health records.” 1, 2019. Health Care Management Review, Vol. 44, pp. 30-40.
Author Biographies
Elizabeth Woodcock (elizabeth@elizabethwoodcock.com) is the executive director of the Patient Access Collaborative and an adjunct assistant professor in the Rollins School of Public Health at Emory University.