Methods and Lessons Learned from a Current State Workflow Assessment following Transition to a New Electronic Health Record System

Abstract

The transition to a new electronic health record (EHR) system requires an understanding of how the new system addresses the needs, business processes, and current activities of a healthcare system. To address such requirements, a multidisciplinary team conducted a current state workflow assessment (CSWFA) of clinical and administrative functions to elicit and document business processes (via process diagrams), requirements, workarounds, and process issues (i.e., user interface issues, training gaps) at one healthcare facility. We provided a novel method of evaluating the implementation process to ensure that a CSWFA was documented with key stakeholders. In this analysis, we describe the CSWFA approach and expected outcomes with a specific emphasis on how a qualitative approach can be integrated to explore underlying patterns and relationships in the data. Overall, this methodology enables practitioners to deliver data-driven support initiatives that optimize EHR implementation while considering user experience, productivity, and patient safety.

Keywords: Workflow assessment, electronic health record, user experience, EHR transition, EHR implementation

Introduction

As healthcare systems work to facilitate clinical and administrative processes, care delivery, and decision-making activities while at the same time improving quality and reducing medical errors¹,², there continues to be an uptick of the implementation of electronic health records (EHRs) throughout the United States. In fact, many healthcare systems are now replacing or converting their existing EHRs to improve workflow, efficiency, and quality.³ Waiting to evaluate an EHR until after it has been implemented creates many challenges and impacts the delivery of optimal care after transition to the new system.³ Thus, it is imperative to understand what, if any, barriers may have existed in terms of workflows and processes. Such an evaluation is especially important because distinct processes to ensure patient care quality were already in place and such behaviors may be challenging to modify after the implementation of a new system. A systematic review of EHR implementation found that documentation time increased significantly after a new EHR was put in place, but workflow (defined as the sequence of physical and mental tasks performed by end users within and among work environments) improved after staff became more familiar with the new system.⁴

Common post-EHR implementation issues include those that involve end users, such as workarounds in workflows and processes after the EHR vendor or its designees were unable to resolve it, as well as user interface issues and staff training needs.^2,4,5,6 Following EHR implementation, managers and informatics staff need a user-friendly data collection tool to better understand if system end-users are able to effectively use the EHR system. This data will inform decision makers on whether workflows are running smoothly and the extent to which workarounds exist.⁵ To date, there is limited information tailored for health information managers and planners that not only collects but evaluates the needs of the end user.

Through a partnership with a large integrated healthcare system in the United States, the purpose of the present study was to examine the business and operational processes that occurred at a single hospital after the implementation of a new EHR. The objective of this study is two-fold. First, we describe and disseminate a technique, Current State Workflow Assessments (CSWFA), that quantifies process issues (e.g., workarounds) that end-users may perform after the implementation of a new EHR. Second, we describe the data that was generated from a CSWFA at a single hospital after implementation of the new EHR. Findings from the CSWFA capture the as-is operational understandings that support EHR conversion or implementation. The results are designed not only to inform process improvements, but also to identify components for workflow optimization and information-system redesign.³

Methods

This assessment engaged leaders and technical teams from multiple informatics-related divisions within a large healthcare system. More specifically, an informatics project team comprised of clinical informaticists, human factors engineers, business process analysts, process architects, requirements analysts, and business process re-engineers, collaborated to deliver a current state assessment of primary care services at one health care facility following deployment of a new EHR. This process, referred to as CSWFA, included the documentation of business processes for six primary care service areas, as well as gaps and pain points reported by end users who had transitioned to the new system. Creating, populating, and managing a repository allowed human factors engineers to generate a dataset for quantifying and categorizing captured gaps and pain points based on the nature of each issue and the impact of each on job satisfaction, productivity, patient safety, and other related factors. All activities associated with CSWFA were conducted virtually using Microsoft Teams. The virtual environment enabled diverse participant involvement from medical facility staff and project team members facilitating the sessions, which was key considering the multidisciplinary approach.

CSWFA Planning

Based on the number of service areas targeted, and the complexity of these services, multiple planning sessions occurred, allowing the site and project team members to convene to identify “deep dive areas” or focus areas of interest that would be prioritized in preparation for the workflow assessment. Focus areas were identified by site staff and leadership based on clinical domains that were experiencing issues, barriers, or unclear definition of processes.  Moreover, the team deliberated prior to these sessions to develop preliminary “as-is” process models.  Existing workflows provided by the EHR vendor were leveraged to create the as-is models using Business Process Model and Notation (BPMN) ⁷, a standardized graphical notation that uses shapes, symbols, and rules for creating process models that are easy to understand. Additional documentation such as existing site-level workflows, Standard Operating Procedures (SOPs), clinical practice guidelines, and policies were also leveraged in the development of As-Is models prior to the beginning of each Workflow Elaboration Session.

Workflow Elaboration Sessions

One, 90-minute Workflow Elaboration Session was conducted for each “deep dive” area in order to clarify and/or confirm the steps associated with the business process being reviewed. An informaticist, a process architect (modeler) and human factors engineer facilitated the session with a specific focus on tasks; user roles responsible for completing tasks; data objects or artifacts associated with tasks; pain points; workarounds; and systems or software being used. Subject matter experts (SMEs) interviewed were system end-users including providers, nurses, managers, pharmacists, and local informatics staff.  During sessions, as key details on the process flow were elicited^7,8, as-is workflows (developed during CSWFA planning) and Current State Workbooks were simultaneously updated to document changes and key details shared by facility staff. Human factors engineers asked specific questions to probe into each SME’s experience using the new system with a specific emphasis on the system interface, as well as the challenges experienced when using the system. Sample questions used by engineers as guides during Workflow Elaboration Sessions can be found in Figure 1.

Questions posed during Workflow Elaboration Sessions enabled the capture of narrative data that could be leveraged to perform qualitative analytics and to answer questions that inform problem identification and issue resolution. This included elements such as systems used, devices required to perform the task, user roles, and triggering events, while also allowing the team to identify and document gaps, inefficiencies, pain points, manual processes, and workarounds implemented by the site.

Workbook Normalization and Review Sessions

Following each Workflow Elaboration Session, one Workbook Normalization and Review Session was conducted to review data captured and validate findings.  The overarching goal of each session was to confirm each captured gap/pain point, while leveraging the expertise of at least one clinical informaticist and at least one human factors engineer to classify each line item by the nature of the specified issue and the impacts it has on the user. This discussion, facilitated by a human factors engineer and a business process re-engineer, encouraged collaborative communication and allowed the informatics team to concur on definitions for elicited pain points before coming to a consensus on the classifications. For each step in the model, the human factors engineer reviewed all captured gaps and requested feedback from at least one clinical informaticist to validate that the pain point was captured accurately. This approach also ensured that expertise from critical areas such as human factors, cognitive engineering, and/or safety engineering was acknowledged in the assessment of each gap.⁹

Qualitative Analysis

We used text-based analytic visualizations to explore the body of text within each Workbook Elaboration Session and to identify themes. We used several approaches for viewing how important a word was to the content in the corpus.¹⁰ Lists of simple word frequencies and other ways to view word significance were viewed by the team. The team also viewed other NLP visualizations as we began to conceptualize themes. The team manually reviewed the list of remaining keywords excluding single characters and function words along with general and nonapplicable words decided upon by the team. With the remaining words, a co-occurrence network was created to help visualize themes. The thematic analysis of key data insights further allowed researchers to understand patterns, relationships, and commonalities among recurring terms. For more information on processes for designing co-occurrence networks, see Arnold (2022).¹¹

Orange data mining toolbox¹² was used to create a pipeline for interactive word co-occurring network visuals. In Orange, we used the preprocess text widget to tokenize the text; connected and configured the corpus to network widget to create co-occurring word networks; used the network visualization widget to view the results; and explored network visualizations while iteratively configuring.

Results/Findings

The Co-occurrence Network (Figure 2) illustrates the frequency of use of terms in the narratives captured in the Workflow Elaboration Sessions and further refined in the Workbook Normalization/Review Sessions. Co-occurrence networks and other Natural Language Processing (NLP) visualizations can be displayed dynamically to systems studies and design teams to support exploration. These visualizations can be configured during the tokenization process, by tailoring frequency or significance assignment or tailoring thresholds of visualization features and relationships. The co-occurrence network in Figure 2 represents the visualization based on one set of thresholds.

There are many visualizations available in text analytics and NLP software including word tree representations, corpus and topic model viewers, and clustering visuals. Exploring options for and iterating on some combination of visualizations for viewing text may help with discovery activities. Approaches should be tailored to the context of work and analyst team’s preferences.  Mihalcea and Radev (2011) provide insight on and uses for graph-based NLP.¹³

The linguistic features and relationships depicted can prompt the review team to further explore the context of specific narratives captured in the Workflow Elaboration Sessions. It can be seen that “medication(s)” and “order(s)” were frequently used terms that often co-occurred. More specifically, the high frequency of “order(s)” led us to focus our deep dive on this topic, guided also by “hypotheses” derived from the connections between “order,” “options,” and “lists.” We examined the Workbooks with a focus on these topic areas, synthesizing qualitative data from different comments where appropriate. Select key insights from this analysis were distilled into Table 1.

The deep dive topics derived from examining the co-occurrence network often turned out to be frequent themes in the synthesized insights. For orders, the issue suggested by the network visualization that ordering options may be a pain point, was verified by detailed examination of the data. Exploring order-related issues in greater detail also revealed other pain points that were not apparent from the co-occurrence network, including a description of a workaround, even though the words “workaround” and “order” were not shown as associated in the network. Thus, the deep dives are guided but not dictated by the themes highlighted in the co-occurrence network. The findings of the deep dives led to specific recommendations, which are described briefly in the Discussion section below as an illustration of the types of recommendations that this methodology can produce.

Discussion

The CSWFA methods employed in the present study provided the agility to comprehensively identify user-described obstacles (i.e., “pain points”) that emerged during the implementation of the EHR in a primary care workflow context. Moreover, the techniques used in our study enabled the analysts to propose solutions (not described in detail in this paper), thus pointing to the method’s utility for improving the efficiency and accuracy of orders and the medication use process in primary care workflows. Specific insights and lessons learned included:

Personnel and Process

• CSWFA should be a multi-professional team approach including most or all the following: clinical staff, informaticists, architects (both process and information), requirements analysts, human factors engineers, and implementation experts who have proficiency with and exposure to the specific new EHR being deployed. This multidisciplinary team can cross-check the findings of the other disciplines and offer new perspectives.
  • The importance of having a team with diverse expertise was especially important due to the documentation and training needs prevalent with new EHR deployments.^1,3,4 The multi-professional nature of the team facilitated an understanding and analysis of the barriers to efficiency and workflow created by these documentation and training gaps.
  • In addition, program management office stakeholders and facility leadership who are consumers of the new EHR should be engaged at all stages of the process, to help define more accurately the high-level workflow definitions and mission-level considerations that drive the task being analyzed.
     
• Content and data fields that emerged from the Workbook Normalization and Review Sessions should be tailored to goals of CSWFA, while at the same using caution not to include extraneous fields (i.e. Devices Used, Handoffs, Hard Stops, etc.)  that may not be used or that will unnecessarily increase workload for a project team.
   
• Workbook data fields should remain modifiable. The team should remain flexible and open to adapting tools based on the site/service line. In addition, the toolkit’s usability must be considered. The Microsoft Excel implementation of the workbooks described in this paper offers the advantages of using a tool with which most users will be familiar. It should be noted that intermediate-level Excel functionality is used in our tools, and future efforts would benefit from developing a reference card or training guide to assist users, especially those facilitating meetings and updating workbooks in real time, with Excel filtering and formatting operations beyond the basic level.

Data Analysis

• Data analysis should be comprehensive. That is, stakeholders should leverage qualitative content analysis results to identify themes. In the context of our study, these included order-related issues around which integrated deep-dive assessment and improvement efforts were constructed, as opposed to a series of unrelated and isolated improvements.
   
• The CSWFA approach should not be primarily quantitative, instead it should provide a starting place and flexible guide for walking through a collection of narratives as the team seeks to interpret data from the Workflow Elaboration and Normalization/Review sessions. Co-occurrence network, topic model, and other natural language visualizations should be configurable and explored interactively by a multi-professional community. By moving between the parts and whole or greater context, this exercise can facilitate interpretation and inform thematic analysis.  This approach can be particularly useful when themes have not formally been predetermined or when the team is challenged by analyzing narratives to identify themes. Additionally, walking through a range of visualizations can help illuminate topics that may otherwise be overlooked. These collaborative activities can contribute to the hermeneutic process as priorities and timelines shift. To this end, narratives from the collection are highlighted in this paper for exemplar purposes. Their inclusion is not intended to provide quantitative information; rather, they are described here to characterize the approach to this work.
   
• Although it would not be the main focus of data analysis, the evaluation should incorporate a quantitative component. Several of the issues identified in Figure 2 and Table 1 related to system shortcomings that affected EHR user workload and the time it took to complete the tasks. Future efforts could be improved by deriving — through structured lines of questioning and ideally augmenting the questioning with time-motion studies — distributions of times that key activities take, and how different tasks and decisions in the model interact. This information can be subjected to computational modeling to identify and quantify measurable improvements of recommended solutions.
   
• Cognitive and decision modeling should be considered to improve the process. To more explicitly capture elements of the workflow that contributed to cognitive workload, decision making uncertainty, and potential for error, analysis of mental workload and decision processes and factors modeling should be conducted. The information derived from this analysis can also be incorporated into the aforementioned computational models.

Follow-On

• A plan should be provided to each site by the analysts to guide site personnel with integration and leveraging of findings generated by CSWFA. It will also be important for each site to communicate back to the analysis team how the results are used for training and implementation. This feedback will help the project team understand if Workflow Elaboration Sessions are eliciting the right level of data.
   
• Develop a training program for CSWFA project teams. The program should include informatics and workflow modeling training as well as the Microsoft Excel workbook training and documentation noted above to build capacity for CSWFA and to support larger-scale implementation rollouts for the EHR.

Much of the work on workflow analysis in healthcare IT has focused on improving the automation to develop and construct workflows¹⁴ or on study of the impacts of EHR or other changes on the general process^15,16.  However, there is a dearth of past work on process improvement in the context of a formally modeled business process coupled with formal thematic analysis of the content.  Some work has been conducted involving both modeling and analysis, conducting extremely useful high-level categorizations of challenging features of current-state workflow models¹⁷.  The present study incorporates the entire process of modeling a workflow, subjecting its content to more rigorous qualitative analysis, and deriving actionable conclusions to be leveraged for near-term process improvement.

Conclusion

This study provides the overall methodology and lessons learned for conducting a current state workflow assessment and qualitative analysis following the implementation of a new EHR. Although the examples included describe a post-go live CSWFA, our methodology is replicable to all phases of EHR implementation. In fact, our previous engagements conducting CSWFA include those medical facilities that are preparing for transition to a modernized system. These assessments assist teams in identifying inefficiencies, as well as identifying  areas where bottlenecks or productivity can be improved. CSWFA results and their application contribute to compliance and quality and may even inform job aids for system end users.

Limitations This study was conducted with healthcare personnel from one facility. Although the large integrated health system studied here has achieved an elevated level of national standardization, facility differences may be inevitable due to factors such as site organizational culture, facility size, and regional variations. Not only do these differences necessitate caution in generalizing the findings of one facility’s CSWFA assessment to other facilities, but they also raise the point that conclusions regarding the CSWFA methods themselves will be strengthened – or revised – following the conduct of these assessments in other locations. Furthermore, we acknowledge that not all organizations may have access to the interdisciplinary expertise described in the Methods section.

Impact. The issues identified through the CSWFA process are reviewed following the assessment and routed to the appropriate party to be addressed, whether this entails workflow reengineering, system bug fixes/enhancement requests, or training/documentation solutions. A number of issues identified in the present assessment have already been addressed and are beginning to result in process improvements that positively impact the site studied and other sites. As the large integrated health system that participated in the present work continues deployment of new EHR systems across the United States, lessons learned about workflow and tooling – after being vetted for applicability across sites as noted above – can be applied to improve the current state at other sites in preparation for an EHR rollout.

References

1. Baumann, Lisa A., et al. “The impact of electronic health record systems on clinical documentation times: A systematic review.” Health Policy, 122, (2018): 827-836.
2. Carayon, Pascale, et al. “Implementation of an electronic health records system in a small clinic: the viewpoint of clinic staff.” Behavior & Information Technology, 28 (2009): 5-20.
3. Helmers, Richard, et al. “Mayo clinic registry of operational tasks (ROOT): A paradigm shift in electronic health record implementation evaluation.” Mayo Clin Proc Inn Qual Out 3 (2019): 319-326.
4. Moy, Amanda, et al. “Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review.” Journal of the American Medical Informatics Association, 23, (2021): 998-1008.
5. HealthIT. “How do I Conduct Post Implementation Evaluation?” Washington DC May 22, 2022. Available online at https://www.healthit.gov/faq/how-do-i-conduct-post-implementation-evaluation.
6. Kennebeck, Stephanie S., et al. “Impact of electronic health record implementation on patient flow metrics in a pediatric emergency department.” Journal of the American Medical Informatics Association, 19 (2012): 443-447.
7. “BPMN Specification - Business Process Model and Notation.” BPMN Specification - Business Process Model and Notation. Accessed March 28, 2023. https://www.bpmn.org/.
8. Scheuerlein, Hubert, et al. “New methods for clinical pathways-Business Process Modeling Notation (BPMN) and Tangible Business Process Modeling (t.BPM).” Langenbeck's Archives of Surgery, 397, (2012): 755–761.
9. Pritchett, Amy. R. and Alexandra Coso Strong. “Integrating cognitive engineering into industry design teams.” Journal of Cognitive Engineering and Decision Making, 10(2) (2016): 134-137.
10. Rose, Stuart, et al. “Automatic keyword extraction from individual documents.” Text Mining: Applications and Theory 1(1-20), (2010): 10-1002.
11. Arnold, Timothy. “Bridging conversation islands to connect healthcare: introducing unique co-occurring word networks to find distinct themes.” Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care 11(1), (2022): 44-47.
12. Demšar, Janez, et al. “Orange: data mining toolbox in Python.” The Journal of Machine Learning Research, 14(1), (2013): 2349-2353.
13. Mihalcea, Rada, and Dragomir R. Radev. Graph-Based Natural Language Processing and Information Retrieval. Cambridge: Cambridge University Press, 2011.
14. Barrison, Philip et al. “Knowledge infrastructure: a priority to accelerate workflow automation in health care.” Journal of the American Medical Informatics Association, ocad026, (2023).
15. Gali, Hilaire, et al. "Impact of electronic health record implementation on clinical workflows for ophthalmology trainees." Investigative Ophthalmology & Visual Science, 60(9), (2019): 5504-5504.
16. Giauque, A. "Effects of EHR Implementation on Processes in Primary Care." In Crossing Borders–Digital Transformation and the US Health Care System, edited by E. Carson and M. Burns, 59-76. New York: Springer, 2019.
17. Pufahl, L., et al. “BPMN in healthcare: Challenges and best practices.” Information Systems, 107, (2022): 102013.

Figure 1 Guiding Questions for Workflow Elaboration Sessions

Figure 2. Co-Occurrence Network of Key Terms from Primary Care Workflow Analysis

 

Note: Network visualization created using Orange: Data Mining Toolbox in Python¹²

Comment: threshold = 3; window = 1; frequency threshold = 3 in corpus to network widget

 

Table 1. Select Key Insights from Orders Management

 

 

Author Biographies

Claire Hayes Watson, PhD, MPH serves as research & health services administrator, program manager, and research scientist for Visionary Consulting Partners, LLC, contractor to Human Factors Engineering, Clinical Informatics and Data Management, Office of Health Informatics, Veterans Health Administration, Washington, DC, chayes@vcp-llc.com

Anthony J. Masalonis, PhD serves as chief human factors analyst for Spectrum Software Technology, Inc., a subcontractor to Visionary Consulting Partners, LLC, contractor to Human Factors Engineering, Clinical Informatics and Data Management Office, Office of Health Informatics, Veterans Health Administration, Washington, DC, anthony.masalonis@sst-it.com

Tim Arnold, PharmD serves as an engineering psychologist for Human Factors Engineering, Clinical Informatics and Data Management Office, Office of Health Informatics, Veterans Health Administration, Washington, DC, USA; and Adjunct Clinical Associate Professor, University of Michigan, College of Pharmacy, Ann Arbor, MI, tim.arnold4@va.gov

Neale R. Chumbler, PhD serves as the executive associate dean for academic, faculty and student affairs & professor and chair of the Department of Diagnostic and Health Sciences College of Health Professions at the University of Tennessee Health Science Center, Memphis, TN, nchumbl1@uthsc.edu

William Plew serves as a systems engineer to Human Factors Engineering, Clinical Informatics and Data Management Office, Office of Health Informatics, Veterans Health Administration, Washington, DC, william.plew@va.gov