Paving the Way: A Cost and Outcome Evaluation Framework for the Transition from the 10th to the 11th Version of the International Classification of Disease

Editor's Note: This article was updated on February 7, 2023

Abstract

Objective: Countries worldwide, including Canada, need tools for informed decision-making on the adoption of ICD-11. The purpose of the current study is to create a cost and outcome estimation framework for the transition from ICD-10-CA to ICD-11 in Canada and anticipate the benefits/outcomes and international considerations of ICD-11 adoption.

Methods: This paper follows a cost and outcome evaluation framework. The costs of transitioning to the ICD-11 coding system are based on literature reviews and interviews adapted for the Canadian context, while the outcomes have been created using a novel methodology involving face and criterion validity.

Results: While it has been difficult and infeasible to include all possible categories and variables identified by the interviewees into the framework, this paper provides a comprehensive guideline of the outcome and cost estimation methodology.

Conclusion: The estimation technique adopted in this paper is unique and may be used as a benchmark methodology by other countries to evaluate the adoption of the ICD-11 coding system.

Keywords

International Classification of Diseases, ICD-11, ICD-10-CA, Cost and Outcome Estimation, Economic Evaluation, Framework, Qualitative Analysis, Database, Coded Data, World Health Organization

Introduction

Health data is collected from every inpatient visit to create coded administrative data. Approximately 5000 (Canadian Institute for Health Information 2019) CHIMA- Certified Health Information Management professionals across Canada use International Classification of Diseases and Related Health Problems (ICD) coding system. Through ICD, codes are assigned to medical diagnosis from patient charts, creating an internationally recognized registry to adhere with the World Health Organization (WHO) standards. Currently, Canada uses ICD-10-CA (10^th version, Canadian Revision) to code, store, and analyze healthcare data that have been widely used for research, provincial and national reports on population health, and system performance. However, due the presence of multiple country-specific modifications of ICD-10-CA, changes in medicine, and it being time for a routinized update, a new and improved version of ICD was required.

In June 2018, the WHO released ICD-11, after a decade-long revision process that involved extensive consultations with international experts. Improved usability may enable easier transition for coding specialists and ensure that the system is widely used.

As Canada and other nations prepare for ICD-11 implementation, there is now a critical need for extensive preparatory work. A seamless transition to ICD-11 will require large investments of time and money; for example, all coding specialists will need to be trained to use new systems, and IT software and supports will need to be upgraded. Starting in 2001, Canadian provinces required a 5-year period to fully transition from the ICD-9 to ICD-10-CA system (this included the transition from ICD-9 to ICD-10-CCI (Canadian Classification of Health Interventions)). This five-year-long transition period resulted in inconsistent national ICD data and data usability challenges (L. walker et al. 2012).

Canadian Institute for Health Information (CIHI) anticipates adopting ICD-11 in Canada; however, the decision for adoption and timing of implementation will be made by provincial/territorial governments in Canada. In order to ensure consistent national ICD data collection during the transition to an updated system, a simultaneous adoption of ICD-11 by all provinces is ideal to optimize data comparability. Currently, there is limited research on the costs and benefits of transitioning from ICD-10-CA to ICD-11 (Education and Implementation Committee 2018; WHO 2019). Evidence for decision-making is crucial to enable an informed and a seamless provincial transition to ICD-11.

The objective of the current study is to create the framework and formulae to enable evidence-informed decisions on whether to implement the International Classification of Disease 11^th Revision (ICD-11) of WHO in Canada. Specifically, we aim to 1) develop a framework for estimating the costs of ICD-10-CA to ICD-11 transition, and 2) to outline considerations to calculate potential benefits and outcomes of adopting ICD-11.

Assumptions

The framework is formulated based on the following two assumptions:

1. The focus of the retrospective study (Khair 2018) on ICD-9 to ICD-10-CA transition is based on acute sector costs/transition issues. We expect ICD-11 implementation to be broader.
2. The model assumes that hospital coding will follow the current (ICD-10-CA) coding procedures. It is difficult to foresee the effects and methods of computer-assisted coding (auto-coding) and the potential costs/savings which may result from it.

Methods

Scoping Review

We conducted a scoping review on studies containing cost and benefit analyses of ICD transitions following guiding methods by (Peters et al. 2015) to provide us with a foundation for our economic evaluation framework. We searched the database PubMed, MedLine, Scopus, and EconLit using the terms international classification of disease, ICD, economics, economic evaluation, cost-benefit estimation, analysis, evaluation, or projection, and combined these terms with big data, healthcare, medicine, health services, or hospital.

Since the number of current topics and knowledge regarding ICD transitions are limited, we decided to conduct a scoping literature review to identify methods and frameworks used in economic evaluations of healthcare technology transitions involving ICD. Another search was done on the Embase database by combining “ICD-10” and “ICD-11” with keywords related to economics. In total, 139 more potential articles were identified by the search strategy. The exact search strategy used keyword + textword searches with the “.mp” extension for the terms “ICD-10” and “ICD-11” using the OR Boolean operator. These results were then combined with a search for articles with “ICD-10” OR “ICD-11” in the title of the paper. Finally, these results were combined with the keywords “economic evaluation.mp” OR “cost-benefit.mp” using the AND Boolean operator. Out of these articles, 22 were found to be of potential relevance based on title and abstract screening by an independent reviewer. A common theme identified within search results was that of budget management. One study completed a cost-benefit analysis on reorganizing the diagnostic groups DSM-V and ICD-11. This strategy on the EMBASE database reveals a broad but sparse range of available literature on the economic impacts of ICD transitions in Canada.

Methods for Creating the Cost Calculation Formulae

Two reports (Libicki and Brahmakulam 2004; Calgary Health Region 2001) outline the necessary information to begin a cost projection for ICD-11. The cost estimation formulae are based primarily on the report published by the RAND Corporation (Libicki and Brahmakulam 2004), with adaptations to fit the Canadian health system. In-person interviews were conducted (between May 2017 and May 2019) in order to gain more insight into the process of ICD-10-CA implementation in the Calgary Health Region. These interviews inform the processes that will be involved in the ICD-11 implementation.

Methods for Creating the Outcome/ Benefit Calculation Formulae

Data from an ongoing ICD-11 field trial conducted by our team will be used for evaluating the outcomes of ICD-11. Data from 3000 inpatients discharged between January 1, 2015 and June 30, 2015 in three teaching hospitals in Calgary, Alberta have been abstracted. We have already validated the ICD-10-CA data and have successfully published the results of that validation study (Quan et al. 2008, 2005).

This unique dataset would be analysed to assess the outcomes of transitioning to ICD-11 using three distinct measures. Both data quantity and quality improvements will be examined.

Quantity of Codes: ICD-11 codes are more numerous in comparison with ICD-10-CA and reflect the ever-changing social values and medical content in healthcare. For example, ICD-11 includes new societally relevant codes (e.g. gaming disorder), risk factors codes, and causation codes. The enhanced coding system will enable increased surveillance of modern health issues. This is why the quantity of variables/codes in ICD-11 is used in our study as a measure of ICD-11 benefits.

1. Quantity of Codes per Record. Using our ICD-11 database, we will calculate the average number of codes per record in both ICD-10-CA and ICD-11, the number of secondary diagnoses coded, and the number of unspecified codes used.
2. Quantity of New Feature Codes. We will also use the number of new feature codes (which are introduced in ICD-11) in recorded ICD-11 data to measure the quantity improvement.

Quality of Codes: We propose to measure quality improvement using Validation of Commonly Used Codes. We have already developed methods to define the Charlson and Elixhauser (Charlson ME, Pompei P, Ales KL 1987; Anne Elixhauser, Claudia Steiner 1930) conditions in the ICD-10-CA data, both common conditions coded in the Canadian healthcare system. These conditions have been used by health services researchers to measure population level burden of disease (health status) or case-mix adjustment with administrative data to assess hospital level differences in care. We will calculate parameters including sensitivity, specificity, positive predictive value, and negative predictive value.

Interviews

Interviews were conducted to gain information on the process of ICD-10-CA implementation and to obtain feedback for the ICD-11 transition cost and outcome framework. Consent was obtained from each participant, and at least two research team members attended the interviews. We conducted in-person or Skype, recorded, semi-structured interviews with individuals or focus groups with Health Information Department Directors across various provinces and territories in Canada. In addition, we interviewed a wide range of stakeholders from the WHO, Canadian Institute of Health Information (CIHI), representatives of the National Coding Advisory Committee (NCAC, decision makers from provinces/territories), Canadian Health Information Management Association (CHIMA), Health Information Management (HIM) coders, provincial managers and directors, as well as software vendors across all provinces and territories in Canada.

Using a qualitative descriptive approach (Sandelowski 2010), we interviewed purposively sampled (e.g., based on role, sex, age and geographical location) stakeholders who will be affected by the transition from ICD-10-CA to ICD-11 — where multiple stakeholders in one province exist, they became a focus group.  Using focus group methodology (Krueger and Casey 2014), we conducted focus groups or individual interviews until no new data were generated. In total, we spoke with 25 stakeholders with representation from nine provinces and three territories in Canada (Musselwhite et al. 2007). Representation from multiple provinces ensured that knowledge translation strategies could be tailored to the unique concerns of each province.

For the sampling strategy, we aimed to interview one representative from each Canadian province and territory. With the help of CIHI, we contacted members of NCAC (National Coding Advisory Committee). The NCAC advises CIHI on ICD code development, coding rules, and guidelines (Canadian Institute for Health Information 2003). Some members who agreed to participate in the interview invited colleagues who had experience with the ICD-9 to ICD-10-CA transition to join the interview. When we exhausted the NCAC contacts, we approached CHIMA to provide contacts for HIM managers and directors for representation from as many provinces as possible.  Informed consent was obtained from all participants. An honorarium of $100 was offered to each participant for their time; some participants declined.

The interview questions were adapted to reflect the challenges specific to Canada. Participants were asked to share past experiences of transitioning from ICD-9 to ICD-10-CA (including facilitators and barriers), as well as suggest possible costs and benefits for the transition from ICD-10-CA to ICD-11. Questions involved identifying the code users in the province as well as understanding whether ICD codes were used in billing. We also asked each province to elaborate on what they think would differentiate their province from other Canadian provinces, which should be taken into account while projecting the total cost of implementing ICD-11. Table 1 provides more examples of the unique considerations for each province, which would affect the transition costs differently.

Duration for focus groups was 45-60 minutes, while individual telephone interviews took slightly less time. Data collection was iterative: information from early focus groups provided areas to probe in subsequent focus groups/interviews. A professional transcriptionist transcribed the discussions verbatim.

Qualitative Analysis

Focus group and interview transcripts were analyzed using conventional content analysis by the study team (Hsieh and Shannon 2005; Vaismoradi, Turunen, and Bondas 2013). The team members: 1) read the transcripts as they became available to acquire an overall sense of the data; 2) highlighted words or phrases that captured key concepts and patterns across participants; 3) combined codes into overarching categories; 4) reviewed categories in relation to one another; 5) synthesized final categories and content into the framework; 6) highlighted exemplary quotes. Discussion took place when need for consensus regarding categories had to be reached.

Preliminary Framework

Based on our scoping review and interviews, we created a preliminary framework using elements from two of the reports (Libicki and Brahmakulam 2004; Calgary Health Region 2001) identified in the scoping review. Three team members (SK, ML and TL) selected relevant elements from the two frameworks, and excluded calculation methods which were not applicable to the Canadian healthcare system.

Ethics Approval

Ethics was obtained through the Conjoint Health Research Ethics Board at the University of Calgary, Ethics ID #REB17-2387.

Results

Scoping Review

Our scoping review identified three reports that evaluated the costs and benefits of the ICD-9 to ICD-10-CA transition. The first is the RAND technical report (Libicki and Brahmakulam 2004), which is the only study that provided a comprehensive methodology for cost-outcome estimation of ICD-9 to ICD-10-CA transition in the United States. The second is a Canadian cost projection business case (Calgary Health Region 2001) for transitioning from ICD-9 to ICD-10-CA. The third study we identified is a transition guide (Committee 2018) describing the stages, benefits and costs of moving from ICD-10-CA to ICD-11 coding system. This paper was prepared by the WHO-FIC Education and Implementation Committee (EIC) and focuses on the Mortality and Morbidity Statistics updates in ICD-11. Since the transition guide does not include a framework to calculate the costs and outcomes of the transition to the ICD-11 coding system, we concentrate on the RAND report and the ICD Business Case to create the primary skeleton of our cost and benefit calculation framework. Our scoping review shows the lack of literature and data needed for economic analysis for ICD-10-CA to ICD-11 transition.

 The variables included in the cost estimation in the RAND report (Libicki and Brahmakulam 2004) were costs of training, productivity losses and system change costs. The business case (Calgary Health Region 2001) offered an example of ICD-10-CA adoption in a Canadian context and was incorporated into our framework. The provincial and federal government provides healthcare funding in Canada. The cost categories in this framework reflect costs incurred by the hospitals, which are covered under the single-payer provincial healthcare budget. This is significantly different from the framework outlined in the RAND report, which incorporated the multi-payer health spending and insurance costs. While the benefits were not enumerated in the business case, the study estimated expected incurred costs. The calculations were based on answers to questionnaires addressing the expected training and project management costs, productivity losses and software application costs.

Interview Results

The results of the interviews ensured the framework fit the Canadian context. Furthermore, we were able to obtain differences in cost estimates for different provinces. The interviews provided robust information on the process of ICD-10-CA implementation and garnered feedback for the ICD-11 transition cost and outcome estimation framework.

Differences in estimation variables were due to the effects of a myriad of factors, including the organizational milieu, information and technological infrastructure, and whether the training sessions are in remote or urban locations. As the interviews progressed, feedback was obtained on the cost estimation framework, as well as factors within the framework categories, such as steps needed for ICD-11 implementation. Provinces were hesitant to comment on benefits of ICD-11 as very little information had been disseminated and made available prior to launching ICD-11. We refined our knowledge translation activities based on the interviews. A snapshot of the main issues raised by the interviewees is provided in Tables 1 and 2.

Table 1: Provincial Considerations for ICD-11 Implementation, Unique by Province

British Columbia mentioned that auto-coding projects would need to be updated/linked. Alberta is moving into Connect Care, which is a health information system allowing health providers to have access to a common system with comprehensive and consolidated patient information.  If ICD-11 is implemented, it may be around the same time period of the launching of Connect Care. This significant system change will have important implication on the costs and benefits calculation of ICD-11 for the province. New Brunswick would require re-training in both English and French. There might also be both online training and in-person training, so travel costs should be considered. They are also considering training physicians to use ICD in the future, so they can enter codes (but this has not happened yet). Manitoba would require higher costs for system changes. Further, Manitoba has a dual abstract system which would affect productivity loss. Provincial governments provide funding support to upgrade IT systems. Northern Territories require training centres in remote locations and would likely have to train staff together in a group (Nunavut and Yellowknife coders were trained together during the ICD-9 to ICD-10-CA transition). There may not be road access in some locations and coders may have to be flown out to the nearest training center. All of these factors contribute to additional cost variables. Nova Scotia finds maintaining connection between coders during training is a challenge, because most coders work from home. The province is also very spread out, having small training sites in rural areas. Lastly, Nova Scotia anticipates using computer-assisted coding around the same time as the ICD-11 adoption. Ontario anticipates challenges in transitioning to ICD-11, because coded clinical data are extensively used for hospital payment. There are many different code users (e.g. Cancer Care Ontario and QBPs). Both of these examples have their own system that would need to be updated. In addition, they use an international system and the full benefits may not be realized until the United States adopt ICD-11 as well. Ontario’s Hospital-Based Allocation Model also dictates the allocation of funds. The range in the variation of ancillary funding may be quite large. Ontario is also challenged to meet mandated quarterly deadlines in addition to the CIHI yearly deadline of May 31st annually, which may cause additional costs due to productivity losses.

Table 2: Summary of Interview Feedback on Framework, Benefits and Knowledge Translation Activities

Cost of Retraining   Code users including coding specialists, coding supervisors and managers, and coding directors would need specific training tailored to their roles Other code users to be trained include Federal entities (Vital Statistics Registry), provincial agencies (HQO), QBPs, eHealth (conducts quality checks), analysts, quality and safety teams, physicians in some cases, and data warehouse maintenance staff that use the data and would need to learn ICD-11 coding Costs of traveling to training sites is higher in rural and remote centres Online training and retraining costs should be considered Educational institutions will require materials and training Productivity Losess Cost Time will be needed to integrate and maintain the new ICD-11 system (e.g. fixing broken code pathways) A break-in period of 6 months to 2 years will be necessary before high-quality data is produced from ICD-11 Adoption of a new system will cause backlog of charts to code, and “there is only so much overtime available” System Change Cost IT systems need to be updated Data collection and analytics are integrated and will be complex to update Licensing fees and implementation costs (e.g. purchasing new tools through the vendor) should be considered

Table 2 illustrates the variables identified through interviews for our framework for estimating costs of the ICD-11 transition. We have incorporated the majority of the considerations brought forward by the interviewees; however, some considerations are not feasible to calculate or measure due to insufficient data (e.g. costs associated with data conversion from ICD-10-CA to ICD-11 using crosswalk tables).

Canadian provinces still use ICD-9 codes for billing and the interviewees identified that unless the single-payer system moved to using the ICD-11 codes, re-training costs and system costs would not be affected. This was a very important insight in calculating the costs of transition as transferring the billing systems to use the new ICD-11 codes would require huge amounts of financial and human resources.

Interviewees also mentioned that if software does not require change-over, it will cost less and take less time to transition. Saskatchewan mentioned using ICD-9 for physician claims and will consider using ICD-11 once it is adopted. When interviewing the Northern Territories on how the ICD-10-CA to ICD-11 transition would affect their territory, the most specific concern was on the geographical location and the need for trainees to be able to access online materials or travel to the training location.

Foreseeable Benefits of ICD-11

Interviewees deduced potential benefits based on their experience with the ICD-9 to ICD-10-CA transition. Possible benefits mentioned by participants included relevant and up-to-date coding and enhanced digital adaptability (e.g. computer assisted coding) which may enable remote coding due to electronic and online browsing systems. The interviewees also mentioned the significance of simultaneous transition across provinces, which will lead to standardization of records and allow better comparability of data. They alluded to the benefits of having the ICD-11 language better matched to clinical terminology. Some participants expressed that pilot projects and experience using the data is needed to fully understand the benefits of ICD-11.

Recommendations for Smooth ICD-11 Transition

Based on the past transition from ICD-9 to ICD-10-CA in Canada, several participants recommended that a preparatory two-year time frame be built-in prior to ICD-11 implementation. Participants emphasized that this two-year period would allow education systems and hospitals to prepare instructional material and make necessary IT system changes. They mentioned that preparation materials should be offered across all provinces prior to the training of ICD-11 users. In addition, interviewees suggested that the transition to ICD-11 would be made smoother if all key stakeholders were involved and consulted throughout the duration of the adoption process. Key stakeholders included vendors, education providers, health system providers (e.g. AHS), and health ministries, all of which should be involved in consistent multi-disciplinary meetings across the country. Finally, some participants felt that the adoption of ICD-11 would cause impetus to streamline data and integrate sectors such as mental health facilities, long-term care, and primary care.

Preliminary Framework

From the RAND report and the ICD business case example, we have established a preliminary cost estimation framework (Figure 1). Costs of ICD transition were classified into three broad categories: cost of re-training, productivity losses, and cost of system changes (Figure 1).

Cost Estimation Calculations

Cost of re-training (See Table 3) has been divided into equivalent full-time coders, analysts, managers and trainers. Cost is calculated based on the number of people to be re-trained, the time it requires to retrain them, and hourly wage in each category. It also includes accommodation expenses, travel costs, classroom and material expenses. The people retrained are those who will record hospital data using ICD-11 codes as seen below.

Table 3: Retraining Costs

Variable	Formula
Coder retraining costs	Coder retraining hours per coder * Number of coders * Hourly wage
Analyst retraining costs	Analyst retraining hours per analyst * Number of analysts * Hourly wage
Manager retraining costs	Manager retraining hours per manager * Number of Managers * Hourly wage
Trainer costs	Number of trainers * Days spent for training* Daily wage of trainers
Educational Institution Training	Number of institutions* Trainer costs
Other staff costs	Number of other staffs * hours worked * hourly wage of staff
Travel and accommodation costs	Cost of Travel + Cost of Accommodation
Other Costs	Classroom Cost + Lunch Costs + Material Costs

*Denotes multiplication

These calculations can be done at any time before the implementation of ICD-11 to depict true costs, adjusted with appropriate discounting to reflect the economic value of time. Discounting is further explained in subsequent sections of this paper. The total cost of re-training is calculated by adding all the components outlined in the above formula.

The productivity of coders (See Table 4) is expected to be affected during the transition phase. Implementation of ICD-11 may coincide with a new fiscal year, which might require coders to work for a longer time to complete a backlog of data during the break-in period. However, gradually, this time difference will decline as coders become more efficient and familiar with the ICD-11 system.

Table 4: Productivity Losses

Variable	Formula
Coder Productivity Loss	Number of coders  * ((Previous average coding time per chart – New average coding time per chart)/Previous average coding time) * 100%) * hourly wage * hours worked

*Denotes multiplication

The productivity loss formula considers the decreased output of coders ,which is expected as coders learn the new ICD coding system. Productivity loss is calculated by the decrease in productivity as measured by the additional time it takes coders to complete a chart using ICD-11 codes and is multiplied by the percentage of decreased output and hourly coder wage to obtain a monetary value. This monetary value may vary by region even if coders are performing the exact same tasks due to differences in hourly wage.

The short-term (~6 months) and long-term (~1 to 3 years) productivity break-in periods will allow for accurate calculation of productivity losses. Break-in period refers to the transition period after ICD-11 is implemented, including intensive training and learning period for all users. For each province, we will use different break-in periods based on our interviews with stakeholders. This is because different provinces required different amounts of time to implement the last ICD-10-CA transition, ranging from 6 months to 3 years.

Other coders and technology support units will require extra time to code during the transition phase, gradually becoming used to the new system. Orientation and training of new hires would also cost the province time. Moreover, time is required to convert data which takes away time from other projects. Reinforcement training after implementation of the ICD-11 system is an added time requirement for code users, which in turn would result in lost work time and thus productivity loss.

For this study, information technology (IT) system change costs (Please refer to Table 5) are expected to include estimations of time to work, cost of designing and adapting/updating software, data conversion, data warehouse, data dictionary, decision support systems, licensing fees, and cost of adopting the new codes, collectively called IT impact costs. In order to accumulate the cost of software application changes during the previous ICD-9 to ICD-10-CA transition for the ICD business case report by Calgary Health Region, questionnaires were given to vendors to bid for contracts. IT system change costs can be estimated based on information collected through a survey of software vendors.

Table 5: System Change Costs

Variable	Formula
IT Impact Costs	(IT support hours needed * hourly wage of IT personnel) + other administrative costs
Vendor Costs	Depends on vendor bidding for estimated service fee

*Denotes multiplication

Additional Cost Considerations

In addition, the cost estimation may involve data quality checks, hours of technical support needed to fix broken pathways (validation and correction) which will affect productivity, quality assurance reports, coding quality support teams, and orientation of new hires. Other secondary users such as Surgical Information Management System, HIS, ADT, Grouper Software users and institutions offering online courses will be affected by the system change and incur costs in terms of both software changes and retraining of workers. The cost estimations will vary in different provinces due to the prevalence of different needs and regulations.

Preliminary Outcome Evaluation

Calculations presented in the RAND report included "more accurate payments for new procedures," "fewer miscoded, rejected, and improper reimbursement claims," "better understanding of the value of new procedures," and "improved disease management" (Libicki and Brahmakulam 2004). However, the information for the listed variables is difficult to collect for the ICD-11 transition in Canada, as many of the necessary variables and scenarios are not applicable in the Canadian single-payer health system. As a result, the only possible benefit calculation for the ICD-11 transition is to assess whether there is an increase in data quantity and quality.

In order to examine the outcomes of moving from ICD-10-CA to ICD-11 in Canada, we propose evaluating the quantity and quality of coded data generated from ICD-11, compared to ICD-10-CA. The following construct outcome indicators can be used to estimate potential benefits.

Percentage of increase in data quantity (Y%): Comparing the number of variables in the original ICD-10-CA coded administrative data vs. recoded ICD-11 data, we will define percentage of increase in data quantity as:

1. Percentage of increase in data quantity (Y₁ %) = (((number of data elements in recorded ICD-11 data – number of data elements in the original ICD-10-CA coded administrative data))/number of variables in the original ICD-10-CA coded administrative data) x 100%.
2. New Feature Codes (Y_2i ) = # of new feature codes in ICD-11 data when i=1, 2, 3, the new feature codes being measured are timing of conditions, causation, extension codes (for improved specificity), respectively.

Percentage of increase in data quality (Z%): Comparing the sensitivity, specificity, positive predictive value (PPV), and Negative Predictive Value (NPV), we define percentage of increase in data quality as:

1. Z_i % = (validity measure of ICD-11 data – validity measure of ICD-10-CA coded administrative data)/ validity measure of ICD-10-CA coded administrative data

when i=1, 2, 3, 4, the validity measure is sensitivity, specificity, PPV, and NPV, respectively, calculated as compared to a chart review (i.e. reference standard).

Using the cost projection as well as the three types of ICD-11 outcome measures constructed, we will be equipped to provide a basic economic evaluation of ICD-11 transition in Canada. The results will be delivered in the following specific format: “if X amount of dollars is spent on implementing ICD-11 in Canada, it will result in a Y1 percent increase in quantity of codes per record, Y2 increase in quantity of new feature codes, as well as Z percent increase in data quality.”

Overcoming Uncertainty in Cost and Outcome Evaluation Estimation

Since proxy measures will be used to estimate the economic impact of ICD-11 implementation in Canada, there exists a degree of uncertainty surrounding the cost of implementation. One method of reducing uncertainties is by acknowledging all possible areas where costs may be incurred (e.g. converting all types of coders to full-time equivalents). Stakeholder interviews will be conducted across different provinces and territories in Canada to collect data and form reasonable assumptions surrounding cost parameters. Moreover, sensitivity analysis will be conducted for the variables, which are likely to change in value (e.g., discounting rate). Sensitivity analysis adjusts the value of interest and reflects the impact that changes in that variable would have on overall costs and outcomes. Due to possible inaccuracies with proxy measures, sensitivity analysis with appropriate percentages (e.g. ±5%, ±10%, ±15) may provide a clearer range of the true economic impact. Sensitivity analysis may also be performed with changes in the discount rate with appropriate percentages.

Discussion

This research is novel, with little literature to refer to in developing our framework. Our methods were created based on the few reports we found, and the expertise of our team. The healthcare industry currently has a collection of raw data due to extensive chart recording practices throughout Canada. As data becomes more readily available, it would be easier to employ advanced statistical analysis and methods to conduct economic evaluative studies. This paper provides a comprehensive guideline for benefit and cost estimation methods in the Canadian context. For countries where coded clinical information is used for payment purposes as well as clinical practices, the implementation of ICD-11 may lead to benefits in the form of resource savings.

The transition to ICD-11 will occur in a unique context, and our framework cannot capture foreseeable futuristic considerations. Although ICD-11 offers an electronic interface, other significant changes in technology are likely to be introduced, including computer-assisted coding. Computer-assisted coding may change the roles of coding specialists from coding health information, to reviewing codes generated by automated systems. Currently, computer-assisted coding has yet to be implemented and therefore costs or benefits of ICD-11 in this context cannot be estimated. Further, using alternative terminology systems such as SNOMED-CT may contribute to unforeseeable costs associated with the transition to ICD-11. Our framework uses current information at hand to make estimates, with a caveat being the large electronic changes in the next decade that are difficult to anticipate.

 International Considerations

There are three internationally significant features of ICD-11 with benefit and cost implications:  a) ICD-11 classification system represents important scientific and medical advances; b) it allows for easier integration with electronic health applications that may support fewer mistakes, more detail, and accessibility for lower resource countries; and c) international collaborators incorporated changes for enhanced accuracy of coding in multiple countries and care settings (WHO 2018). In general, the number of countries using ICD is expected to increase – a definite benefit for each country to have disease data where none may have previously been captured. As a result, more countries could be included in international health data comparisons for better disease surveillance.  As in our proposed method to estimate benefit, the number of conditions coded per patient can potentially increase (WHO, 2018). With definitions for health conditions embedded in the browser, consistency of code choice is potentially enhanced, regardless of country using ICD-11, potentially improving the criterion validity.  As well, international consultation during the development phase meant that cultural differences were accounted for in a greater degree than in past classifications.

Of these features, accessibility to a complete electronic platform has the greatest economic implication in that any country with internet and computerized data capture, can readily adopt and use ICD-11.  As in the proposed cost estimation framework, re-training, productivity loses, and system changes, depend on the current personnel and infrastructure for capturing coded data.  Countries differ greatly in coders (e.g. physicians, nurses, training health information specialists) and complexity of infrastructure (e.g. USA), and from few conditions captured (1-6 diagnosis coding fields in the following countries: Guatemala, Uruguay, India, Indonesia, Mauritius and United Republic of Tanzania) to up 25 conditions captured as in Canada (Otero Varela et al. 2019). WHO working groups are working to optimize adoption by multiple countries with translations into multiple languages (e.g. Spanish, French, and Russian) already underway, and training materials under development.

Strengths

Our framework provides a comprehensive structure to conduct cost and outcome evaluation of ICD-11 transition. It provides a novel and practical approach with mixed methods to estimate the costs and benefits/outcomes of ICD-11 transition. With stakeholders’ views incorporated while constructing the framework, it reflects multi-disciplinary perspectives. With ICD-11 being released in June 2018, this framework is timely for guiding research to provide empirical evidence urgently needed by policy makers in Canada to make decisions and plan resource allocations accordingly. It is also internationally applicable; the development of a framework for estimating the economic impact of ICD-11 implementation would be of value beyond Canada.

Challenges and Recommendations

Due to the significantly different health system in Canada, which is a single-payer system compared to the multi-payer system in the United States, it has been difficult to construct benefit scenarios like those outlined in the RAND report. Interviews and discussions with experts have shown that the ICD health codes are not used for disease management or referred to by physicians to identify possible solutions for the patients. However, from our interviews, a strong recommendation was that the codes could be used to inform on the progress of patients and may even facilitate early discharge depending on an analysis of the charts by the physicians. Another limitation of this study is that the framework is limited to inpatient coding transition. Future research will need to address associated costs of transitioning to ICD-11 in the outpatient realm. Further, we were unable to reach software vendors to interview regarding IT system changes during the anticipated ICD-11 transition. To gain insight into these possible costs, we were able to obtain a general understanding of IT system costs from the participants we involved in the present study.

Considerations for Knowledge Translation

Decision-making on implementation of ICD-11 will be complex and will require consideration of multiple factors. Essential questions must be answered, including:

1. who will be affected by the ICD-11 transition;
2. what are the costs of the transition;
3. what are the benefits; and
4. what resources and support will be needed?

Early preparation for the transition to ICD-11 is essential to provide ample time for software vendor and IT integration and updating training processes. The past ICD-10-CA transition required a very long time to be adopted. The lack of preparation created productivity losses, which further contributed to an increase in costs. The framework considers this productivity loss as a cost category and employs a formula to include the different times it takes to code the same chart using ICD-11 and ICD-10-CA codes.

Further, information on ICD transition has been limited. Data collection following ICD-11 implementation will provide comprehensive information on future costs and benefits of ICD-11 (e.g., number of codes used, validity of coding, productivity losses), ultimately improving capacity for future research and system performance evaluation. Finally, collaboration with key stakeholders (i.e. CIHI, CHIMA, and WHO) in the implementation of ICD-11 is recommended to inform all provinces of adoption considerations.

Conclusion

Economic evaluation is an important way of knowing whether a project is feasible. The paper serves to inform the decision-making process for stakeholders in Canada and other countries to understand whether or not the adoption of ICD-11 would be beneficial for their respective countries.  Our framework is currently the only available tool to identify costs and outcomes of ICD-11 and can be used globally for similar cost estimation projections. Although a rigorous cost benefit evaluation on ICD-11 transition remains infeasible at this stage, the results of our overarching project offer concise and valuable information to decision-makers on ICD-11 implementation and adoption. The framework also enables organizations within each country to plan a budget and operate in a timely manner.  Future research will employ the cost and outcome/benefit framework for ICD-11 transition estimations.

Support

The study is funded under the Testing the World Health Organization’s Beta Version of ICD-11 Grant by The Canadian Institutes of Health Research.

Acknowledgements

We would like to thank CIHI and CHIMA for their valuable contributions in providing us with professional contacts for our interviews.

Notes

Calgary Health Region. 2001. “ICD-10 Business Case.”

Canadian Institute for Health Information. 2003. “Data Quality Is Top Priority.” CIHI Directions ICIS 10 (1): 1–6. http://secure.cihi.ca/cihiweb/dispPage.jsp?cw_page=news_dir_v10n1_dataquality_e.

Canadian Institute for Health Information. 2017. “Canada’s Health Care Providers: Provincial Profiles, 2008 to 2017 — Data Tables.” CIHI. https://www.cihi.ca/en/canadas-health-care-providers-provincial-profiles-2008-to-2017-data-tables.

Charlson, M E, P Pompei, K L Ales, and CR MacKenzie. 1987. “A New Method of Classifying Prognostic Comorbidity in Longitudinal Studies : Development and Validation.” Journal of Chronic Disease 40 (5): 373–83. http://www.sciencedirect.com/science/article/pii/0021968187901718.

Education and Implementation Committee. 2018. “Transition Guide: International Classification of Diseases for Mortality and Morbidity Statistics (ICD-11 MMS).”

Elixhauser, Anne, Claudia Steiner, D Robert Harris, and Rosanna M Coffey. 1998. “Comorbidity Measures for Use with Administrative Data.” Medical Care 36 (1): 8–27. http://www.jstor.org/stable/3766985.

Hsieh, Hsiu-Fang, and Sarah E Shannon. 2005. “Three Approaches to Qualitative Content Analysis.” Qualitative Health Research 15 (9): 1277–88. https://doi.org/10.1177/1049732305276687.

Khair, Shahreen. 2018. “The Economics of Big Data in Healthcare: A Case Study of the ICD-9 to ICD-10 Transition,” 121. https://doi.org/http://dx.doi.org/10.11575/PRISM/33222.

Krueger, Richard A, and Mary Anne Casey. 2014. Focus Groups: A Practical Guide for Applied Research. Sage publications.

L. Walker, Robin, Deirdre A. Hennessy, Helen Johansen, Christie Sambell, Lisa Lix, and Hude Quan. 2012. “Implementation of ICD 10 in Canada: How Has It Impacted Coded Hospital Discharge Data.” BMC Health Service Research 12 (1): 149. https://doi.org/10.1186/s1472-6963-0-123.

Libicki, Martin, and Irene Brahmakulam. 2004. The Costs and Benefits of Moving to the ICD-10 Code Sets. RAND: Science and Technology.

Musselwhite, Kimberly, Laura Cuff, Lisa McGregor, and Kathryn M. King. 2007. “The Telephone Interview Is an Effective Method of Data Collection in Clinical Nursing Research: A Discussion Paper.” International Journal of Nursing Studies 44 (6): 1064–70. https://doi.org/10.1016/j.ijnurstu.2006.05.014.

Otero Varela, Lucia, Chelsea Doktorchik, Natalie Wiebe, Cathy Eastwood, and Hude Quan. 2019. “ICD-Data Features and Coding Training Worldwide: An International Survey.”

Peters, Micah D.J., Christina M. Godfrey, Hanan Khalil, Patricia McInerney, Deborah Parker, and Cassia Baldini Soares. 2015. “Guidance for Conducting Systematic Scoping Reviews.” International Journal of Evidence-Based Healthcare 13 (3): 141–46. https://doi.org/10.1097/XEB.0000000000000050.

Quan, Hude, Bing Li, L. Duncan Saunders, Gerry A. Parsons, Carolyn I. Nilsson, Arif Alibhai, and William A. Ghali. 2008. “Assessing Validity of ICD-9-CM and ICD-10 Administrative Data in Recording Clinical Conditions in a Unique Dually Coded Database.” Health Services Research 43 (4): 1424–41. https://doi.org/10.1111/j.1475-6773.2007.00822.x.

Quan, Hude, Vijaya Sundararajan, Patricia Halfon, Andrew Fong, Bernard Burnand, Jean-Christophe Luthi, L Duncan Saunders, Cynthia A Beck, Thomas E Feasby, and William A Ghali. 2005. “Coding Algorithms for Defining Comorbidities in ICD-9-CM and ICD-10 Administrative Data.” Medical Care 43 (11): 1130–39.

Sandelowski, Margarete. 2010. “What’s in a Name? Qualitative Description Revisited.” Research in Nursing & Health 33 (1): 77–84. https://doi.org/doi:10.1002/nur.20362.

Vaismoradi, Mojtaba, Hannele Turunen, and Terese Bondas. 2013. “Content Analysis and Thematic Analysis: Implications for Conducting a Qualitative Descriptive Study.” Nursing & Health Sciences 15 (3): 398–405. https://doi.org/doi:10.1111/nhs.12048.

World Health Organization. 2019. “ICD-11 Implementation or Transition Guide” 1–39.

Author Biographies

Shahreen Khair, Research Associate, Department of Community Health Sciences, Cumming School of Medicine, University of Calgary

Hude Quan, Professor and Director for the Centre for Health Informatics, Department of Community Health Sciences, Cumming School of Medicine, University of Calgary

Cathy A. Eastwood, Adjunct Assistant Professor, Department of Community Health Sciences, Cumming School of Medicine, University of Calgary

Chelsea Doktorchik, Research Associate, Department of Community Health Sciences, Cumming School of Medicine, University of Calgary

Jiawei Zhao, Bachelor of Health Sciences, University of Calgary

Tatiana Losev, Program Assistant, College of Discovery Creativity & Innovation, Taylor Institute for Teaching and Learning, University of Calgary

Keith Denny, Director, Clinical Data Standards and Quality, Canadian Institute for Health Information

Sharon Baker, Manager, Classifications and Terminologies-Development

Mingshan Lu, Professor, Department of Economics, University of Calgary