by Mehrnaz Mashoufi, MSc; Haleh Ayatollahi, PhD; and Davoud Khorasani-Zavareh, PhD
Introduction: Emergency care is usually conducted within limited time and with limited resources. During emergency care processes, data quality issues should be taken into account. The aim of this study was to assess the quality of emergency care data from the perspectives of different data stakeholders.
Method: This survey study was conducted in 2017. In this research, the viewpoints of three groups of data stakeholders, including data producers, data collectors, and data consumers, were collected regarding data quality in emergency care services. Data were collected by using a standard information quality assessment questionnaire.
Results: The mean values for each dimension of data quality were as follows: sound data (6.23), dependable data (6.28), useful data (6.30), and usable data (6.35), with 0 being the lowest possible score and 10 being the highest. The role gap analysis suggested a clear gap between data producers and data customers at the university level.
Conclusion: Overall, data quality in emergency medical services was not at a high level. Although data quality was improving, the levels of data completeness, compatibility, and usability were low. To improve the usability of emergency medical service data, more attention should be paid to the dimensions of accuracy, completeness, and consistency of data sources.
Keywords: subjective evaluation; data accuracy; emergency medical services
In a health system, high-quality data provide valid evidence of patient health status and can help policy makers plan for better healthcare services.1 However, many challenges are associated with data quality in healthcare that might be related to the tools and methods for data collection, data processing, and data usage, which are constantly changing.2 Therefore, information problems related to patient care are often encountered during daily activities. These problems can have a negative impact on the delivery of care and may lead to misunderstandings about patient information.3 In fact, if information deficiencies are not properly identified and managed, serious consequences such as medical errors may occur.4–10 Thus, it is important to identify and manage information deficiencies and their impact on the workflow of hospitals.
The functioning of emergency medical services is considered a key benchmark in the field of public health services, and whether an emergency medical service is fulfilling its functional requirements, such as increasing data quality, is significant for measuring the performance of a health system.11 A number of studies showed that the practice of physicians in emergency medical services is associated with numerous interruptions and often includes a variety of tasks and simultaneous communications.12–15 As a result, information deficiencies occur frequently during prehospital and hospital emergency care.16
From the perspective of emergency medicine, interruption in work processes increases the potential for errors in the information production cycle, such as data collection, storage, and utilization. Therefore, regular assessment of data quality seems to be necessary in order to avoid poor data quality.17 The evaluation of data quality is conducted both objectively and subjectively.18 In the objective method, quantitative indicators that are derived from measuring the dimensions of data quality are calculated, while in the subjective method, which is usually done with the use of a questionnaire, the perspectives of data stakeholders (data producers, data collectors, and data consumers) regarding data quality are examined. The results of both types of evaluation will help to identify deficiencies in data quality.19, 20 However, the importance of subjective evaluation and review of the end users’ or data customers’ views should not be underestimated.21
Because information has a fundamental role in the provision of continuous care and the performance of emergency medical services,22–24 it is crucial to assess the quality of information and the dimensions of data quality. Therefore, the aim of this study was to assess the quality of emergency data from the perspectives of different data stakeholders in various organizations.
This survey study was conducted in Iran in 2017. To assess the quality of data, we examined the views of three groups of data stakeholders: namely, data producers, data collectors, and data consumers. All potential participants (n = 234) from four organizational levels were invited to take part in the study. The first level included deputies and experts from the emergency care department and the disaster management center at the Ministry of Health, the second level consisted of heads and experts of the emergency care department and the disaster management center affiliated with medical universities, the third level included nurses and emergency medicine specialists who worked in the emergency departments of three teaching hospitals, and the fourth level included prehospital emergency medical staff.
Before the research was conducted, ethical approval was obtained from the Institutional Review Board. Data were collected with the use of a standard information quality assessment questionnaire.25 Because of the large number of questions, some of which were repeated questions with negative verbs, and also because of the time constraints of the emergency department staff, some of the questions were removed, and a questionnaire containing 66 questions was distributed among the participants. The validity of the questionnaire was confirmed based on the opinions of five experts in the field of health information management. The reliability of the questionnaire was checked by measuring the internal correlation coefficient, Cronbach’s alpha, in a pilot study (α = 0.96).
The questionnaire consisted of eight demographic questions and six sections. The first part dealt with data types and stakeholders. The second part was about data quality, and the third part concerned the characteristics of the organization and the context assessment. The fourth, fifth, and sixth parts of the questionnaire were about data collection, data storage, and information use in emergency services, respectively. Responses were given on an 11-point Likert scale. Zero was equivalent to “completely disagree,” while ten was equivalent to “completely agree.” Data were collected and entered into SPSS software and were analyzed with descriptive statistics and the product and service performance for information quality (PSP/IQ) model. This model has been used in the methodology for information quality assessment.26 Table 1 presents the conceptual framework of this model.
A total of 193 participants (82 percent of the 234 eligible individuals) completed the questionnaire, and of these, 188 questionnaires were analyzed. Five questionnaires were excluded because a large number of questions had not been answered. In terms of age, the highest frequency (27.5 percent, n = 47) were 28 to 33 years of age. The mean age of men was 36.6 ± 9.1 years, and the mean age of women was 33.3 ± 7.1 years. The majority of participants were nurses (62.2 percent, n = 117), and most of the participants had a bachelor’s degree (71.3 percent, n = 134). Other demographic information of the participants is shown in Table 2.
The findings indicated that clinical data were used with the highest frequency (68.1 percent, n = 128) and other data (such as the time of activities and missions) were used with the least frequency (6.2 percent, n = 5) in the field of emergency medical services.
The first part of the questionnaire was related to the quality of emergency data (mean score, 6.03 ± 1.68). In this section, the highest frequency of participants (60.3 percent, n = 111) believed that the data measurement unit was obvious, and 105 participants (56.7 percent) believed that authorized staff have access to the needed information. However, according to the results, data were not consistent across different sources (34.6 percent, n = 64). The second part of the questionnaire was about the activities that organizations performed regarding the quality of data (mean score, 5.90 ± 2.7). In this section, the highest frequency of participants assumed that their organization was trying to identify deficiencies (53.3 percent, n = 99). The third part of the questionnaire was about data collection (mean score, 6.2 ± 5.25), and the highest frequency (53.1 percent, n = 97) was related to those participants who thought they knew the sources of the data. The fourth part of the questionnaire was about data storage (mean score, 6.2 ± 3.33), and the majority of participants (65 percent, n = 121) noted that they were aware of the process of data storage on the computer. The fifth part was about the use of data (mean score, 6.2 ± 5.10), and the majority of participants maintained that they knew who uses the data (62.7 percent, n = 116); see Table 3.
In the current study, the level of data quality was determined for each organization (Table 4 and Table 5). As Table 4 shows, the quality of data in prehospital emergency care had the highest mean value (7.24 ± 1.43), and the quality of data in the Ministry of Health had the lowest mean value (5.30 ± 1.34). The analysis of variance showed that there was a significant statistical difference regarding the level of data quality among different organizations (p = .002).
Table 5 shows the mean values for information quality in the PSP/IQ model. According to Table 5, the highest mean value was related to data relevancy (6.74), while the lowest mean value was related to data consistency (5.45).
This analysis is a useful diagnostic technique to determine the difference in perceptions of data quality between roles as a data quality benchmark, to compare the roles of the individuals to identify data quality problems, and to determine a method for data quality development.27, 28
As Figure 1 shows, the fourth organization (prehospital emergency care) had the best performance compared to the other three organizations. The size of the gap between the data producers and data consumers in this organization and in the first and the third organizations (the Ministry of Health and the emergency department of the hospitals) was very small. The small gap implies that organizations are in the position of developing data quality and that the quality of data was acceptable for data consumers. However, there was a huge gap between data producers and data consumers at the second organization (medical universities), which indicated that the data producers and data consumers at these organizations disagreed on the level of data quality. In fact, data producers were not aware of the problems of data consumers. Although the gap was small in the Ministry of Health and the emergency department of the hospitals, these organizations had to take actions to improve data quality. The gap direction was positive for all four organizations. This finding showed that these organizations should focus on the needs and the problems experienced by data consumers.29–31
The benchmark gap analysis compares the status of different organizations with the best-performing organization.32, 33 In this study, the quality of emergency care data was compared in four different organizations. Figure 2 shows cumulative percentage frequencies for the participants’ responses in each organization. For example, 90 percent of respondents in the prehospital emergency care believed that the quality of their data was good. As Figure 2 shows, there was no significant gap between the organization with the best performance (prehospital emergency care) and the other three organizations. From the participants’ perspectives, the three other organizations were on the pathway to develop data quality.
As Table 6 shows, data quality dimensions such as consistency (5.45 ± 2.6), completeness (5.69 ± 2.41), and appropriate amount (5.92 ± 2.33) had the lowest mean values, while relevancy (6.74 ± 2.21) had the highest mean value. Moreover, the analysis of variance showed a significant statistical difference between data quality dimensions at different organizational levels, and the Tukey post hoc test (see Table 7) suggested that there was a significant statistical difference between the Ministry of Health and the prehospital emergency care.
Data quality assessment is crucial for organizational decision making and adjustment of organizational strategies and performance. During the evaluation process, the root causes of data deficiencies, errors, and procedures required for information quality control will be identified and ultimately can lead to the development of data quality.34 Although a number of dimensions, characteristics, and criteria have been identified as important for measuring data quality,35–38 data quality should be regarded as a multidimensional concept,39, 40 and it is better to use both quantitative (measuring quality indicators) and qualitative (exploring stakeholders’ views) methods to evaluate it.41 The purpose of this study was to evaluate the quality of data by evaluating different dimensions from the perspective of stakeholders at four different organizational levels.
In the present study, the views of three groups of data stakeholders, including data producers, data collectors, and data consumers, regarding the quality of emergency care data were investigated. However, because data producers and data collectors have similar roles, both were considered data producers; thus, data producers and data consumers were the two groups that were compared as data stakeholders.42
The PSP/IQ model was used to analyze the stakeholders’ perspectives.43 According to the PSP/IQ model, data quality dimensions are divided into four main dimensions: sound, dependable, useful, and usable. Two main dimensions, namely, sound and dependable, represent data properties that can be independent of context. The other two dimensions, useful and usable, represent characteristics of the data that are relevant to meeting the needs and expectations of data customers. In fact, data should be useful and accessible for use.44 In addition, procedures must provide a complete information product for use.45 In other words, the dimensions of sound and useful represent the quality of data as an information product, and dimensions of dependable and usable represent the quality of data as an information service.46, 47
The findings of the present study showed that in the organizations studied, except the Ministry of Health, useful and relevant emergency care information was provided. However, there were some doubts about whether accurate and error-free information is provided to consumers and whether information is easily accessible and usable.
However, as noted, the situation was slightly different in the Ministry of Health. Although useful, complete, and error-free data were provided to the consumers, data were less usable, because of low accessibility. In addition, because the mean value of timeliness was low, the quality of the data could be affected.
The findings of the study also showed that the quality of data in the emergency medical services was not at a high level. The quality of data was higher in prehospital emergency care and was lower in the Ministry of Health compared to the rest of organizations, and there was a statistically significant difference between these two organizations. Because operations in prehospital emergency care are heavily dependent on information, access to the correct, useful, and usable information is necessary in this field. The results also showed that the prehospital emergency care has been more successful than other organizations at providing high-quality information in terms of fitness for use.
The various levels of data quality in different organizations can be related to data features, such as accuracy, completeness, and timeliness at the operational level. Because prehospital emergency care data are mostly operational and the data produced in the Ministry of Health are predominantly strategic data, the reason for high data quality in prehospital emergency care and low data quality in the Ministry of Health can be justified. As a result, senior executive officers should pay more attention to data quality in the field of emergency medicine, and they should have data quality assurance plans among the organizational priorities.48
The findings indicated that in emergency medicine, the level of data completeness and compatibility was low, and as a result, this situation affected the soundness of the data. In fact, the lack of high-quality information could be due to the gap between the real-world requirements and the expectations of information quality, which may lead to misinterpretation and a low level of information accuracy.49 The accuracy of data is very important in healthcare because these data are a basis for decision making and clinical measures, and any inaccuracy can cause serious harm and even death to the patient.50 Besides, the accuracy of data implies data validation.51, 52 However, if correct data lack the other criteria for quality, such as timeliness and ease of understanding, the data would not be suitable for use, because these criteria affect the usefulness of the data.53
According to the results, useful data were generated in the field of emergency medicine, and the highest mean value was related to relevancy. These data also generate added value for organizational tasks and data customers.54 Therefore, it seems that relevant data can meet the needs of customers in this area and are useful. Although usable data should be easily accessible and understandable to the users,55–57 the findings indicated that the lowest mean value was related to the usability of data and data accessibility. Accessibility is a prerequisite for data quality, and if data are inaccessible, all other aspects of data quality will be overshadowed.58 This dimension has been identified as critical in several studies.59–61 Similarly, in a study by Kahn et al., no statistically significant difference was found between the mean values for the main dimensions of data quality in three organizations, but the researchers noted that the data had low accuracy and usability.62
Regarding availability and accessibility of information in the emergency department, Remen and Grimsmo found that patient medical and drug history data were more available and accessible than other data in the records. However, most of the data needed to be searched, and important information was available in only 10 percent of cases.63 Similarly, other studies suggest that access to information systems is important for emergency department staff, but these systems only satisfy a small part of their information needs. These studies suggest that developing access to information is beneficial for emergency care services, though the confidentiality of information can also be threatened.64–66
Completeness denotes the presence of all necessary data and conceptually implies having all the necessary and appropriate parts. Besides, it indicates the overall adequacy of the data content to meet organizational needs.67 In the present study, the completeness of the data was not acceptable, and users’ requirements have not been clearly identified in this area. The results of the studies conducted in the emergency department show that information gaps can be seen in different situations, such as transferring the patient from nursing homes to the emergency department. In the present study, the mean value for the completeness of data in the emergency department and prehospital emergency care was lower than other mean values.68, 69 Similarly, in Murphy and Reddy’s study,70 the findings revealed that false, obsolete, incomplete, incompatible, and finally missing data were among the problems that the emergency department staff faced. Therefore, the results of the current study are in line with Murphy and Reddy’s findings. In another study, conducted by Xie and Helfert,71 the findings showed that most of the information defects in the emergency department were related to eight dimensions: timeliness, accuracy, conciseness, relevancy, completeness, accessibility, understandability, security and privacy. The results of the current study are in agreement with the findings of Xie and Helfert.
To improve data quality, the challenges and causes of poor data quality should be identified.72 The ability to measure the quality of data stored in databases is an important measure in this regard. The findings showed that the mean value for the features of data quality development was low. This result implies that most organizations were not still able to assess data quality in their organizations. The difficulty of accurately measuring data quality is another issue that exacerbates the problem. Moreover, the importance of using a data quality assurance team in organizations, the development of data quality, and efforts to improve applicability of data sources in all departments are among the data quality-related efforts73, 74 that apparently have not yet reached maturity in the organizations under study. In this regard, another study highlighted the importance of stakeholder engagement in data quality assessment.75
On the basis of the results, it can be concluded that despite the importance of the quality of emergency care data in caring for patients and for decision making, data quality was not at a high level in the organizations under study. The research implications can be considered in two aspects. First, more attention should be paid to the quality of emergency care data either by conducting data quality assessments periodically or by monitoring the quality of data regularly. To achieve this objective, data quality professionals can be appointed to assess the quality of data by using subjective and objective methods. In fact, data quality professionals know how to calculate different dimensions of data quality and how to improve them. The other aspect is related to the data stakeholders and their information needs. Proper planning is required to identify and meet the needs of data stakeholders. Again, data quality professionals and researchers can help to solve these issues.
In this study, only the three largest emergency departments were selected because of time and resource constraints, and their emergency physicians and nurses were asked to participate in the study. Although the number of emergency departments was limited, the researchers believe that the participants were representative of a larger population. The survey questionnaire can be also used in other emergency departments to assess the quality of data in other settings. Moreover, only a quantitative approach was used in this study. To investigate the reasons for low data quality in emergency medical services, conducting qualitative research in different organizations is suggested.
Assessing the quality of data is an important step in developing data quality measures. In this research, the stakeholders’ viewpoints on the quality of emergency care data were investigated. The results showed that the inherent characteristics of emergency care data, such as completeness and consistency, had a lower mean value than other characteristics. Useful data were generated in this field, but data usability was not acceptable. Although data were useful, because of inaccessibility the data were not easily used. Because users have a fundamental role in determining the quality of information, their participation in the data quality assessment process is essential. We recommend that the Ministry of Health, as a stakeholder that is the head of emergency medical services, pay more attention to the quality of data and data quality assessment in the field of emergency medicine. In addition, the development of data quality measures should be planned and implemented at various levels.
This study was funded and supported by Iran University of Medical Sciences, Tehran, Iran (IUMS/SHMIS_93/10).
Mehrnaz Mashoufi, MSc, is a PhD student in Health Information Management at Iran University of Medical Sciences in Tehran, Iran.
Haleh Ayatollahi, PhD, is an Associate Professor in Medical Informatics at Iran University of Medical Sciences in Tehran, Iran.
Davoud Khorasani-Zavareh, PhD, is an Associate Professor in Health Disaster Management at Shahid Beheshti University of Medical Sciences in Tehran Iran.