Objective: The expansion of mobile applications as a tool for road traffic health and safety may develop several issues from the perspective of information management. Quality assessment of these apps, especially from an information system management perspective, appears inevitable, as their possible low quality may cause irreversible injury or fatal consequences. This study aimed to evaluate the quality of the apps in the three subcategories of road traffic safety apps (including Accident Record and Report (ARR), Distraction Management (DM), and Vehicle Operating, Fixing, and Maintenance (VOFM)) using the Mobile Application Rating Scale (MARS), which rates 23 evaluation criteria organized in five domains (Engagement, Esthetics, Information, and Subjective Quality) with particular attention to the five rights framework of health information system.
Method: The researchers retrieved road traffic health and safety mobile apps from Google Play. First, the domain expert panel (n= 7) (from disciplines of HIM and medical informatics) was formed. They scrutinized and discussed the MARS items and mapped them into the five rights framework of information quality. Moreover, the researchers assigned the apps to the information system or decision support system category. Two researchers independently reviewed the apps and conducted the qualitative content analysis to categorize them into ARR, DM, and VOFM classes. Finally, the quality of the apps was assessed using the MARS rating scale (max=5) in terms of 1) app classification category with a descriptive aim; 2) app subjective and objective quality categories comprised of engagement, functionality, esthetics, and information sections; and 3) an optional app-specific section. The mean scores for the subjective quality, objective quality, and app-specific sections were calculated separately for each mobile app. A score ≥ 3.0 was considered acceptable.
Results: A total number of 42 apps met the criteria for the assessment. The average objective quality scores were computed as 2.6, 2.2, and 3.0 for the ARR, DM, and VOFM apps, respectively. Therefore, the quality of the apps in the ARR and DM subgroups was not acceptable. Moreover, the quality of the apps in the VOFM subcategory was considered moderate. Furthermore, the subjective quality and app-specific sections of apps in the ARR and DM categories were less than moderate. Most apps had the potential of an information system or decision support system. Also, the criteria measured by MARS could be mapped to the five rights framework of information management.
Conclusion: The findings of this study revealed the existing gaps in three subcategories of road traffic safety apps. Considering the multiple criteria of the MARS and having in mind the framework of five rights, developers of the apps may develop better products in road traffic health and safety.
Keywords: five rights framework, mobile apps, traffic safety, information system, decision support, MARS, digital health, traffic accident
Road traffic accidents and injuries are one of the global health challenges. The number of deaths due to road traffic has constantly increased from 1.15 million in 2000 to 1.35 million in 2018.1 Therefore, road accidents are not only a road safety problem but also a public health issue.2,3 E-health is one of the new approaches to support public health , and m-health has a great potential to expand e-health to the public due to the ubiquity of mobile devices and mobile technologies toward improving public health.4 Mobile health covers medical and public health procedures supported by mobile devices, including mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices. Mobile health and wellness apps are expected to improve safety, health, and quality of life through behavioral feedback and targeted information.5 The mobile application can be considered a specific subdomain of an information system and, in some cases, it can act as a decision support system.6,7 Similar to other information systems, developing a health and traffic safety mobile app creates a system with input, processing, output, outcome, and impact. Data collection is the starting point of the information management process in mobile apps. Also, the quality of health information is an essential criterion of health information management: the right information should be available to the right person, in the right format, at the right time, and at the right place through the right channel to support health management decisions.8 The five rights framework reflects the importance of tailoring the information provision to the user’s needs.
Considering the public health aspect of road traffic issues, three main elements that can play a role in accidents are road infrastructure, vehicles, and drivers. Among these, the driver has the largest share of accidents. Therefore, if the right information (e.g., driving behavior, drowsiness alerts, hazardous areas, early warning of weather and road conditions, and vehicle condition) is given to the drivers at the right time at the right place through the right channel and in the right format, they can make better decisions and respond quickly to serious driving conditions.9 Nonetheless, roads and vehicle traffic are an essential part of people's daily lives. Therefore, monitoring their situation has received much attention. For this purpose, intelligent transportation systems (ITS) have been built. Many of these involve the installation of dedicated sensors in vehicles (e.g., GPS-based tracking units) or on the road (e.g., inductive loop vehicle detectors and traffic cameras.), which can be an expensive proposition and usually limited to the busiest road routes.10 From another point of view, the widespread penetration of mobile phones has dramatically improved communication in the community. Cellphones are like small multipurpose computers that, in terms of CPU power and RAM size, are similar to laptops found a few years ago. In addition, more and more users own these smart terminals, and their main use is gradually shifting to functions such as web browsing, social networking, multimedia streaming, online games, and other applications. In this platform, new information and communication services can be introduced using smartphones in various fields.11 Therefore, mobile applications could be used as public health information or decision support systems.
Although mobile phones seem to distract drivers and cause road accidents,12-15 there are features in these devices that make them an opportunity to prevent accidents, such as lane detection, vehicle detection, vehicle distance estimation16 and also drowsiness management, distraction management, and speed limit warning. In addition, with the prevalent use of smartphones, many companies have recently developed unique apps to improve public service quality, people security, and safety.17 On the other hand, similar work has been proposed and implemented in connection with the development of mobile apps. Their primary focus areas have been traffic management, routing and navigation, driving behavior analysis, vehicle and road safety, and emergency services.18,19 All these functionalities are possible in the context of information systems or decision support systems manifested in the mobile application form. One of the essential issues about these apps is their quality. Their evaluation is inevitable because their poor performance may cause irreversible injury or fatal results. Traditional systems used to test app quality, such as user star ratings (app ratings on a scale of 1 to 5 stars) and reviews, can provide fake or subjective reviews and give users the wrong signals.20 Moreover, app descriptions in the Google Play Store are often incomplete or inaccurate and are not a valid tool for assessing the quality of an app, especially when dealing with sensitive issues such as road traffic safety.21
Because of the need to ensure better app quality for users, a Mobile App Rating Scale (MARS) has been presented by a multidisciplinary team of experts. “The MARS is a simple, objective, and reliable tool for classifying and assessing the quality of mobile health apps. It can also be used to provide a checklist for designing and developing new high-quality health apps.”22 MARS is a 23-scale tool that provides an in-depth evaluation of app quality by testing and grading the app in several areas, including user engagement, functionality, aesthetics, information, and subjective quality. Each item is scored using a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent).23
In the previous study, the researchers grouped the road traffic apps into two categories using Haddon’s factors and public health approaches: Road Traffic Training (RTT) and Road Traffic Health & Safety (RTHS) apps. Each of these groups has some subcategories. The RTHS category has 11 subcategories, including (accident record and report; alcohol-free driving; distraction management; driving/driver behavior feed backing; drowsiness management; eco-driving and fuel saving; real-time traffic information/alerting; ridesharing service; safe driver service; speed camera and police detector; and speed limit warning). Furthermore, the RTT category has three subcategories: driving performance; traffic rules and road signs; and vehicle operating, fixing, and maintenance.24 The researchers intend to use the features of three subgroups of apps (Accident Record and Report (ARR), Distraction Management (DM), and Vehicle Operating, Fixing, and Maintenance (VOFM)) to make a multipurpose mobile app for use in Iran as a part of the Ph.D. thesis. This study evaluated the quality of three subgroups of road traffic apps using the MARS with a focus on the five rights framework of information management.
The researchers used the MARS rating scale, a reliable tool for assessing the quality of m-health apps for assessing the quality of subgroups of apps (ARR, DM, and VOFM). One of the key guidelines used in the development of MARS is Healthcare Information and Management Systems Society (HIMSS).25 The HIMSS guidelines for evaluating the usability of m-health apps use a Likert scale of “strongly agree” to “strongly disagree” to rate each criterion. HIMSS criteria have usability measures for rating efficiency, effectiveness, user satisfaction, and platform optimization, but no measures of rating information quality have been included. The MARS rating scale is comprised of two sections. The first is the classification section, which collects descriptive and technical information about the app. This section has six items of descriptive and technical information for each app: 1) descriptive information (name, number, and type of ratings for all versions; developer; version; cost; platform; description; update); 2) focus; 3) theoretical background and strategies; 4) affiliations; 5) age group; and (6) technical aspects (social sharing, web access, app community, login, password protection, and reminder functions). The second section is the app quality category, divided into objective and subjective quality. The rating scale assesses app quality on four dimensions. All items are rated on a 5-point scale from “1. Inadequate” to “5. Excellent.” Objective quality has four sections (engagement, functionality, esthetics, and information) with 19 items, while subjective quality consists of four items, for a total of 23 items. In addition to these two categories, there is an optional app-specific section with six items (awareness, knowledge, attitudes, intention to change, help-seeking, and behavior change). These added items can be adjusted and used to assess the perceived impact of the app on the user's knowledge, attitudes, and intentions to change as well as the likelihood of actual change in the target health behavior.26
Two authors independently reviewed each of the apps in the three subgroups: 1) ARR (three apps); 2) DM (25 apps); and 3) VOFM (24 apps). Also, the researchers installed apps when possible to better assess the apps. Before scoring each app, the reviewers used each app for at least one week to understand the app’s functionality. They also learned how to complete the MARS. Data were analyzed using descriptive and analytical statistics. The mean score of each MARS section was calculated. The mean score of the four objective quality sections (engagement, functionality, esthetics, information) was calculated separately from that of the subjective and app-specific sections to strengthen the impartiality of the measure. Kendall’s coefficient concordance was used to calculate the interrater agreement between two raters. The analysis was performed with SPSS Statistics version 21. Moreover, the researchers assigned the apps to the category of decision support system or other information systems.
Expert panel (n= 7) (disciplines of HIM and medical informatics) was formed. Then the panel scrutinized and discussed the MARS items and mapped them into the five rights framework of information management. The five rights is well-known in HIM. It has also been referred as the Five Rights of Clinical Decision Support (CDS) and has been used widely in public health and healthcare.27
The MARS Items Mapped into the Five Rights of Information Management
Expert panel mapped the MARS items into the five rights of information management. Table 1 shows the detail.
Evaluation Analysis of the Apps as a Type of Information System by the MARS
The Kendall coefficient for the agreement was 0.96 (p = 0.06), which indicates a good agreement between the two evaluators. Disagreements over each of the concessions were discussed and resolved by consensus.
As shown in Table 2, the highest mean score in the accident record and report (ARR) subgroup for engagement (4.2), functionality (4.5), esthetics (4.0), information (2.6), and subjective quality (4.3) was related to AYS Accident Report and SafeDrive. The overall mean MARS objective quality score, which allows the evaluation of the general app quality (maximum of 5 points), was 2.6 points (SD 1.2); thus, the quality of the three included apps in the ARR subcategory was not considered acceptable (< 3.0). The score of the subjective quality section was 3.2 points (SD 1.6), and that of the app-specific section was 2.2 points (SD 1.3). When the scores of the six MARS sections (four objective, one subjective, and one app-specific) were compared, the score of the information section (mean 1.6, SD 0.9) was lowest than the others. In this subcategory, the highest Google Play Store user rating score was 4.1 points related to the SafeDrive app.
As it is evident from Table 2, all the apps categorized in the ARR subgroup act as the information system as well as the decision support system.
The highest mean score in the distraction management (DM) subcategory for engagement (4.4), functionality (4.8), esthetics (4.3), information (2.7) and subjective quality (3.8) was related to Car Mode, MessageLOUD and TextDrive. The lowest total mean score of the DM subgroup was related to the information section (1.4). In addition, the highest total mean score of the DM subgroup was related to the functionality section (3.2). The total mean of the MARS objective quality score for the DM subcategory was 2.2 points (SD 0.7); therefore, the quality of the 22 included apps in the DM subcategory was not considered acceptable (< 3.0). The score of the subjective quality section was 2.7 points (SD 1.1), and that of the app-specific section was 2.4 points (SD 1.2). In this subcategory, the highest Google Play Store user rating score was 4.6 points related to the DriveCare app. More detailed information is presented in Table 3. All apps in the subcategory of distraction management can be considered information systems; however, they can indirectly act as a decision support system in some sense.
The highest mean score in the VOFM subcategory for engagement (4.6), functionality (4.8), esthetics (4.7), information (3.4), and subjective quality (4.8) belonged to BMW Driver's Guide, MINI Driver's Guide, NISSAN Driver's Guide and Mercedes-Benz Guides. The highest total mean score of the VOFM subgroup was related to the functionality section (3.7). In addition, the lowest total mean score of the VOFM subgroup was related to the information section (2.2). The overall mean MARS objective quality score for the VOFM subcategory was 3.0 points (SD 0.9); thus, the quality of the 17 included apps in this subcategory was considered moderate. The score of the subjective quality and app-specific sections was the same (3.4 points, SD 1.0). In this subcategory, the highest Google Play Store user rating score was 4.6 points related to the Car Scanner ELM OBD2 app. As it is evident from Table 4, all three apps categorized in the VOFM subgroup act as the information system as well as the decision support system. See Table 4 for more detailed information.
Figure 1 compares the mean scores of each MARS section in the road traffic safety apps of the three subgroups. The VOFM subgroup in all of the MARS sections has the highest score.
The present study evaluates the objective quality (engagement, functionality, esthetics, and information) and the subjective quality of the available apps for the three subcategories of road traffic apps in the Google Play Store. Considering the apps’ functionalities, the researchers assigned the apps to the information system or decision support system category. As such, it was possible to map the different aspects of the MARS tool into the five rights of information management.
Quality evaluation using the MARS tool showed that out of 42 apps in three subgroups (accident record and report; distraction management; and vehicle operating, fixing, and maintenance), the VOFM subgroup with 17 apps are qualitatively acceptable, concerning the MARS mean ratings of ≥3 out of 5 points.
Comparing the six sections of MARS (four sections of objective quality, subjective quality section, and the app-specific section) in the ARR and VOFM subgroups, the most significant results were related to the engagement, functionality, and esthetics of the apps because they were visually pleasing and descriptive enough. In contrast, the information section of these apps needs to be improved. A review of the DM subgroup apps with the MARS score showed that except for the mediocre performance section, all sections had low mean scores and should be reviewed by developers because these apps claim safety for users and their inefficiency and inadequate information may cause hazards to users while driving. While a lack of up-to-date and scientific information about apps in road traffic health and safety could be a barrier to proper and reliable guidance for users, simple, high-performance aesthetics, including visual appeal, could encourage the use of mobile apps.28,29 Also, the lowest mean score for the engagement was in the DM subgroup, evaluated based on the entertainment, interest, customization, interaction, and attractiveness of the target group and the esthetic scale, evaluated in terms of layout, graphics, and visual appeal. This finding is consistent with previous studies using the MARS to evaluate the quality of mobile apps for asthma management, where the esthetic score is lower, indicating that this factor is less considered in health and safety design.30
In addition, it is essential to evaluate the effectiveness of apps to help users become familiar with road traffic health and safety issues. Therefore, it is substantial to conduct studies examining road traffic apps' quality, efficiency, and reliability. Moreover, it is notable that traffic safety experts should evaluate such apps to provide better information to assist users in making safety-related choices.31
The subjective quality and app-specific sections in the ARR and DM subgroups were less than moderate, so they need to be improved by the app developers. This finding is in line with the results of previous studies using the MARS for quality assessment of mobile apps for food allergies.32 The subjective quality and specific parts sections noted the general conception of users of the app, which, if positive, leads them to recommend and use it. Hence, more engagement can be needed to improve users’ understanding. Therefore, it is vital to expand users’ subjective quality view and impact of the app-specific, mainly affecting users’ perception of the app.33
As the results show, in most of the apps surveyed in the three subgroups, the user star rating score is above average. When we compare these results with the total MARS score, especially in the two subgroups DM and VOFM, we see that the user star rating is higher than the total score of MARS. Similar to the previous studies, there is a clear difference between a quality assessment obtained by a researcher using a more objective tool such as the MARS and a real-world user who tends to rate the quality of the app by star rating in a very subjective way.34 Therefore, the MARS quality evaluation is a more objective tool to provide more accurate app quality information and recommendations for developing future apps.
By definition, the right information provided to the end user must be evidence-based, derived from a set of recognized guidelines, or based on a standard of practice. If too much information is given to the end user, it may create too much cognitive load and cause him to ignore the warning.35 One of the criteria of app objective quality of the MARS is the quality of the information provided in the app, which states that the developed app must contain quality and documented information.
Engagement and esthetics criteria of app objective quality are related to graphic design, overall visual appeal, color scheme, attractiveness, customizable, and interactive to the audience. These criteria correspond to the right intervention format of the five rights that it states. Decision support is implemented in various formats—such as alerts, order sets, protocols, monitoring systems, and information buttons. Therefore, it is essential to choose the best format to solve the problem.36
The right time in the workflow is related to the functionality of the MARS app objective quality, which involves the app’s functioning, ease of learning, navigation, flow logic, and gestural design of the app. Also, subjective app quality and app-specific criteria are related to the right person, which caused users’ knowledge, attitudes, and intentions to change and the likelihood of actual change in health behavior.
The moderate quality of MARS was identified for the VOFM subcategory from three subcategories of road traffic apps, although the objective and subjective quality of the reviewed apps should be improved, and the existing apps should be tested experimentally. Through mapping the MARS items into the five rights framework, it can be concluded that the five rights of information management are yet to be realized in the mobile apps targeting road traffic health and safety. The domain app developers can use these results to develop new reliable apps in the field of road traffic health and safety toward promoting public health.
1. Chang, F.-R., et al. Global Road Traffic Injury Statistics: Challenges, Mechanisms and Solutions. Chinese Journal of traumatology, 2020. 23(4):216-218.
2. Binder, S. and J.W. Runge. Road Safety and Public Health: A US Perspective and the Global Challenge. Injury Prevention, 2004. 10(2): p. 68.
3. Martinez, R. Traffic Safety as A Health Issue. in Transportation, Traffic Safety and Health. 1997. Berlin, Heidelberg: Springer Berlin Heidelberg.
4. Grady, A., et al. Improving the Public Health Impact of eHealth and mHealth Interventions. Australian and New Zealand Journal of Public Health, 2018. 42(2): p. 118-119.
5. Pagliari, C., et al. What is eHealth (4): A Scoping Exercise to Map the Field. Journal of Medical Internet Research, 2005. 7(1): p. e391.
6. Nickerson, R.C., U. Varshney, and J. Muntermann. Taxonomy Development in Information Systems: Developing A Taxonomy of Mobile Applications. European Journal of Information Systems, 2013. 22(3): p. 336-359.
7. Watson, H.A., R.M. Tribe, and A.H. Shennan. The Role of Medical Smartphone Apps in Clinical Decision-Support: A Literature Review Artificial Intelligence in Medicine, 2019. 100(101707).
8. Abdelhak, M. and M.A. Hanken. Health Information: Management of A Strategic Resource. Fifth edition. ed. 2016, St. Louis, Missouri: Elsevier.
9. Bassoo, V., et al. A Framework for Safer Driving in Mauritius. Future Computing and Informatics Journal, 2017. 2(2): p. 125-132.
10. Mohan, P., V.N. Padmanabhan, and R. Ramjee. Nericell. Rich Monitoring of Road and Traffic Conditions Using Mobile Smartphones. in Proceedings of the 6th ACM Conference on Embedded Network Sensor Systems. 2008. ACM.
11. Zaldivar, J., et al. Providing Accident Detection in Vehicular Networks through OBD-II Devices and Android-based Smartphones. in 2011 IEEE 36th Conference on Local Computer Networks. 2011. IEEE.
12. Zatezalo, N., M. Erdogan, and R.S. Green, Road Traffic Injuries and Fatalities among Drivers Distracted by Mobile Devices. Journal of Emergency, Trauma, and Shock, 2018. 11(3): p. 175-182.
13. Schroeder, P., et al. National Survey on Distracted Driving Attitudes and Behaviors-2015. 2018, United States. National Highway Traffic Safety Administration.
14. Schroeder, P., M. Meyers, and L. Kostyniuk. National Survey on Distracted Driving Attitudes and Behaviors–2012. (Report No. DOT HS 811 729). Washington, DC: National Highway Traffic Safety Administration. 2013.
15. Berenbaum, E., et al., Y TXT N DRIVE? Predictors of Texting While Driving Among A Sample of Ontario Youth and Young Adults. Accident Analysis and Prevention, 2019. 122: p. 301-307.
16. Tang, S.J.W., et al. Real-time Lane Detection and Rear-end Collision Warning System on a Mobile Computing Platform. in Computer Software and Applications Conference (COMPSAC), 2015 IEEE 39th Annual. 2015. IEEE.
17. Cardamone, A.S., et al. Willingness to Use Mobile Application for Smartphone for Improving Road Safety. International Journal of Injury Control and Safety Promotion, 2016. 23(2): p. 155-169.
18. Mohan, P., V.N. Padmanabhan, and R. Ramjee. Nericell. Rich monitoring of Road and Traffic conditions using Mobile Smartphones. in Proceedings of the 6th ACM conference on Embedded network sensor systems. 2008. ACM.
19. Forni, A. and R. van der Meulen, Gartner Says Worldwide Sales of Smartphones Grew 9 Percent in First Quarter of 2017, Gartner Inc. (2017).
20. Kuehnhausen, M. and V.S. Frost. Trusting smartphone apps? To install or not to install, that is the question. in 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA). 2013. IEEE.
21. World Health Organization. mHealth: New Horizons for Health through Mobile Technologies: Based on the Findings of the Second Global Survey on eHealth (Global Observatory for eHealth Series, Volume 3). 2011.Geneva.
22. Stoyanov, S.R., et al. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR mHealth and uHealth, 2015. 3(1): p. e27.
24. Aghayari, H., et al. Mobile Applications for Road Traffic Health and Safety in the Mirror of the Haddon’s Matrix. BMC Medical Informatics and Decision Making, 2021. 21(1): p. 230.
25. Stoyanov, S.R., 2015.
27. Campbell, R.J. The Five Rights of Clinical Decision Support: CDS Tools Helpful for Meeting Meaningful Use. Journal of AHIMA, 2013. 84(10): p. 42-47 (web version updated February 2016).
28. Boulos, M.N.K., et al. How Smartphones Are Changing the Face of Mobile and Participatory Healthcare: An Overview, with Example from eCAALYX. Biomedical Engineering Online, 2011. 10(1): p. 24.
29. Silvennoinen, J., M. Vogel, and S. Kujala, Experiencing Visual Usability and Aesthetics in Two Mobile Application Contexts. Journal of Usability Studies, 2014. 10(1): p. 46-62.
30. Ramsey, R.R., et al. A Systematic Evaluation of Asthma Management Apps Examining Behavior Change Techniques. The Journal of Allergy and Clinical Immunology: In Practice, 2019. 7(8): p. 2583-2591.
31. Mani, M., et al. Review and Evaluation of Mindfulness-based iPhone Apps. JMIR mHealth and uHealth, 2015. 3(3): p. e82.
32. Mandracchia, F., et al. Mobile Phone Apps for Food Allergies or Intolerances in App Stores: Systematic Search and Quality Assessment Using the Mobile App Rating Scale (MARS). JMIR mHealth and uHealth, 2020. 8(9): p. e18339.
34. Domnich, A., et al., Development and validation of the Italian version of the Mobile Application Rating Scale and Its Generalisability to Apps Targeting Primary Prevention. BMC Medical Informatics and Decision Making, 2016. 16(1): p. 83.
35. Campbell, R.J., The Five Rights of Clinical Decision Support: CDS Tools Helpful for Meeting Meaningful Use. Journal of AHIMA, 2013. 84(10): p. 42-47 (web version updated February 2016).
Hossein Aghayari (email@example.com) is a PhD candidate in the Department of Health Information Technology, School of Management and Medical Informatics, at Tabriz University of Medical Sciences, Tabriz, Iran.
Leila R Kalankesh (firstname.lastname@example.org) (corresponding author) is a professor of medical informatics at Road Injury Research Center, School of Management and Medical Informatics, at Tabriz University of Medical Sciences, Daneshgah Ave, Tabriz, Iran
Homayoun Sadeghi-Bazargani (email@example.com) is a professor of epidemiology at the Road Traffic Injury Research Center at Tabriz University of Medical Sciences, Tabriz, Iran, and affiliated with the International Safe Community Certifying Center, Stockholm, Sweden.
Mohammad-Reza Feizi-Derakhshi (firstname.lastname@example.org) is a professor of computer engineering at the Faculty of Electrical and Computer Engineering at the University of Tabriz, Iran.