Development of a Weighted Well-Being Assessment Mobile App for Trauma Affected Communities: A Usability Study
By Steve Moeini PhD; Valerie Watzlaf PhD, MPH, RHIA, FAHIMA; Leming Zhou, PhD; and Rev. Paul Abernathy, MPIA
A well-being mobile app was built and tested by performing a usability study in a trauma affected community (TAC). Seven usability tasks were given to social workers during Phase 1. Phase 2 of the usability study was a re-test of the same tasks with the same social workers after refinements were applied. The results showed that most users preferred darker foreground colors, lighter background colors, larger fonts, and larger sized UI components. Statistically significant improvements were found after changes were implemented to the app and included time for page navigation (Z = -2.366, p = 0.018), logout (Z = -1.997, p = 0.046), and item selection in a page (Z = -2.371, p = 0.018). UI positioning and size changes proved to be a significant determinant of user satisfaction based on the positive feedback received from the computer systems usability questionnaire (CSUQ). (User1: p = .000, User 2 withdrew; User3: p = .010, User4: p = .000, User5: p = .001, User6: p = .006, User7: p = .025). HIM professionals assisted in the design, development, and administration of the usability study. This is another area in which HIM professionals are needed when assessing health and wellness in communities affected by trauma.
Keywords: Mobile app, wellness, usability study, trauma affected communities (TAC)
The collection of health information is paramount in the current information age. Over a decade ago health organizations were embarking toward fluid data warehouse designs and big data strategies. A decade later the focus has shifted toward machine learning and applying techniques of artificial neural networks to gain further insights and to predict future health outcomes. The HIM profession as a whole is one that is highly data dependent and research such as this leads the profession closer to understanding the ever encompassing and newly discovered segments of human populations that generate new clusters of health information. This paper discusses a potential new area for data collection and ultimately discovering new insights in future studies utilizing this data. Individual assessment of health and well-being has far reaching capabilities, which can affect community satisfaction and community well-being.1 In order to address individual health and wellness, we can leverage mobile technology. One area that can benefit from the ubiquitous nature of mobile devices and social networking are trauma affected communities (TACs). TACs are most often a byproduct of an individual’s exposure to violence, sexual assault, and/or disasters. TACs could best be defined as a group of individuals who have been affected mentally and/or physically by violence and/or disaster and who share the same socioeconomic issues as a collective. In addition, trauma affected services are services catering to this at risk cohort with the understanding that the services given are attuned to individuals within this community having suffered past trauma.2 Due to the nature of trauma, especially in early childhood, higher incidence of chronic disease and behavioral health issues are prevalent, which increase healthcare costs for this population. Moreover, delivery of care issues, such as missed appointments and inability to follow treatment regimens, become increasingly difficult in this segment of the population due to emotional and behavioral instability. In addition, this population also see skyrocketing healthcare costs in the form of high-volume emergency department visits.3 Given the difficulty to traditional treatment adherence and follow-up within TACs, new healthcare strategies are needed. One such strategy is to gauge the well-being of each individual within TACs via a behavioral health community organizer (BHCO) who utilizes mobile-based technologies to gather wellness data. In a trauma informed service, the BHCOs are similar to social workers and understand the plight of the participants as they too live and work in this community.
The BHCOs who enter into these communities must be highly mobile and be able to track participants who may not be at home. Mobile technology is used to increase productivity and decrease time lost and data management issues prevalent in traditional paper-based approaches. For example, in a traditional clinic-based approach, participants would be required to adhere to a schedule and to follow through with appointments. This is a challenge within this population due to the inconsistency of meeting appointment times, for example. Using a mobile solution solves this problem by creating a point of care service in which individuals, cognizant of trauma, are deployed within these communities to gather data remotely, going into the community to generate data and analytics at the point of care.
It should be noted that even though we are leveraging mobile health-based technologies, the most important piece in this ecosystem is the BHCO. The ability of the BHCO to establish trust and rapport with the community is paramount.
The goal of this research is to provide a quality health and well-being assessment to TACs utilizing mobile technology. A mobile-based weighted well-being scale was leveraged to assess TACs.4 The benefit of this scale is that it was built from the ground up focusing specifically on the issues facing TACs. Its scale is deployed via a mobile app which runs on tablets. This allowed the BHCO to be highly mobile in capturing population data with the ease of use of the mobile app. In addition, the weighted scale has been validated through expert input and tested in previous research.5 Research such as this may share some small similarities to other mobile health (mHealth) initiatives that have been deployed in low socio-economic status (SES) countries.6-9 However, as of this writing, this is the first weighted well-being mobile application to address the area of TACs. The structure of this paper is set up as follows: background information and literature review are provided as well as the methodology for the usability study starting with initial requirements analysis provided by the client, the FOCUS Pittsburgh Free Health Center (FPFHC), which serves this TAC in Pittsburgh, PA. Next, we describe the system design and steps followed in conducting the usability study. Finally, the results of the usability study, discussion, and conclusions are presented.
A literature review revealed a large number of well-being questionnaires. Many of the questionnaires evaluated short and long term disabilities toward quality of life, or to evaluate one domain in general, and others looked at multiple domains,10-14 while others focused on chronic illness and cancer.15-18 However, the questionnaires that dealt with multiple domains still focused mostly on the physical/mobility side of well-being with the addition of a relational context.19 None of the questionnaires found in the literature were specifically designed to evaluate the overall well-being of individuals in TACs. The literature review also indicated that the five domains of physical health,20, behavioral health,21, 22, (SES),23 relationships,24-27 and spiritual life,28-30 are directly related to the well-being of an individual. A focus group of community leaders and residents of a TAC in Pittsburgh rated the importance of these domains. Based on the prior research in this area, this research team developed the Well-being Relational Stability Competency Index (WRSCI), which was based off of many well-known and validated questionnaires such as PROMIS,31-33 Urban Poor Quality of Life, Friedman well-being scale, Adverse Childhood Experience (ACE) test,34 and survey questions created by the Western Pennsylvania Regional Data Center, to name just a few. The remaining new questions were generated from the research team of experts, with input from members of the TAC from FPFHC in Pittsburgh. Each new question was subject to content analysis, a content validity ratio, and a content validity index determined by a focus group. The WRSCI questionnaire was then reviewed by a group of stakeholders (community leaders and representatives of the Pittsburgh community). The research team then adjusted the well-being questionnaire. Another round of reviews was conducted by the stakeholders and a final version was approved.5, 35 Both the prior study and this current usability study are part of a larger study called the Neighborhood Resilience Project aimed at addressing the well-being of TACs.
Requirement Analysis of Client
The beta version of the mobile app was developed for the Android operating system via Android Studio integrated development environment (IDE). The app is optimized for Android-based tablets (Nexus 9) running version 22 or higher. Prototyping of the app was done based on a set of rules gathered during requirements analysis from the FPFHC and our research team.
Prior to the implementation of a mobile app, the users of the system interacted with the participants and gathered data via paper forms. They would create participant lists and manually go through the paper questionnaire, etc. The goal of the app was to facilitate the real world functions of the users more efficiently, while at the same time maintaining the privacy and security of participants and their data. The mobile app is a digital representation of the WRSCI questionnaire.
Ease of use is vital when developing the mobile app and navigation should be quick and fluid. Therefore, our goal was to perform a usability study to determine usability issues and make improvements for the final version of the mobile app wellness assessment for TACs.
The requirement analysis of the client, FPFHC includes:
1. Authorization of who can access the system
a.) Not storing personal information within the app. A unique identifier was created for each participant. Participant information was stored in a database on a server.
b.) The participants cannot be identified by the app data alone.
c.) Authorized personal (BHCOs) enter the data. Participants do not enter data into the app; they simply respond to the BHCO when asked questions.
2. Large number of question sets
a.) Questions are split into domains and within those domains exist subdomains.
i.) Domains: The survey has five domains (physical, spiritual, relational, socioeconomic, and behavioral) which make up the different sections. Each domain has a set of subdomains associated with it. A user working with a participant will traverse through the main domains. This is the high-level view of the survey.
ii.) Subdomains: Domains are made up of subdomains. Subdomains are bucketed areas that contain all the questions. One or more questions can belong to a subdomain. For example, the Physical Domain is broken up into smaller subdomains, such aspain, fatigue and medication; behavioral domain contains the subdomains positive reactions, negative reactions, traumatic events and resilience etc.
3. Make the design of the questionnaire as simple to use as possible. A justification for speed, reliability, and ease of use must be accomplished.
4. Scalable creating a large social network of patient participants.
a.) BHCOs will target the single participant and then lead to the assessment of the family of the participant, then the street block, next the community, followed by the city and then the state and even the country as a whole.
5. Accessibility built in to address the cohort using the app
a.) Must address usability issues of the BHCOs (such as readability of font size, background color and general usability of user interface components)
6. Data collection must have the ability to send the data to the server at the end of assessment to handle all data storage and analytics and then be displayed on a Web portal. This will be done automatically without user direction.
Implementing the Client’s Requirements (System Design)
Listed below are how the design requirements for the mobile app were implemented:
1. The app required an Internet connection to save data to the server.
2. Prior to a live participant assessment on the app, authorized personnel logged in and inputted the participant’s private information into a secure server. The system assigns a generic ID to the participant. The app receives the generic ID associated with the participant.
3. Due to the large amount of question sets, all of the loading and storing of questions was done remotely and accessed by the mobile app. A separate database housed all the questions.
a.) The structure of the database was built based on a standard star schema model, with various dimensions and fact tables.
4. The app presented a simplified design to be more efficient than the paper counterpart. Areas of simplicity include:
a.) Large fonts for better readability
b.) Distinct background colors to make text easily readable
c.) Large drop-down menus for easy and quick selection
d.) Linear navigation pane that pulls out data via swipe gestures or a single button press
5. Upon completing the questionnaire, the app displayed the final well-being score and displayed a bar chart showing the domain scores for the given participant. A bar chart was chosen because it’s easy to understand and ease its ease of use with the graphic design. We wanted to show the participants and providers the results immediately for ease of understanding and explanation while they are with a healthcare provider such as the BHCO.
Usability Study of Mobile App
To assure a high-quality mobile app that met all of the client and design requirements, a usability study with the BHCOs was conducted. Usability studies are performed to observe users as they perform specific tasks while using a specific system or device. The purpose of the usability study is to determine the user friendliness of the app and determine if there are specific changes that need to be made to improve the initial design of the mobile app. It should be noted, the term user interface (UI) refers to the components that make up the interface of the mobile application. For example, the drop-down box placement, navigational sliders, and radio buttons make up the UI of the mobile application described in this paper. The placement and design of the these components affect the overall experience of the user with the application.
Study Participants and Inclusion Criteria
Inclusion criteria included an English-speaking cohort of varying age and gender who are BHCOs that work/volunteer at the FPFHC where participants may be triaged if needed. All BHCOs were over 18 years of age and had a background in social work or had worked with members of a TAC previously. Almost all the participants use mobile devices on a daily basis and have used a smartphone device or tablets in the past. The demographic backgrounds of participants are shown in Table 1. Nine participants were gathered at the start of the study, but two BHCOs were lost to scheduling conflicts and one BHCO was unable to complete the CSUQ in phase 1 and was also removed from that specific portion in phase 2 (this individual was still present for the usability testing portion, even-though they didn't complete the CSUQ responses, leaving a final total of N = 7 for the usability testing itself).
Task Scenarios and Video Recording
Phase 1 of the usability study consisted of 7 tasks (Table 2). The prototype app was designed and loaded on the tablet used by the BHCO. An overhead camera was used to capture user tactile response with the mobile app. The camera was an invaluable resource in this study as it helped to aid retro analysis of each users initial learning curve and statistical analysis.
The time on task data was gathered via retrospective analysis of the video recordings. Video recordings were also used to time users on various task scenarios.
The study began with talk aloud scenarios from the principal investigator (PI) who would read aloud each task scenario, followed with the user completing the task. After the user completed one task, they were asked several follow-up questions and to rate their experience. Each BHCO user was also timed on four key areas which made up all of the tasks. Timed tasks included initial login into the app, traversing between domains, traversing between pages inside domains, and logging out. Since a key component of the app is the use of drop-down menu, users were also judged on the time it took to select a value from a drop down.
Phase 2 of the usability study was a refinement of the app considering the changes received from phase 1. The task list for phase 2 was reduced since user preferences were gathered in phase 1 and we wanted to see if the changes made in phase 1 would improve user task performance. Phase 2 usability tasks are shown in Table 3.
At the end of each usability study (phase 1 and phase 2) session users were asked to complete the IBM Computer System Usability Questionnaire (CSUQ). (Appendix A) The CSUQ was used to measure participants’ overall satisfaction with the mobile app. The CSUQ scores are on a seven-point scale, the lower the response, the higher a user’s satisfaction with the system.
Data generated in this research took two forms. We gathered metrics from video recordings such as time spent on task. Descriptive statistics were used to quantify the time on task scenarios. Secondly, a Wilcoxon signed-rank test was used to compare Phase 1 and Phase 2 results for both the time on task scenarios and the CSUQ. Wilcoxon signed-rank test can be used to compare repeated measurements on a single sample to see differences in their mean ranks. In the case of this paper we are repeatedly measuring outcomes pre- and post-intervention.
After each session ended, the user filled out the CSUQ to rate their experience. Based on the Time on Task results in phase 1 and 2 (Table 4), in addition to the time it took for and the CSUQ phase 1 results (Table 5), it was clear more changes were needed to improve ease of use and overall usability. The results in phase 1 of the CSUQ were fairly scattered with issues still being present in the app build. Most issues were related to sizing and readability of content. Phase 2 CSUQ results showed a more satisfied cohort. A Wilcoxon signed-rank test showed that the CSUQ results post phase 1 user interface adjustments were significantly favorable among participants. All of the participants who re-rated the app saw a significant usability improvement. (User1: p = .000, User3: p = .010, User4: p = .000, User5: p = .001, User6: p = .006, User1: p = .025)
Participants rated each item shown in Table 6 based on a Likert scale. Table 6 displays the user interface likeability ratings given after each talk aloud scenario in phase 1. The usability components in Table 6 range from common things like different font styles/size and common UI modules. A brief description of each UI component used in the phase 1 trial follow. A listview in software interfaces is merely an itemized list, potentially separated out by small gaps in between each item. The background color of the listview item can be modified for better readability. A radio button is a selected icon in a group/set of buttons, only one of which is selectable at a given time. Drop-down lists are fairly common, made up of a list of items allowing a user to select an item in that list. Slide-out navigation is an interface component that can be activated by a finger gesture usually by swiping the edge of a device and dragging down momentarily. Sliders are activated by finger press and hold of a button and then dragged to its desired location on a horizontal line.
Upgrades to the Mobile App Based on Phase 1 Results:
In phase 2 of our usability study, we summarized all the major issues from phase 1 and presented a finalized mobile app which included modified portions of the apps’ UI to better address the shortcomings of phase 1 of the usability study. To increase efficiency and reduce time spent interacting with the UI, most questions from the questionnaire were converted to radio buttons. The logout button was consolidated into the main slide-out navigation window. This allowed us to remove any extra menu items from the taskbar. Having only one navigation pane simplified the user experience. The page selection dropdown also posed an issue in phase 1 mainly due to its small size and location on the screen. It was important to separate anything related to page numbers from any other integer/counter based interface items to limit confusion. In phase 2 we enlarged the drop-down page navigation item and scaled it horizontally almost all the way to the width of the tablet (2d). Figures 1a-d and Figures 2a-f below shows the comparison between pre and post UI design changes running on the tablet, respectively.
Starting with Figure 2a-f below, the app is made up of the login screen (2a), participant selection screen (2b), which is made up of all patients who belong to a BHCO, slide-out navigation pane used to jump to each wellbeing domain (2c) and the general questionnaire view (2d), which the patient uses to answer each question. Even though there were significant changes to parts of the UI in phase 2, some sections did not change very much. This included the login screen (1a vs 2a) and slide out navigation (1c vs 2c).
Additionally, phase 2 of the app redesign saw the addition of a user session, shown in figure 2e. The intent of the user session is that a patient may have multiple assessments over time. In order to track each snapshot of the progress in time a session variable is needed. Each session can be analyzed independently to see if a patient improved over time. A session can also be paused and later resumed.
Lastly, Figure 2f shows the patient’s results once the questionnaire is completed. Upon completion of the WRSCI questionnaire the patient is presented with their well-being score, the outcome of each section of the well-being questionnaire, and their ACE score. It should be noted that a higher well-being score means the patient is doing well in those areas. However, opposite to that is the ACE score. An ACE score is an aggregate of different types of abuse, neglect and other adverse childhood trauma/experiences. Ideally, the goal for a patient is to have a low ACE score. A high ACE score represents a higher probability of increased problems later in a child’s life and into adulthood.
Users who had issues with the app in phase 1 saw them reduced in phase 2, mainly due to UI enhancement. Figure 3 shows a closer view of the redesigned page drop-down which allows participants to move between pages. This was one of the biggest changes in the UI redesign. Subtle changes to this area as can be seen from figure 1d to 2d made a large impact for time on task scenarios and general quality of life improvements for users. Textual and background color changes and clear/large defined fonts seen below helped users with usability and navigation.
In comparison to the time on task of phase 1, a drop in time to completion was seen, most likely due to a decrease in the initial learning curve and UI updates. The total time spent logging into the app dropped by about 41 percent, navigation between domains dropped by 33 percent, navigation within pages dropped 79 percent and total time to logout dropped by 73 percent. A Wilcoxon signed-rank test showed that time on task measurements for a set of scenarios during the mobile app usability test showed statistical significance when the page navigation dropdown values were redesigned from its original app stage build to new version (p = 0.018) with (alpha = .05). In addition, the changes to the logout placement button (p = 0.046) and time from dropdown button press to value selection (p = 0.018) were also statistically significant, respectively. See Figure 4 for time differences (in seconds, total time for all users combined) between the phase 1 and phase 2 navigation, logout placement and dropdown UI changes.
Table 7 below displays a final breakdown of the various components that were rated dealing with satisfaction during the talk aloud task scenarios in phase 2. Quality of life improvements in the UI helped to increase user satisfaction.
In this study we tested a wellness mobile app against a cohort of users which consisted of BHCOs from the FPFHC in Pittsburgh. The main purpose of the usability study was to test the ease of use of the mobile app and at the same time administer the well-being questionnaire in the TAC. The implementation of the mobile app along with a web portal administration will reduce paperwork and increase clinical productivity allowing for a smoother rapport between BHCOs and participants. Compared to other studies dealing with well-being questionnaires, this is the first of its kind, to our knowledge, to use a mobile-based weighted well-being questionnaire for TACs.
Most of the usability issues were initially caught by the first five participants, backing Nielsen’s postulate of the +5 rule.36Despite the background of the participants having used smartphones and who continue to use mobile devices regularly/daily, there was a small learning curve to understand the Android keyboard layout. Some other small issues included understanding the functioning of the Android soft keys found on all Android devices. These are generally the universal back button, the application select button, and the home button, all of which are located at the bottom of the mobile device. Overall, phase 2 proved to be the turning point for a successful completion to the usability study. Almost all users rated with high satisfaction with the following comments:
"The app is much better with increased font and dialog box sizes”
“The app is great”
“App is much easier to use”
“Having used the iPhone for so long it is still difficult because of the small buttons. This version’s icons and buttons are larger and easy to select"
The data that is generated from the questionnaire does not reside locally on the tablet. Rather it is sent to the server in real time as the BHCO is selecting answers on the questionnaire. As such, the device requires an always on Internet connection via cell network or Wi-Fi. Therefore, one drawback may be data loss should the cell network connection fail.
In this paper the authors presented a two-phased usability study. The initial phase consisted of the raw app design built for testing. The initial phase posed some usability issues since it presented an alpha build of the app. Phase 2 represented a refinement over phase 1 with some slight modifications of the app design. Based on user response, phase 2 was successful. The app changes for phase 2 testing proved vital to the users and received general praise. Most of the users were happy with its current progress. Having the BHCOs participate in the usability study was important because they were able to improve the very product that they would be using during their participant well-being assessments in the community.
The tasks presented in this study could be applied to future research in which a larger cohort is used with a clear separation in age range to see whether there are differences in time on task scenarios between the age groups.
We thank the participants from the FOCUS Pittsburgh Free Health Center for choosing to participate in this study and also FOCUS Pittsburgh for allowing us to conduct the study on-site.
Dr. Steve Moeini is a business intelligence engineer with over 15 years experience building analytics solutions in the healthcare sector. His background includes relational modeling and analytical design.
Dr. Valerie Watzlaf is vice chair of education and associate professor within the Department of Health Information Management in the School of Health and Rehabilitation Sciences (SHRS) at the University of Pittsburgh. She also holds a secondary appointment in the Graduate School of Public Health.
Dr. Leming Zhou is an associate professor and the program director for the Master of Science in Health Informatics program within the Department of Health Information Management. He also holds secondary appointments in the Department of Bioengineering in the School of Engineering and McGowan Institute for Regenerative Medicine.
Rev. Paul Abernathy is the CEO of Neighborhood Resilience Project (NRP) located in Pittsburgh, PA. Rev. Paul has worked with the research team mentioned above in building various NRP projects related to TACs.