Abstract
This study evaluated the readability and suitability of a university health center public website. Readability formulas estimated the reading grade and age required for comprehension of health information. The Suitability Assessment of Materials (SAM) instrument determined adequacy of the webpages for the intended audience. Readability showed the reading grade level, representing the youngest reader able to process the material, ranged from 10.1 to 14.6, averaging 12.5 (midway through 12th grade in the US educational system). Full comprehension required higher education levels, up to postgraduate. Suitability scores for some webpages indicated deficiencies in readability, motivation to learn, and instructions for healthy behavior changes. Content on the website may be difficult for some students to comprehend based on the reading grade level, but overall suitability results are satisfactory. All webpage updates should bear these parameters in mind to ensure content is fully accessible to college students, faculty, and staff.
Keywords: readability, suitability, health literacy, college students, health center, college website
Introduction
Health literacy requires the ability to find, understand, and use information to manage health and health-related decisions about oneself and others.1 Several groups are generally considered to be at high risk for low health literacy, including immigrants, the elderly, certain ethnic groups, and people with lower education levels.2 In the US, 21 percent of English-speaking adults aged 16 to 65 have low literacy skills and are challenged by tasks such as comparing and contrasting information and paraphrasing.3 People with low health literacy are limited in their ability to access and understand health information and are at higher risk of having poor health and poorer health outcomes than those with higher health literacy.4-6
College students would not typically be considered a group at risk for low literacy. However, college students are unique in that although they have high educational achievement, they may lack experience navigating the healthcare system on their own. Newly responsible for meeting their own healthcare needs, college students are especially inclined to look to the internet for answers to health concerns. Research indicates, however, that they struggle with understanding health information and recognizing credible online sources.7,8 As they progress in college from freshman to senior, students’ health literacy improves.9 This may be due to better critical-thinking skills that enable students to find, understand, and use information to manage health.10 Nevertheless, the National Assessment of Adult Literacy (2005) indicates that 3 percent of people with a college degree have below basic health literacy skills.11
Serving adults with literacy challenges includes ensuring that health information is understandable, that is, it places a low health literacy demand on the readers.12 This responsibility on the part of health-related organizations to provide easy access to comprehensible healthcare resources patients can use to manage their health is termed “organizational health literacy” (OHL).13 Sectors included in the efforts to improve OHL include not only public health and healthcare, but also education.14 Organizational health literacy includes writing health information at a level that patients low in health literacy can understand; making the purpose of materials clear and actionable; effectively employing font, color, white space, and graphics; and creating accessible and navigable patient portals and websites.15 In general, these issues can be divided into two concerns: readability and suitability.
Background
A reasonable goal for most health care instructions is a sixth-grade reading level.16 Federal plain language guidelines,17 however, clarify that this depends on the audience and the type of information. In health information, medical terms can be confusing to patients and can skew readability scores to a higher level.18 The Joint Commission (2010) recommends that health-related materials be written in a manner equivalent to a fifth-grade education level.19
Readability is the ability to read easily and is typically measured by the number of syllables, words, and sentences within the text to obtain a US school grade reading level as a reference.20 To improve readability, medical terms may need to be eliminated and replaced with less precise terms. If larger words cannot be replaced with a simpler term, a definition should be provided, or links provided to definitions housed elsewhere.21 Simple sentences should be used, and complex information avoided.22,23
Suitability is the appropriateness of material for a given audience and looks at several variables to determine how well information can be read and understood, such as clarity of purpose, layout, and use of visuals.24 Users pay more attention to nicely displayed information they can find easily, and graphics are an important tool for helping readers interpret health information.25,26 Inconsistent text readability levels as well as variability of content quality of the websites promotes difficulty in navigating already confusing information.27
Although little is known about the readability of university student health center websites specifically, reviews of websites for lung cancer, breast cancer, heart attack, and stroke have been found to have significantly high reading levels (“fairly difficult”), which are considered inaccessible to people with low literacy.28 Even higher, “difficult,” reading levels have been noted on websites targeting younger adults, such as those containing information about anorexia nervosa.29 Given the complex interaction of college students’ high educational attainment but low experience navigating the healthcare system, readability scores based on grades beyond high school may not accurately reflect the ability of college age students to access and use health information. Campus resources, such as student health centers, should be geared to college students in their early academic years to ensure understanding is achieved.30 This is a relevant concern for campus student health centers, which typically provide online materials through their websites to assist students with a variety of health issues. These websites may be one of the first places students look for health information, especially for treatment and provider options.31 Understanding the college student audiences, including those with possible literacy challenges, is one way to improve understanding of a website’s content.
The purpose of this research is to provide information about readability and suitability of existing health information on one university website for student health services. We posed the following questions: 1) Are the university student health center webpages presented at appropriate readability levels for college student understanding? 2) Are the university student health center webpages suitable for college students?
Methods
Readability Assessment
Four clinical webpages located under the “Services” tab from a university student health center were analyzed at the request of student health center administrators. The request was part of an ongoing quality improvement program to ensure webpage information, particularly clinical information, is useful and understandable by the university students. The review looked at four webpages focused on clinical services: Primary Care, Pharmacy, Dental Center, and Specialty Care.
The Primary Care webpage includes topics such as general health care, gynecological care, stress management, immunizations, and victim services. In addition to prescription medications and over-the-counter drugs, the Pharmacy webpage includes information on asthma education. The Dental Center webpage includes information on exams, cleaning, fillings, bridges, and crowns. The Specialty Care webpage includes topics such as international health and travel clinic, sports medicine, dietitian, psychiatry, physical therapy, and referral services. Each webpage was loaded into Readability Studiosoftware version 2019.3 for Windows32 for readability score output. Analysis of the readability output was conducted by three researchers using the results of three measures, as described below.
Readability was measured with three formulae: Flesch-Kincaid Grade Level (F-KGL), Flesch Reading Ease (FRE), and Simple Measure of Gobbledygook (SMOG). Flesch-Kincaid Grade Level uses the number of words, sentences, and syllables within the text to derive a reading grade level based upon US school grade as a reference. The mathematical “yardstick” created by Flesch (1946) is a valuable tool for evaluating resources in a variety of media.33 The Flesch Reading Ease uses the average words per sentence and the average syllables per word to determine a readability score.34 The range is 0 (very difficult to read) to 100 (very easy to read). Average documents should be within the 60-70 range, as this would indicate that eighth and ninth graders can easily understand the contents.35 A readability formula developed by McLaughlin (1969), the Simple Measure of Gobbledygook (SMOG), estimates the years of education a person needs to fully understand a piece of writing.36 It is based on the number of syllables in a group of sentences. Higher numbers of polysyllabic words in a passage converts to a higher grade level. For this analysis, incomplete threshold was set at eight words as the minimum length allowed for an incomplete sentence to be considered valid. This ensures inclusion of bulleted points, which lack punctuation, found on some of the webpages.
Suitability Assessment
Suitability was evaluated using the Suitability Assessment of Materials (SAM) instrument.37 This evaluation pinpoints areas that may need further clarification or instruction. The SAM evaluates materials with scores of 0 (not suitable), 1 (adequate), or 2 (superior) for each of 22 subcategories. A total (adding all points) is then calculated to determine suitability, with a score determined after dividing by the maximum points of 44. A total score of 70 percent or above is considered superior, 40-69 percent is adequate, and 39 percent and below is not suitable.38 Unsuitable ratings would then be the focus for specific revisions of the materials. Six areas are assessed:
- Content: Readers should understand the purpose of the materials.
- Literacy demand: Includes readability, writing style, sentence construction, vocabulary, and topic captions.
- Graphic illustrations, lists, tables, and charts. Includes type of illustrations, relevance, and captions.
- Layout and typography: Fonts, layout, and subheadings are included.
- Learning stimulation and motivation: Includes interaction, desired behavior patterns, and motivation to learn.
- Cultural appropriateness: Includes logic, language and experience, cultural image and examples, and suitability for the population.
Two student reviewers (authors KT and RP) who were interested in health literacy research volunteered to participate in the reviews, which were done in February and March 2021. The students were sophomores majoring in biomedical sciences, one female and one male. The students conducted independent reviews of the webpages within the Services Tab using the SAM criteria after they were trained on usage of the instrument. Thus, coders were of similar characteristics as the target audience, as recommended by Manganello and colleagues.39 Disagreements were resolved through discussion and consensus agreement.
Results
Readability
Based on the results in Table 1, the Specialty Services webpage shows the highest grade level requirement for both the Flesch-Kincaid and SMOG, as well as the most difficult reading content on the Flesch Reading Ease. The Flesch-Kincaid indicates the Specialty Services webpage content is suitable for a college sophomore with the sixth month of class completed. The Flesch Reading Ease score of 26 indicates the text is difficult to read. This is below the standard recommended range of 60-70.40 The SMOG, which tests for 100 percent comprehension, indicates that the Specialty Services webpage is suitable for a college senior with the second month of class completed. (It should be noted, however, that the Specialty Services webpage included information about a wide range of specializations, making it difficult for content creators to avoid introducing new medical terms into each section, which inevitably raised readability scores.)
As a comparison, the Pharmacy webpage is suitable for a high school sophomore with the first month of classes completed according to the Flesch-Kincaid. The Flesch Reading Ease score is higher than the Specialty Services webpage but did not attain the recommended range of 60-70. The SMOG indicates the reading comprehension to be at a college freshman level. Dental and Primary Care webpages readability scores fell between these two extremes.
Suitability
See Table 2.
The reviewers felt the Dental Services webpage was adequate or better with the exception of “desired behavior patterns,” which may be too general (scored 0, not suitable). Total SAM score for the Dental webpage was 62.5 percent (adequate). The Pharmacy webpage was considered superior in most of the subcategories, with an overall SAM score of 80 percent (superior). The Primary Care webpage also received several superior scores but scored as not suitable for “motivation” where students may not feel confident that the tasks are doable (scored 0). The overall SAM score is 75 percent (superior). Specialty Services was considered adequate in most of the subcategories with two inadequate for “reading grade level” and “motivation” (scored 0) and an overall SAM score of 65 percent (adequate). As noted, the Specialty Services webpage shows the highest grade level requirement for readability due to the wide range of specializations and medical terms used.
Cultural appropriateness was scored N/A on all of the webpages in the Services tab. The student reviewers found the SAM instrument difficult for rating cultural appropriateness of a website designed for a highly diverse and large student body. The need to interpret cultural variations in analysis of the webpages, including logic and experiences from the student body viewpoint, added to the student reviewers’ hesitancy, and they did not feel they had adequate understanding to provide an accurate rating. However, neither of the reviewers felt this detracted from the suitability of the health-related information in this tab. In other words, generally they found the content to be culturally appropriate for a university student audience. The SAM creators do allow for the ability to use N/A for a category if it is not applicable to the material, and those points are subtracted from the total points to result in 40 possible points (rather than 44 points).41
Research Questions
Are the university student health center webpages presented at appropriate readability levels for college student understanding? There are mixed results, with the reading levels ranging from 10.1 to 14.6 reading grade level requirements. The SMOG tested for understandability, with results showing a range from 13 to 16.2 grade level requirements for full comprehension of the material. This could mean freshmen and sophomore students will struggle with the content.
Are the university student health center webpages suitable for college students? The SAM results are overall adequate, with a few subcategories scoring at not suitable. In addition to readability improvements as noted above, the category of learning stimulation and motivation with specific behavior and skills that students believe they can achieve should be considered to improve suitability.
Discussion
College students, overall, are not in a high-risk literacy category due to their academic attainment. Most have adequate health literacy.42,43 However, this literacy may not extend to ability to find and use health information from the internet, including from health services websites.44 Therefore, even health literature aimed at college students should be comprehensible at lower levels than is indicated by reading grade level scores. Although the authors are not aware of specific guidance for this audience, it is reasonable to assume that health information should be geared at least two grade levels lower than other information, in line with Joint Commission (2010) recommendations for general audiences.45 One of the university health center webpages we analyzed was written at the 10th grade level, but others were higher, suggesting that information may be difficult for a minority of students to fully comprehend. This is not unusual, as McInnes and Haglund (2011) conducted research looking at websites of numerous health conditions and found those ending in the URL extension “.edu” were the most difficult to read and required more than a 12th grade education.46
The two student reviewers who provided input for this research anecdotally noted a preference for pictures and other visual tools to make finding and comprehending material faster. As evaluated by these members of the target audience, layout, although adequate, was a low scoring element across all four webpages. This is at least partially due to the graphics and design constraints placed on the Student Health Center webmaster due to university-wide website guidelines. The same situation is likely to pertain at other institutions. Nevertheless, our findings suggest that as university health center resources are updated or added, assessing web-based resources for readability and suitability is a worthwhile endeavor. Many readability tools are readily available in common word processing software or free web-based calculators.
Limitations
The use of two reviewers for suitability scoring, although common practice in such studies,47,48 is a limitation. A larger, more formal focus group of freshmen level students could have provided more in-depth suitability information. Additionally, reviewers found the cultural appropriateness category difficult to apply to a university website intended for a broad audience as opposed to a specific ethnic group, so information about that aspect is limited. This issue can be addressed more in depth in a focus group format as well.
Finally, this study investigated four key webpages of a single university health center website. Generally, all other university health center webpages visited by the researchers varied greatly in the content organization, with no consistent layout or categorization of topics. This makes direct comparisons difficult; therefore, we provide information about one school’s webpages as a basis for readability assessments by other academic institutions. Thus, findings cannot be generalized to health information on websites at other universities.
Conclusion
People with low health literacy are limited in their ability to access and understand health information and are at higher risk of having poor health than those with higher literacy. Although college students are not at high risk for low literacy, they may struggle with health websites that require a high reading level for comprehension. Reviewing student health websites for readability and suitability gives a focus and baseline for improvements in understandability of content.
University health center websites have the opportunity to provide prefiltered peer reviewed information relevant to the student body.49 High quality health information is important for college students who rely on these websites for reliable guidance for health concerns. As expectations have risen in the Healthy People 2030 report to include organizational health literacy, academic health centers are obligated to carefully review their websites and other resources to meet the needs of the students they serve.50
Notes
1. Office of Disease Prevention and Health Promotion. Health Literacy in Healthy People 2030. U.S. Department of Health and Human Services. Accessed August 1, 2022. https://health.gov/our-work/national-health-initiatives/healthy-people/healthy-people-2030/health-literacy-healthy-people-2030
2. Logan, Robert A., Winston F. Wong, Michael Villaire, Gem Daus, Terri Ann Parnell, Earnestine Willis, and Michael K. Paasche-Orlow. "Health literacy: A necessary element for achieving health equity." NAM perspectives (2015). https://doi.org/10.31478/201507a. 2015. Accessed August 1, 2022. https://nam.edu/perspectives-2015-health-literacy-a-necessary-element-for-achieving-health-equity/
3. Mamedova, Saida, and Emily Pawlowski. "Adult Literacy in the United States. Data Point. NCES 2019-179." National Center for Education Statistics (2019). Accessed August 1, 2022. https://nces.ed.gov/pubs2019/2019179.pdf
4. Cabellos-Garcia, Ana Cristina, Antonio Martinez-Sabater, Enrique Castro-Sanchez, Mari Kangasniemi, Raul Juarez-Vela, and Vicente Gea-Caballero. "Relation between health literacy, self-care and adherence to treatment with oral anticoagulants in adults: a narrative systematic review." BMC Public Health 18, no. 1 (2018): 1-12.
5. de Melo Ghisi, Gabriela Lima, Gabriela Suéllen da Silva Chaves, Raquel Rodrigues Britto, and Paul Oh. "Health literacy and coronary artery disease: a systematic review." Patient education and counseling 101, no. 2 (2018): 177-184. doi:10.1016/j.pec.2017.09.002
6. The Joint Commission. Advancing effective communication, cultural competence, and patient- and family-centered care: A roadmap for hospitals. 2010. Accessed August 1, 2022. https://www.jointcommission.org/assets/1/6/ARoadmapforHospitalsfinalversion727.pdf.
7. Ickes, Melinda J., and Randall Cottrell. "Health literacy in college students." Journal of American College Health 58, no. 5 (2010): 491-498. doi:10.1080/07448481003599104
8. Noblin, Alice, Kendall Cortelyou-Ward, and Ashley Rutherford. “Health literacy and the college student: impact of a medical terminology course on intention to use a personal health record.” Internet Journal of Medical Informatics 7, no. 1 (2015) 1-8.
9. Dolezel, Diane, Ram Shanmugam, and Eileen E. Morrison. "Are college students health literate?" Journal of American College Health 68, no. 3 (2020): 242-249.
10. Vamos, Sandra, Paul Yeung, Till Bruckermann, Emma F. Moselen, Robyn Dixon, Richard H. Osborne, Olga Chapa, and Donna Stringer. "Exploring health literacy profiles of Texas university students." Health Behavior and Policy Review 3, no. 3 (2016): 209-225.
11. Kutner, Mark, Elizabeth Greenberg, and Justin Baer. "A First Look at the Literacy of America's Adults in the 21st Century. NCES 2006-470." National Center for Education Statistics (2006).
12. Seitz, Christopher M., Samantha Shiplo, Taylor Filippini, Zubair Kabir, Jeffrey L. Lennon, and Donald Fowler. "The reading level of government and voluntary health organization smoking cessation websites: a descriptive analysis." American Journal of Health Education 48, no. 6 (2017): 392-399.
13. Brach, Cindy, Debra Keller, Lyla M. Hernandez, Cynthia Baur, Ruth Parker, Benard Dreyer, Paul Schyve, Andrew J. Lemerise, and Dean Schillinger. "Ten attributes of health literate health care organizations." NAM perspectives (2012).
14. Santana, Stephanie, Cindy Brach, Linda Harris, Emmeline Ochiai, Carter Blakey, Frances Bevington, Dushanka Kleinman, and Nico Pronk. "Practice Full Report: Updating Health Literacy for Healthy People 2030: Defining Its Importance for a New Decade in Public Health." Journal of Public Health Management and Practice 27, no. 6 (2021): S258.
15. Farmanova, Elina, Luc Bonneville, and Louise Bouchard. "Organizational health literacy: review of theories, frameworks, guides, and implementation issues." INQUIRY: The Journal of Health Care Organization, Provision, and Financing 55 (2018): 0046958018757848.
16. Doak, Cecilia, Leonard Doak, and Jane Root. Teaching Patients with Low Literacy Skills. 2nd Edition. Philadelphia: J.B. Lippincott Co; 1996.
17. Federal plain language guidelines. (2011). Accessed August 1, 2022. https://www.plainlanguage.gov/media/FederalPLGuidelines.pdf.
18. Habeeb, Amir. "How readable and reliable is online patient information on chronic rhinosinusitis?" The Journal of Laryngology & Otology 135, no. 7 (2021): 644-647.
19. The Joint Commission, 2010.
20. McLaughlin, G. Harry. "SMOG grading–A new readability formula.” The Journal of Reading 12, no. 8 (1969): 639-646.
21. National Cancer Institute. Making Health Communication Programs Work. U.S Department of Health and Human Services. 2001. Accessed August 1, 2022. https://www.cancer.gov/publications/health-communication/pink-book.pdf
22. Abdel-Wahab, Noha, Devesh Rai, Harish Siddhanamatha, Abhinav Dodeja, Maria E. Suarez-Almazor, and Maria A. Lopez-Olivo. "A comprehensive scoping review to identify standards for the development of health information resources on the internet." PloS one 14, no. 6 (2019): e0218342.
23. Bellon-Harn, Monica L., Vinaya Manchaiah, and Krittika Kunda. "Readability, quality, and suitability of English-language Internet information about children with primary speech and language disorders." Journal of Consumer Health on the Internet 24, no. 3 (2020): 228-250.
24. Doak, Cecilia, 1996.
25. Abdel-Wahab, 2019.
26. Bellon-Harn, 2020.
27. Arts, Hanne, Henna Lemetyinen, and Dawn Edge. "Readability and quality of online eating disorder information—Are they sufficient? A systematic review evaluating websites on anorexia nervosa using DISCERN and Flesch Readability." International Journal of Eating Disorders 53, no. 1 (2020): 128-132.
28. Mcinnes, Nicholas, and Bo JA Haglund. "Readability of online health information: implications for health literacy." Informatics for health and social care 36, no. 4 (2011): 173-189.
29. Arts, Hanne, 2022.
30. Vamos, Sandra, 2016.
31. Perrault, Evan K. "Campus health centers’ lack of information regarding providers: a content analysis of division-I campus health centers’ provider websites." Health Communication 33, no. 7 (2018): 860-866.
32. Oleander Software, Ltd., Pune, Maharashtra, India, 2019.
33. Flesch, Rudolph. "A new readability yardstick." Journal of applied psychology 32, no. 3 (1948): 221.
34. Ibid.
35. Doak, Cecilia, 1996.
36. McLaughlin, G. Harry, 1969.
37. Doak, Cecilia, 1996.
38. Ibid.
39. Manganello, Jennifer A., Vani R. Henderson, Amy Jordan, Nicole Trentacoste, Suzanne Martin, Michael Hennessy, and Martin Fishbein. "Adolescent judgment of sexual content on television: Implications for future content analysis research." Journal of Sex Research 47, no. 4 (2010): 364-373.
40. Flesch, Rudolph, 1948.
41. Doak, Cecilia, 1996.
42. Ickes, Melinda J., 2010.
43. Dolezel, Diane, 2020.
44. Noblin, Alice, Meghan Hufstader Gabriel, Kendall Cortelyou-Ward, and Khristen Holmes. "Health literacy among visiting college students in the US: A pilot study." Journal of American College Health 70, no. 2 (2022): 589-597.
45. The Joint Commission, 2010.
46. Mcinnes, Nicholas, 2011.
47. Rathod, Pritesh, Lydia Ould Brahim, Eric Belzile, and Sylvie Lambert. "An evaluation of the suitability, readability, and quality of online resources for the self-management of depression." Patient education and counseling 102, no. 5 (2019): 952-960.
48. Wilson, Feleta L. "The suitability of United States Pharmacopoeia Dispensory Information psychotropic drug leaflets for urban patients with limited reading skills." Archives of Psychiatric Nursing 13, no. 4 (1999): 204-211.
49. Van de Vord, Rebecca. "Distance students and online research: Promoting information literacy through media literacy." The Internet and Higher Education 13, no. 3 (2010): 170-175.
50. Santana, Stephanie, 2021.
Author Biographies
Alice Noblin (Alice.Noblin@ucf.edu) is an associate professor and interim associate dean for academic affairs in the College of Community Innovation and Education at the University of Central Florida.
Richard I. Zraick (Richard.Zraick@ucf.edu) is a professor in the School of Communication Sciences and Disorders at the University of Central Florida.
Ann Neville Miller (Ann.Miller@ucf.edu) is a professor and interim assistant director of academic programs in the Nicholson School of Communication and Media at the University of Central Florida.
Mary Schmidt-Owens (Mary.Schmidt-Owens@ucf.edu) is the associate director of healthcare compliance at the University of Central Florida’s Student Health Services.
Michael Deichen (Michael.Deichen@ucf.edu) is the associate vice president of Student Health Services; a professor of family medicine in the College of Medicine; and a vice chair of the Institutional Review Board at the University of Central Florida.
Kelly Tran (kellytran2001@Knights.ucf.edu) is a biomedical science student at the University of Central Florida.
Raj Patel (rpatel4001@knights.ucf.edu) is a biomedical science student at the University of Central Florida.