Vol. 18, Issue 4

Evaluation of a Telehealth Application (Sehha) Used During the COVID-19 Pandemic in Saudi Arabia: Provider Experience and Satisfaction

By Mohanad M. Alsaleh, MS; Valerie J.M. Watzlaf, PhD, MPH, RHIA, FAHIMA; Dilhari R. DeAlmeida, PhD, RHIA; and Andi Saptono, PhD

Abstract

Introduction: COVID-19 has drastically transformed healthcare delivery and forced many to utilize telehealth. This study aimed to comprehensively evaluate the telehealth service “Sehha” used during COVID-19 in Saudi Arabia and assess the provider experience and satisfaction with Sehha.

Methods: A questionnaire was distributed by the Ministry of Health (MoH) to 362 physicians using Sehha. The questionnaire items were adapted from previous studies and then tested for content validity and reliability (α = 0.88).

Results: The findings showed that most of the physicians improved their experience in telehealth because of COVID-19. The majority of the physicians (67.6 percent) reported being satisfied with Sehha. However, the most commonly perceived challenge by the physicians was difficulty in providing accurate medical assessments.

Conclusion: COVID-19 has remarkably uncovered numerous benefits of telehealth. Therefore, telehealth should remain a permanent model of healthcare delivery with consideration of further telehealth development initiatives.

Keywords: telehealth, mHealth, Sehha, provider, satisfaction, experience, COVID-19, Saudi Arabia, MoH

Introduction

The novel coronavirus disease (COVID-19) pandemic has dramatically altered the operation of many industries, including healthcare across the world. In the Kingdom of Saudi Arabia (KSA), the first case of COVID-19 was reported on March 2, 2020, which led to 362,549 confirmed cases of COVID-19, including 6,214 deaths as of December 31, 2020.¹ Consequently, the Saudi officials and highest authorities established strict regulations and movement restrictions across the country to stop the spread of the virus.² Thus, it has been found that many healthcare organizations have either adopted or activated a telehealth system to virtually provide healthcare services so that the virus can be contained and healthcare services continue to be provided.

Telehealth is defined as the use of electronic information and telecommunication technologies to deliver and support health-related services remotely. Telehealth services can be provided in two forms: synchronous “real-time, interactive virtual visits” and asynchronous “store-and-forward.”³ Also, telehealth is one technology that has the potential to increase access to healthcare services, reduce healthcare costs, and sustain the continuity of care.⁴

During the COVID-19 pandemic, the Saudi Ministry of Health (MoH) progressively utilized digital health technologies such as telehealth to provide care virtually. One of the telehealth applications used in Saudi Arabia is called Sehha. The Sehha app, which translates to “Health,” is a telehealth application primarily designed to provide telehealth services such as teleconsultation, e-prescription, and telediagnosis through the MoH’s accredited doctors via chat, voice, or video calls.⁵ This telehealth service is available to all Saudi citizens and non-Saudi citizens at no cost.

This particular app holds great promise for increasing healthcare access, cutting healthcare costs, and improving healthcare outcomes. Thus, evaluating the Sehha app and the provider experience and satisfaction with Sehha during such a global health crisis may create new telehealth initiatives and developments based on the user’s experience and eventually facilitate the adoption of telehealth systems across the country.

Background

In the era of COVID-19, many countries worldwide have heavily relied on digital technologies, including telehealth services, to combat the spread of COVID-19, ensure the economy does not collapse, and continue offering education and health-related services for individuals. For example, China has enabled telehealth technologies and activated virtual care to be utilized by healthcare professionals to provide health services. The West China Hospital of Sichuan University, in collaboration with ZTE China, has been utilizing 5G technology to provide remote healthcare services and treatment.⁶

In the United States, the government has facilitated the use of telehealth, in which several policies were waived or reduced, such as HIPAA, so that healthcare organizations and patients could utilize virtual care instead of traditional (face-to-face) care. On March 17, 2020, the Centers for Medicare and Medicaid Services waived 1,135 requirements and policies to expand telehealth coverage for all Medicare patients during COVID-19.⁷ Although privacy and security rules were relaxed, healthcare organizations were advised to implement and utilize a HIPAA-compliant platform to provide a secure connection between providers and patients, thus protecting patient data privacy.

Similarly, the Kingdom of Saudi Arabia (KSA) has been proactively developing e-health and telehealth applications to achieve its strategic objectives: increasing access to care, promoting public health, and improving health outcomes. However, the MoH has partnered with other governmental agencies such as the Saudi Authority for Data and Artificial Intelligence (SDAIA) and developed multiple mobile apps and platforms aiming to attend to the needs of the public and mitigate the risk of COVID-19.⁸ The most commonly utilized mobile applications and services during COVID-19 are Tabaud, Tawakkalna, Sehhaty, Call Service Center (937), and Sehha (see Appendix 1 for a summary of apps used during COVID-19 in KSA).^9,10

Although the Sehha app was developed early in 2018, no scientific studies assessing the provider experience and satisfaction with the Sehha app have been conducted according to the literature review. Also, the effectiveness, ease of use, usefulness, and challenges of the Sehha app from a provider’s standpoint have not been examined since the pandemic hit KSA. Consequently, this study aimed to:

Assess the provider experience and satisfaction with the Sehha app during COVID-19 in KSA.
Examine the challenges faced by the provider using telehealth technologies such as Sehha.
Identify areas requiring improvement in the Sehha app.

However, this study’s findings may significantly contribute to the future developments of telehealth across KSA. Also, determining the impact of COVID-19 on the provider’s perception and experience in telehealth would offer valuable insights to the government regarding the use of telehealth. Since telehealth is a promising model of healthcare delivery and a robust approach to increase access to care and cut healthcare costs, it is imperative to ensure telehealth stays effective and efficient post-COVID-19.^11-13

Methods

A cross-sectional descriptive study was performed using an online questionnaire to evaluate the provider experience and satisfaction with the Sehha app during COVID-19 in KSA. To collect data, a 5-point Likert scale (where 5= strongly agree; 4= agree; 3= neutral; 2= disagree; 1= strongly disagree) was applied to all questionnaire items except for the following parts where multiple-choice items were formulated with a blank text box in case the participants wish to report anything other than the given choices:

Participant Demographics and Characteristics
Perceived Challenges and Concerns
Areas of Improvement

Data Collection Instrument Development and Validity

The questionnaire items (25) were obtained from different sources and related studies focusing on the provider experience and satisfaction with telehealth.^14-18 In addition to that, questionnaire items developed by other researchers to assess the system usability and acceptance, satisfaction, and future use (e.g., Technology Acceptance Model (TAM), Telehealth Usability Questionnaire (TUQ), and Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ)) were adapted as well (see Appendix 2).^19-21 The questionnaire items were sectioned and coded as follows:

Participant demographics and characteristics (6 questions)
Perceived impact of COVID-19 on provider experience “PIC” (5 questions)
Perceived usefulness “PU” (3 questions)
Perceived ease of use “PEU” (3 questions)
Perceived effectiveness “PE” (3 questions)
Satisfaction and future use “SFU” (3 questions)
Perceived challenges and concerns (1 question)
Areas of improvement (1 question)

From December 2020 to January 2021, the MoH sent the questionnaire to a list of clinical staff containing 362 physicians known and identified to be utilizing the Sehha app. On December 3, 2020, a pilot study was conducted to measure the questionnaire’s validity and reliability, wherein the questionnaire was sent out to five family physicians at the MoH who voluntarily agreed to participate in the pilot study. The pilot participants were asked if they understood the questions, wanted to include or exclude a question, and if they wanted to add other comments.

The returned feedback and notes from the pilot respondents were considerably obtained, and the questionnaire was amended accordingly. Based on the notes and responses, none of the pilot respondents suggested including and excluding any questions, and they found the questions understandable and feasible to the study’s subject. The questionnaire was validated using face and content validity methods, and the reliability of the questionnaire was calculated using Cronbach’s alpha coefficient (α = 0.88).

Statistical Analysis

Non-parametric tests (e.g., Mann-Whitney, Kruskal-Wallis, and Spearman’s correlation) were performed using the Statistical Package for Social Sciences software (SPSS v.27.0) to test the following null hypotheses at a statistical significance of p ≤ 0.05:

There are no significant differences in the participant demographics (e.g., age, gender, nationality, years of experience, medical specialty, and physician’s rank/grade) in relation to satisfaction with Sehha and preference for telehealth visits over traditional visits during COVID-19.
There is no impact of the variables (ease of use and usefulness) on provider satisfaction.
There is no relationship between the level of satisfaction and preference for telehealth.

Ethical Considerations

This research study was approved by the Institutional Review Board at the University of Pittsburgh (IRB: STUDY20100022) and by the Central Institutional Review Board at the MoH (IRB: 20 –17 8E). All participants were informed about the purpose of the study, and their consent for participation was taken. To ensure the confidentiality and privacy of the data, all questionnaires were recorded anonymously, and no personal nor identifiable data were collected.

Results

Among all received responses from the questionnaire (138), only 114 responses were considered complete and valid, while 24 returned questionnaires were excluded due to incompletion and invalidity. Since the questionnaire was distributed to 362 physicians and only 114 physicians completed the questionnaire, the response rate was 31 percent.

About half of the participants were men (56.1 percent), in the age group of 31-40 years (51.8 percent), and were of Saudi nationality (72.8 percent). Regarding the medical specialty, the majority of the respondents were family physicians representing 75.4 percent of the study sample. Regarding the physician’s rank/grade, about 47.4 percent were consultants (see Table 1 for the participants’ characteristics).

However, the COVID-19 pandemic seemed to positively impact the physician’s perception of and experience in telehealth. The majority of the physicians (84.3 percent) stated that their experience in telehealth has improved. About 80 percent of them reported an enhancement in their understanding of telehealth because of COVID-19. When it comes to preference, 43.8 percent of the respondents stated that they preferred telehealth visits over traditional visits, 35.1 percent of the respondents did not prefer telehealth over traditional care, while 21.1 percent of the respondents neither agreed nor disagreed “neutral” regarding the preference for telehealth. Most of the respondents (88.6 percent) strongly agreed or agreed that more telehealth services would be utilized in the future in KSA, as telehealth has proved its significance in providing care for patients living in remote, underserved, and restricted areas (see Table 2).

In this study, the perceived usefulness of the Sehha app was measured to identify whether or not the physician perceived it to be useful for providing telehealth services. Fifty-two percent of the respondents stated that Sehha helped them achieve their patients’ needs more quickly compared to face-to-face visits. Also, the majority of the respondents (83.3 percent) strongly agreed or agreed that Sehha would increase access to care (see Table 3).

Regarding the ease of use, 82.4 percent of the physicians found Sehha easy to use, and more than half of the respondents (59.7 percent) felt comfortable interacting with their patients through Sehha. However, the majority of the respondents (75.4 percent) stated that they could not easily access their patients’ medical records/electronic health records while using Sehha (see Table 4).

Moreover, this study’s findings showed that about 58 percent of the physicians believed that the quality of care provided via Sehha was not as good as in-person care. Overall, 68.4 percent of the respondents believed that the Sehha app was acceptable for virtually providing healthcare services (see Table 5).

One of this study’s main objectives was to measure the physician’s satisfaction with the Sehha app. The majority of the respondents (63.2 percent) would use telehealth technologies such as Sehha to deliver virtual care and telehealth services in the future. Overall, 67.6 percent of the respondents were satisfied with the work they have done through Sehha (see Table 6).

Regarding the challenges and concerns, about 74 percent of the physicians selected inaccurate medical assessments as their primary concern using the Sehha app. Also, about 71 percent of the physicians cited overlapping of medical consultations as a significant concern, in which past and current medical consultations would overlap and cause disruption in patient care. The physicians also reported other concerns (10 percent) such as increased workload, trust issues between the physician and the patient, and connection failure (see Table 7).

However, the findings showed that the Sehha app had multiple areas requiring improvement. According to the study’s findings, 86.8 percent of the physicians indicated integrating the Sehha app with other electronic systems as the most significant area requiring improvement, followed by involving other medical specialists (81.6 percent), and then increasing access to patient data (78.1 percent) (see Table 8).

The Kruskal-Wallis test showed a statistically significant difference between the age groups and satisfaction with Sehha (p= 0.005), which means that the older the provider is, the less satisfied they are with Sehha. Also, the Kruskal-Wallis test showed a statistically significant difference between the medical specialty groups and preference for telehealth visits with a p-value of 0.011, which indicates that medical specialties could impact preference for telehealth visits over traditional visits. However, no other significant differences were found among the other groups concerning the preference for telehealth visits and satisfaction (see Table 9).

However, the impact of the two variables (usefulness and ease of use) on provider satisfaction was measured by conducting the Kruskal-Wallis test to identify any significant impacts. The Kruskal-Wallis test showed statistically significant results with a p-value of 0.001 for all items concerning the ease of use (H= 52.204, H =17.810, H =38.341) and usefulness of the Sehha app (H= 60.019, H= 35.431, H= 49.313) in relation to provider satisfaction (see Table 10). Also, the Spearman’s correlation coefficient showed a strong positive correlation between preference for telehealth and provider satisfaction (R= 0.709) with a p-value of 0.001. This correlation signifies that the providers who prefer telehealth visits over face-to-face visits are also satisfied with the telehealth services they provide via Sehha.

Discussion

To the best of the authors’ knowledge, no comprehensive scientific studies have covered the provider experience and satisfaction with the Sehha app pre- and during the COVID-19 pandemic. Moreover, the effectiveness, ease of use, usefulness, and challenges of the Sehha app have not been examined from a provider’s perspective since the app was first introduced in 2018.

Also, the Sehha telehealth app is one of its kind that is governmentally supported, free of charge, available to everyone (citizens and non-citizens), and able to conduct audio-visual “virtual” visits. Therefore, it was vital to assess such an app to promote and improve telehealth in Saudi Arabia. Even though other mHealth apps developed by the MoH offer various healthcare services, Sehha is the only government-operated mHealth app primarily designed to provide telehealth visits for everyone.

Although COVID-19 has negatively altered our economy, society, and healthcare system, this study showed that COVID-19 positively impacted the provider’s perception of and experience in telehealth and redefined the practice of telehealth and virtual care.²² The findings showed that about 80 percent of the physicians improved their telehealth experience and understanding during COVID-19. More importantly, COVID-19 has revealed the precious core of telehealth and exposed numerous benefits of telehealth.

Regarding the quality of care, one study showed that 78.6 percent of the providers believed that the quality of care provided by telemedicine was excellent,²³ while this study showed that 57.9 percent of the Sehha users did not perceive the quality of care provided via Sehha as good as traditional care. Multiple factors might be the reasons behind the reported low quality of care, including inefficient workflows of conducting a telehealth visit and insufficient quality monitoring strategies. Also, since the physicians using the Sehha app stated that there was a lack of access to patient data at the point of care, inaccurate medical assessments and diagnoses could be made, leading to poor-quality healthcare services.

In terms of the provider satisfaction with telehealth, multiple studies reported acceptable to high levels of provider satisfaction with a telehealth system.^24-26For example, a study conducted by Becevic et al. (2015) showed that 86 percent of providers were satisfied with the work done through telehealth.²⁷ Those previous studies endorse the findings of this study, in which the majority of the providers (67.6 percent) were satisfied with the Sehha app.

According to one study, an inaccurate and unreliable medical assessment is one of the major telehealth concerns faced by providers,²⁸ which aligns with the findings of this study, in which 73.7 percent of the physicians stated that they were mainly concerned about the accuracy of their medical assessments while using the Sehha app.

Another study conducted by Van Kuppenveld et al. (2020) stated that real-time access to patient data enables the physician to provide better medical assessments, thereby improving health outcomes.²⁹ However, the findings showed that 75.4 percent of the physicians were unable to access patients’ medical records while providing care via Sehha; thus, the majority of the physicians (78.1 percent) emphasized the need for increased access to patient data. Inability to access the patient data might be due to a lack of integration and interoperable infrastructures.

Limitations and Future Work

Even though the researchers strived to recruit as many as possible for the study, the response rate was low (31 percent). This might be due to multiple reasons such as ineffective distribution of the questionnaire, lack of interest in the research subject, and skipping the questionnaire due to time constraints. Since the MoH has multiple mHealth apps designed for specific purposes, it is vital to assess the Sehha app’s feasibility of integrating with other systems so that a more centralized, comprehensive app is developed and can be used to offer more than just a telehealth service.

Conclusion and Recommendations

This study aimed to evaluate the aspects of the Sehha app, the provider experience and satisfaction with the Sehha app, and identify the challenges faced by the provider to pinpoint possible opportunities for the development of telehealth in KSA. The findings showed that about 80 percent of the physicians stated that their experience in telehealth has significantly improved because of COVID-19. The results showed that more than half of the physicians (67.6 percent) reported being satisfied with the Sehha app. The findings also revealed a strong positive correlation between preference for telehealth visits and provider satisfaction.

However, this study discovered that the Sehha app lacked integration with other electronic systems, had limited access to patients’ medical records, and lacked diversity in medical specialties. The Sehha app comes with multiple challenges and concerns. The difficulty in providing accurate medical assessments was the most perceived challenge by the physicians. Last, COVID-19 has notably accelerated the adoption and use of telehealth and revealed numerous benefits that were not even realized before the pandemic. Thus, telehealth should remain sustained after the era of COVID-19, and healthcare leaders should reconsider the status of telehealth.

Based on the findings of the study and participants’ responses, 13 key recommendations have been articulated for the current telehealth platform “Sehha” and any future telehealth systems that the MoH or other healthcare organizations intend to develop:

Integrate the Sehha telehealth platform with other electronic health systems such as electronic health records and mobile health apps available within the MoH for more effective communication, improved productivity, and increased interoperability.
Involve other medical specialists in the Sehha telehealth platform (e.g., dermatologists, dentists, psychiatrists) for better accuracy of medical assessments and quality of care.
Implement a user agreement policy as a standard requirement for all users (patients and providers), outlining the users’ rights and responsibilities and defining the rules, terms, and conditions for using the platform.
Monitor and audit the visits and interactions conducted within the platform between patients and providers for quality improvement purposes.
Enable accessibility to patient data and promote sharing of health information for improved continuity and coordination of care.
Balance the workload and number of consultation requests to avoid overlapping of consultations and disruption of patient care.
Boost awareness of the telehealth benefits and encourage people and providers to leverage this technology by advertising the uses and benefits of Sehha through social media platforms and mass media.
Ensure the providers’ devices and networks are secure and safe to protect data privacy and confidentiality.
Establish security and privacy protocols that align with the national standards to support patient privacy, control the flow of patient data, delegate access to the data, and enable interoperability with other electronic systems
Provide a demonstration/tutorial video of how the app can be effectively optimized so both providers and patients can refer to it if needed.
Enable the platform to be downloaded and functional not only on smartphones but also on personal computers and laptops to increase its utilization and flexibility.
Ensure the Sehha telehealth platform is regularly maintained, updated, and sustained post-COVID-19.
Identify use cases to locate errors and gaps in the process and design for improving the efficiency and effectiveness of the Sehha telehealth platform.

Last, telehealth has the potential to solve the most pressing issues of healthcare, including access to care, quality of care, and cost. The COVID-19 pandemic has impacted various industries, specifically the healthcare industry, and revealed numerous benefits of telehealth that were not even realized before the pandemic. Telehealth can be the new norm of delivering healthcare services and a key enabler of digital transformation, wherein quality healthcare services will be available at a modest cost.

Acknowledgments

The authors would not have been able to collect the data for this research without the cooperation and assistance of Dr. Yahia Dhafar, assistant director of professional affairs and general supervisor of the Sehha app; Dr. Abdullah Alkattan, PharmD, at the MoH; and Dr. Manal Alsharif, regional supervisor of the Sehha app. The authors would like to express their gratitude to everyone who contributed to this research by providing their valuable insights and thoughts during the formation of the research study.

Notes

1. MoH. COVID 19 Dashboard: Saudi Arabia. Saudi Ministry of Health. 2020. https://covid19.moh.gov.sa/.

2. UNDP. “Saudi Arabia’s Ruthless Fight Against Coronavirus.” 2020. https://www.sa.undp.org/content/saudi_arabia/en/home/library/saudi-arabia-s-ruthless-fight-against-coronavirus.html.

3. ATA. “Telehealth: Defining 21st Century Care.” 2020. https://www.americantelemed.org/resource/why-telemedicine/.

4. Wosik, J., et al., “Telehealth Transformation: COVID-19 and the Rise of Virtual Care.” Journal of the American Medical Informatics Association. 2020. 27(6): p. 957-962.

5. MoH. “MOH Apps for Smartphones.” 2020. https://www.moh.gov.sa/en/Support/Pages/MobileApp.aspx.

6. Jnr, B.A. “Use of Telemedicine and Virtual Care for Remote Treatment in Response to COVID-19 Pandemic.” Journal of Medical Systems. 2020. 44(7): p. 1-9.

7. Lee, I., et al. “Telehealth: Helping Your Patients and Practice Survive and Thrive During the COVID-19 Crisis With Rapid Quality Implementation.” Journal of the American Academy of Dermatology. 2020. 82(5): p. 1213-1214.

8. Hassounah, M., H. Raheel, and M. Alhefzi. “Digital Response During the COVID-19 Pandemic in Saudi Arabia.” Journal of Medical Internet Research. 2020. 22(9): p. e19338.

9. MoH.

10. Hassounah.

11. Whitelaw, S., et al. “Applications of Digital Technology in COVID-19 Pandemic Planning and Response.” Lancet Digit Health. 2020. 2(8): p. e435-e440.

12. Smith, A.C., et al. “Telehealth for Global Emergencies: Implications for Coronavirus Disease 2019 (COVID-19).” Journal of Telemedicine and Telecare. 2020. 26(5): p. 309-313.

13. Khairat, S., et al. “Interpreting COVID-19 and Virtual Care Trends: Cohort Study.” JMIR Public Health and Surveillance. 2020. 6(2): p. e18811.

14. Glaser, M., et al. “Provider Satisfaction and Patient Outcomes Associated With a Statewide Prison Telemedicine Program in Louisiana.” Telemed J E Health. 2010. 16(4): p. 472-9.

15. Becevic, M., et al. “User Satisfaction With Telehealth: Study of Patients, Providers, and Coordinators.” Health Care Manag (Frederick), 2015. 34(4): p. 337-49.

16. Baert, S., et al. “The COVID-19 Crisis and Telework: A Research Survey on Experiences, Expectations and Hopes.” 2020.

17. Ayatollahi, H., F.Z. Sarabi, and M. Langarizadeh. “Clinicians’ Knowledge and Perception of Telemedicine Technology.” Perspectives in Health Information Management. 2015. 12: p. 1c.

18. Alaboudi, A., et al. “Barriers and Challenges in Adopting Saudi Telemedicine Network: The Perceptions of Decision Makers of Healthcare Facilities in Saudi Arabia.” Journal of Infection and Public Health. 2016. 9(6): p. 725-733.

19. Parmanto, B., et al. “Development of the Telehealth Usability Questionnaire (TUQ).” International Journal of Telerehabilitation. 2016. 8(1): p. 3-10.

20. Bakken, S., Grullon-Figueroa, L., Izquierdo, R., Lee, N. J., Morin, P., Palmas, W., and Starren, J. “Development, Validation, and Use of English and Spanish Versions of the Telemedicine Satisfaction and Usefulness Questionnaire.” Journal of the American Medical Informatics Association. 2006. 13(6), 660-667.

21. Davis, F. D. “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology.” MIS quarterly. 1989. 319-340.

22. Wosik.

23. Al-Sofiani, M.E., et al. “Rapid Implementation of a Diabetes Telemedicine Clinic During the Coronavirus Disease 2019 Outbreak: Our Protocol, Experience, and Satisfaction Reports in Saudi Arabia.” Journal of Diabetes Science and Technology. 2021. 15(2): p. 329-338.

24. Glaser.

25. Becevic.

26. Al-Sofiani.

27. Becevic.

28. Khemapech, I., W. Sansrimahachai, and M. Toachoodee. “Telemedicine–Meaning, Challenges and Opportunities.” Siriraj Medical Journal. 2019. 71(3): p. 246-252.

29. van Kuppenveld, S.I., et al. “Real-Time Access to Electronic Health Record via a Patient Portal: Is it Harmful? A Retrospective Observational Study.” Journal of Medical Internet Research. 2020. 22(2): p. e13622.