Evaluating Motivation for the Use of an Electronic Health Record Simulation Game

by Alexander McLeod, PhD; Barbara Hewitt, PhD; David Gibbs, PhD; and Caitlin Kristof

Abstract

Experiential learning via simulation offers a variety of benefits including reduced risks, repetitive exposure, and mastery of complex processes. How to motivate people to engage in and enjoy playing games is an important concept in the creation of serious games focused on learning new skills. This study sought to determine the motivators that increase users’ pleasurable experience when playing an electronic health record simulation game. To examine how intrinsic and extrinsic motivation affected both engagement and enjoyment, we surveyed students of health professions at one university. Results indicate that while both forms of motivation are significant in increasing engagement and enjoyment, extrinsic motivation such as badges, points, and scoreboards were much more important than internal motivations for our participants. These findings have implications for the development of an electronic health record simulation game.

Keywords: Gamification; electronic health record; experiential learning; simulation; intrinsic motivation; extrinsic motivation

Introduction

Digital health information maintained in electronic health record (EHR) systems is an asset of significant value to both patients and healthcare organizations that requires governance by well-trained and experienced health information management (HIM) professionals. Students who will become HIM professionals must develop competency and efficiency in a variety of complex processes governing health information, which is the basis for clinical decisions that affect patient health. Health information must be accurate, complete, and available when and where it is needed for use in clinical and business decisions by all health professionals. The terms technology iatrogenesis or e-iatrogenesis refer to errors that occur because of technology introduced into the already complex healthcare delivery system.1 To minimize these errors, HIM professionals must achieve high levels of competency and efficiency through effective instruction and extensive practice. Issues with health information create substantial risks for patients, providers, and facilities. Furthermore, health information is highly targeted for compromise by perpetrators of medical identity theft as well as those seeking ransom.2 Governance of health information addresses not only the risks to patient health, but also the risks to patient privacy. The challenges associated with health information governance are evolving rapidly and result in the need for new tools and methods to learn and practice these skills in a safe, low-risk environment, such as that provided by simulation.

This article presents the results of a study examining health profession students’ perceptions of game design elements and how these elements can increase the user’s enjoyment when playing games. This study is based on the application of simulation and gamification techniques to enhance the experience of health profession students learning to use EHR systems. Inspired by previous experience teaching complex systems and encouraged by the literature about simulation in other areas of healthcare, the authors became curious about the effectiveness of a comprehensive simulation platform specifically designed to enable teaching and repetitive practice of the complex high-risk processes associated with EHR systems and how simulation game design might affect user enjoyment and engagement.

As a step toward understanding the potential of an EHR simulation platform, we conducted a survey of students in our college of health professions to explore game design elements, how these elements motivate participants, and the positive effects on engagement and enjoyment. In the sections that follow, we first provide a review of literature that informed our research question. Next we review motivation and outcomes. After the literature review, we describe our methods and results. We close with a discussion of findings and conclusions drawn from the study.

Literature Review

The current literature on gamification appraises a variety of elements associated with game design.3 Game genres, types, motivations, and other game design characteristics are important in understanding specific game preferences and motivations that might contribute to the design of an EHR simulation game. The purpose of the research was to discover what game design elements motivated HIM students from both an intrinsic and extrinsic perspective with the ultimate goal of making a simulation game engaging and enjoyable.

Gamification

Although the literature includes many examples of simulation and experiential learning in healthcare settings such as anesthesiology,4, 5 cardiology,6, 7 intensive care,8, 9 nursing,10–12 obstetrics,13, 14 outpatient healthcare,15, 16 and many other disciplines,17 research on the combination of gamification and EHRs is sparse. A notable exception is that of Mohan et al.,18 who provided six designs for consideration in development of an EHR simulation. One curriculum development study reported that simulation might be helpful and effective in EHR instruction using a “well-organized chart,” which was defined as documentation of actions reflecting current standards.19 Additionally, Kuljis et al.20 suggested that simulation in healthcare could benefit from a review of simulation in other settings, such as business manufacturing.

Both game genre and game type should be considered in the design of an EHR simulation game because they allow designers to customize the game, addressing specific learning objectives and taking advantage of analogous games that have already been designed.21 Genres such as fantasy, discovery, and serious games need to be taken into account to support and frame the design process. Common categories of game types include shooter, adventure, strategy, puzzle, and sports games. Although some games are played entirely for fun and entertainment, serious games are used as learning platforms, integrating aspects of real-world learning or skill development, and have been created for a variety of purposes, including medical education, retail, service industry, public safety, weight management, surgery, health education, and diabetes management.22–26 An EHR simulation game would be considered a serious game because students would be engaged in learning and skill development while playing to learn in an enjoyable, safe virtual environment.

Unlike serious games, fantasy games provide virtual worlds where many people opt to play, avoiding reality.27 Often, fantasy games are set in imaginary worlds and incorporate fantasy characters, graphics, narratives, and scenarios, providing alternate realities. A large advantage of these imaginary worlds is that users are insulated from repercussions.28–30 Although our literature search did not reveal any previous works relating EHRs to fantasy games, gamifying of learning materials is particularly advantageous in the healthcare field because it facilitates learning while removing the risk of error that the patient and healthcare learner may experience in real-life scenarios. With an EHR, real-life dangers that exist include erroneous patient data, patient identification challenges, security breaches, medication errors, and other situations—all of which can be safely practiced in a simulated environment while building both confidence and expertise. However, it is not only the safe environment that makes gamification appealing: increasing student engagement and enjoyment while learning new skills is also beneficial to learners.

Motivation, Enjoyment, and Engagement

The motivation for playing is crucial to increasing player engagement and enjoyment and has been researched extensively. The extent to which a participant engages in games directly correlates with learning outcomes.31 Research suggests that a combination of both intrinsic and extrinsic motivation can be used to engage players.32 Rieber33 found that when an engaging game is intentionally connected to learning content, the content itself becomes more enjoyable. This enjoyment, in turn, supports internal motivation, which is also referred to as intrinsic motivation.34, 35 Game designs that support intrinsic motivation include elements supporting personal achievement, self-determination or drive, team play, social relationships, and networks.36–38

In contrast to intrinsic motivation, extrinsic motivation uses external rewards to motivate the player.39 Certain game design characteristics are suggested to directly influence extrinsic motivation.40 Elements of game design that increase extrinsic motivation include the tracking of points and recognition-based rewards. Examples of recognition-based rewards include badges, scoreboards, leaderboards, progress bars, and level-of-progression scores.41 Because the goal is to motivate players both intrinsically and extrinsically, extrinsic game elements might be used to initially engage the user with the expectation that the enjoyment of playing would then intrinsically motivate and engage the user over the long term.42, 43

The end result of motivating participants is to engage them in learning and produce greater academic outcomes.44 People learn more quickly when they are engaged in content designed to be fun, resulting in deeper understanding, greater skill development, more confidence, and greater achievement.45 Engagement is often described as a heightened sense of concentration and can result in great enjoyment in which people lose a sense of time and surroundings. This engagement phenomenon has been termed flow theory, and it characterizes an enjoyable increase in interest and productivity.46 Hence engagement is an important aspect of game design.

While engagement is important in gamification, so too is enjoyment.47 Often enjoyment is overlooked, particularly in the completion of tedious tasks.48 Motivation can increase enjoyment if game design elements are properly used to spontaneously generate student interest.49 Self-determination theory proposes that people actively participate when they are supported and are comfortably enjoying the acquisition of new experiences.50

According to this conceptualization, gamification can be considered to have three main parts:

  1. the implemented motivational strategies,
  2. the resulting psychological outcomes, and
  3. the final behavioral outcomes.

In a systematic review, Hamari et al.51 conceptualized a structure for evaluating gamification through a three-part framework: the implemented motivation, the psychological outcome that resulted, and behavioral outcomes. Reviewing the gamification literature, they concluded that gamification leads to increased motivation, engagement, and enjoyment.52 Motivated participants are comfortable learning new things, are interested in the topic, and enjoy the learning experience.

Research Questions

We crafted our research questions invoking the motivation outcome framework suggested by Hamari et al.,53 shown in Figure 1. Given the preceding review and the motivation outcome framework, we sought to examine the following questions:

  • How do extrinsic motivators affect people’s engagement?
  • What is the effect of extrinsic game motivators on participant enjoyment?
  • What is the relationship between intrinsic motivators and participant engagement?
  • How will intrinsic motivation affect a person’s enjoyment?

Methodology

Many researchers have examined intrinsic and extrinsic motivations with the goal of improving performance, participation, engagement, and enjoyment.54–56 Methods of measuring these motivations have varied, and historically researchers exploring intrinsic and extrinsic motivation have created latent variable models and used a survey method. We used a model fashioned after prior research to examine the relationships between perceptions of motivation and how it affects pleasurable use by decomposing motivation into two constructs—intrinsic and extrinsic motivation—while incorporating two constructs of pleasurable use: engagement and enjoyment. Figure 2 details the relationship between intrinsic motivation, extrinsic motivation, engagement, and enjoyment in our research model.

Procedures

We obtained institutional review board permission to operationalize our research design. To test this predictive model, we crafted a survey using the literature previously described and distributed it to students majoring in the College of Health Professions at a major public university in the southern United States. The college currently provides degree programs in clinical laboratory science, communication disorders, HIM, healthcare administration, nursing, physical therapy, radiation therapy, and respiratory therapy. Convenience sampling was used, and an e-mail was sent to students in these majors inviting them to take part in the online survey. Seventy-nine participants completed the survey, which is shown in Appendix A. The Likert-scale survey captured the items of interest related to intrinsic motivation, extrinsic motivation, engagement, and enjoyment. Demographics of the respondents are included in Table 1.

Analysis

We used partial least squares path modeling (PLS-PM) in RStudio to analyze the data following predictive modeling techniques.57 PLS-PM analysis is commonly used in conducting latent variable research and provides a robust way of analyzing survey data.58–61 To validate the psychometric properties of the measures, the average variance extracted, Dillon-Goldstein’s rho, Cronbach’s alpha, and factor loadings were calculated. Table 2 shows each of these values for this model’s constructs.

Although there is no standard method for calculating statistically acceptable composites, the generally accepted rule is for composite reliability to be greater than 0.7.62 In this study, the lowest composite reliability was found for engagement at 0.92, followed by extrinsic motivation at 0.95. Thus, composite reliability as measured by Dillon-Goldstein’s rho was greater than 0.9 for all constructs, demonstrating sufficient reliability for all latent variables.

The factor loadings for the latent variables were calculated following the method of Sanchez63 with the use of PLS-PM in RStudio. The factor loadings are provided in Appendix B. Validity of individual items was examined by verifying loadings greater than 0.7 for each construct. All items loaded at values greater than 0.70. Nine indicators loaded greater than 0.90, six loaded between 0.80 and 0.90, and two loaded between 0.80 and 0.70. All items loaded greater “on-factor” than “off-factor,” demonstrating good convergent and discriminant validity.

Results

The structural path model was formulated to test the motivation framework. A bootstrap resampling method produced 500 samples in order to obtain the path coefficients and R2 quantities. Statistical significance was then calculated for each path by calculating t-tests. Figure 3 shows the β coefficients and p-values extracted via PLS-PM. The model accounted for a significant portion of the variance in enjoyment (R2 = 0.69) and engagement (R2 = 0.63).

The coefficient or path value for extrinsic motivation toward engagement was β = 0.64, with p < .01. Extrinsic motivation to enjoyment produced β = 0.66, with p < .01. Intrinsic motivation to engagement was also significant, with β = 0.24 and p < .01. Intrinsic motivation to enjoyment yielded a path value of β = 0.27, with p < .01. Table 3 reports our total effects findings for all participants in the survey, including the sample mean, standard deviation, t-statistic, and p-value.

The total effects findings indicate that these relationships were significant. We performed a power analysis to determine how many respondents would be required for a medium effect size with a power of .80 at an alpha level of .05, following Cohen.64 We found that 67 participants would be required when analyzing two independent variables; thus with 79 valid responses we had adequate power for our study.

Discussion

This study quantitatively explored how intrinsic and extrinsic motivation affect individuals’ engagement and enjoyment when playing games. The four research questions in this work explored how intrinsic and extrinsic motivators increase when an individual is more engaged in and/or enjoys learning while playing a game. The first research question sought to determine whether students are more likely to be engaged in electronic games if extrinsic motivators are available. The students responded that extrinsic motivators would increase their engagement in electronic games.

The second question explored whether individuals would enjoy games more if extrinsic motivators were present. The students indicated that they would enjoy gamification if extrinsic motivators such as badges, scoreboards, and/or other reward systems were available. Thus it is important to include extrinsic motivators in electronic games when these games are used to teach EHR concepts.

The third research question considered whether intrinsic motivation increases engagement in computer games. A person may be intrinsically motivated when the person feels that an opportunity is fun or challenging. Intrinsic motivation occurs even in the absence of external rewards and/or reinforcements. Students felt that being internally motivated would increase their engagement in games.

Finally, the fourth research question asked if intrinsic motivation affected a person’s enjoyment of an electronic game. Internal motivation should provide support for individual enjoyment and pleasure when playing. This form of motivation drives the individual at a personal level.

Among our population of respondents, the extrinsic motivation scores (.66 and .64) were more than double the intrinsic motivation scores (.24 and .27) for both engagement and enjoyment. While the results for intrinsic motivators were not as high as those for extrinsic motivators, students still indicated that intrinsic motivators influenced whether they enjoyed games and would be engaged in playing games that had a learning element. To ensure that individuals use game-based learning tools, one must ensure that the incentives to learn are fun as well as challenging. These results provide interesting insight into students’ perceptions of whether gamification can increase their engagement in and enjoyment of learning EHR systems.

Anecdotally, we also asked our respondents why they play games. Forty-five reported playing games out of boredom; 40, for the mental challenge; 19, for social interaction; 18, for educational purposes; 2, for physical challenges; and 10, for other reasons. We also asked participants about the types of games they play. In order of importance, the responses were puzzle, strategy, adventure, educational, multiplayer, simulation, none, role-playing, shooter, athletic, platform, other, and serious games. Appendix C provides the counts by game type.

Conclusion

Teaching students to use EHR systems is a challenging task because the systems are large and complex, requiring significant time to learn and practice. Although people can be trained to use these systems, often this training lacks authenticity because it is presented in lectures or case studies and the audience has little incentive to engage in learning the system. However, when gamification is used to teach systems, learners may be more inclined to engage with the technology for longer periods of time. Well-designed gamification motivates learners, increasing their enjoyment and making learning by doing fun. The individuals who responded to this survey were more extrinsically motivated, meaning that adding features such as badges, scoreboards, and/or other reward systems is more important than ensuring that the game is simply challenging or fun. This finding supports the view that gamification of EHR system learning would be beneficial to our target audience.

Limitations

This paper has several limitations. First, this study used students from a single department within a single college of a university. However, because the study was focused on HIM professionals and the survey was completed by HIM students near completion of their program of study, these students are likely to be a good representation of early-career HIM professionals. Early-career HIM professionals often hold jobs requiring them to engage with EHR systems.

Second, this paper proposes implementation of an EHR simulation game; however, we question whether the results would differ after respondents played the game. These results were based on responses to a survey. We question whether the results would be similar in a different type of study, such as an experiment with an actual EHR simulation game.

Third, other factors may affect a person’s gaming engagement and enjoyment. The chosen motivations do not represent all factors affecting the dependent variables, and other, more important effects remain undetermined. This study was therefore limited by the selection of the included variables.

Future Research

Although this study examined whether both intrinsic and extrinsic motivation influence one’s engagement and enjoyment in a gamified learning environment, other antecedents such as self-efficacy or voluntariness should be explored as well. Future research should explore whether results would differ on the basis of gender and/or ethnicity. Additionally, the current model explores whether intrinsic and extrinsic motivators are both antecedents of enjoyment and engagement; however, it is possible that extrinsic motivators could actually be antecedents to intrinsic motivators. Finally, this research might benefit from an extension of the research model to include other behavioral outcomes, such as intention to use a system and technology acceptance.

 

Alexander McLeod, PhD, is an assistant professor at Texas State University in San Marcos, TX.

Barbara Hewitt, PhD, is an assistant professor at Texas State University in San Marcos, TX.

David Gibbs, PhD, is an assistant professor at Texas State University in San Marcos, TX.

Caitlin Kristof is a research assistant at Texas State University in San Marcos, TX.

 

 

Notes

  1. Mohan, V., Scholl, and J. A. Gold. “Intelligent Simulation Model to Facilitate EHR Training.” AMIA Annual Symposium Proceedings (2015): 925–32.
  2. Genes, R. “Code Cyber: Preventing Breaches at Hospitals and Health Care Practices.” Journal of Health Care Compliance 18, no. 3 (2016): 13–18.
  3. Gibbs, D., B. Hewitt, and A. McLeod. “The Gamification of Electronic Health Records: A Systematic Literature Review.” Educational Perspectives in Health Informatics and Information Management (2016).
  4. Gaba, D. “Improving Anesthesiologists’ Performance by Simulating Reality.” Anesthesiology 76, no. 4 (1992): 491–94.
  5. Steadman, R. H., A. R. Burden, Y. M. Huang, D. M. Gaba, and J. B. Cooper. “Practice Improvements Based on Participation in Simulation for the Maintenance of Certification in Anesthesiology Program.” Journal of the American Society of Anesthesiologists 122, no. 5 (2015): 1154–69.
  6. Issenberg, S. B., W. C. McGaghie, D. L. Gordon, S. Symes, E. R. Petrusa, I. R. Hart, and R. M. Harden. “Effectiveness of a Cardiology Review Course for Internal Medicine Residents Using Simulation Technology and Deliberate Practice.” Teaching and Learning in Medicine 14, no. 4 (2002): 223–28.
  7. Issenberg, S. B., W. C. McGaghie, I. R. Hart, J. W. Mayer, J. M. Felner, E. R. Petrusa, R. A. Waugh, D. D. Brown, R. R. Safford, and I. H. Gessner. “Simulation Technology for Health Care Professional Skills Training and Assessment.” JAMA 282, no. 9 (1999): 861–66.
  8. Gold, J. A., A. S. Tutsch, A. Gorsuch, and V. Mohan. “Integrating the Electronic Health Record into High-Fidelity Interprofessional Intensive Care Unit Simulations.” Journal of Interprofessional Care 29, no. 6 (2015): 562–63.
  9. March, C. A., D. Steiger, G. Scholl, V. Mohan, W. R. Hersh, and J. A. Gold. “Use of Simulation to Assess Electronic Health Record Safety in the Intensive Care Unit: A Pilot Study.” BMJ Open 3, no. 4 (2013): e002549.
  10. Cairco, L., J. Bertrand, M. Gupta, R. Armstrong, S. Babu, L. Hodges, and T. Fasolino. “Towards Simulation Training for Nursing Surveillance.” Carolinas Women in Computing (2012). Available at https://www.cs.clemson.edu/vegroup/pubs/2012/2012_nursing.pdf.
  11. Zhang, W., B. Kaplan, and D. Ura. “A Comparison of Students’ Perception on Effectiveness of Integrating Electronic Health Records Into Simulation in Undergraduate Nursing Program.” Presented at the 25th International Nursing Research Congress, Hong Kong, July 26, 2014.
  12. Beyea, S. C., L. K. von Reyn, and M. J. Slattery. “A Nurse Residency Program for Competency Development Using Human Patient Simulation.” Journal for Nurses in Professional Development 23, no. 2 (2007): 77–82.
  13. Marzano, D. A., R. D. Smith, I. Greenfield, E. Beene, A. M. Piehl, and M. Hammoud. “On-Unit Obstetrics Team Simulation Enhances Implementation of a New Electronic Health Record [345].” Obstetrics & Gynecology 125 (2015): 109S.
  14. Macedonia, C. R., R. B. Gherman, and A. J. Satin. “Simulation Laboratories for Training in Obstetrics and Gynecology.” Obstetrics & Gynecology 102, no. 2 (2003): 388–92.
  15. Gold, J. A., A. S. Tutsch, A. Gorsuch, and V. Mohan. “Integrating the Electronic Health Record into High-Fidelity Interprofessional Intensive Care Unit Simulations.”
  16. Clague, J. E., P. G. Reed, J. Barlow, R. Rada, M. Clarke, and R. H. Edwards. “Improving Outpatient Clinic Efficiency Using Computer Simulation.” International Journal of Health Care Quality Assurance 10, no. 5 (1997): 197–201.
  17. Cowperthwait, A., J. Saylor, A. Carlsen, L. A. Schmitt, T. Salam, M. K. Melby, and S. D. Baker. “Healthcare Theatre and Simulation: Maximizing Interprofessional Partnerships.” Clinical Simulation in Nursing 11, no. 9 (2015): 411–20.
  18. Mohan, V., G. Scholl, and J. A. Gold. “Intelligent Simulation Model to Facilitate EHR Training.”
  19. Milano, C. E., J. A. Hardman, A. Plesiu, M. R. E. Rdesinski, and F. E. Biagioli. “Simulated Electronic Health Record (Sim-EHR) Curriculum: Teaching EHR Skills and Use of the EHR for Disease Management and Prevention.” Academic Medicine 89, no. 3 (2014): 399.
  20. Kuljis, J., R. J. Paul, and L. K. Stergioulas. “Can Health Care Benefit from Modeling and Simulation Methods in the Same Way as Business and Manufacturing Has?” In Proceedings of the 39th Conference on Winter Simulation: 40 Years! The Best Is Yet to Come (WSC ’07). Piscataway, NJ: IEEE Press, 2007, 1449–53.
  21. Grace, L. “Game Type and Game Genre.” 2005. Available at http://aii.lgracegames.com/documents/Game_types_and_genres.pdf.
  22. Lewis, M. W. “Analysis of the Roles of ‘Serious Games’ in Helping Teach Health-related Knowledge and Skills and in Changing Behavior.” Journal of Diabetes Science and Technology 1, no. 6 (2007): 918–20.
  23. McCoy, L., J. Lewis, and D. Dalton. “Gamification and Multimedia for Medical Education: A Landscape Review.” Journal of the American Osteopathic Association 116, no. 1 (2016): 22–34.
  24. Graafland, M., J. Schraagen, and M. P. Schijven. “Systematic Review of Serious Games for Medical Education and Surgical Skills Training.” British Journal of Surgery 99, no. 10 (2012): 1322–30.
  25. Walsh, C. M., M. E. Sherlock, S. C. Ling, and H. Carnahan. “Virtual Reality Simulation Training for Health Professions Trainees in Gastrointestinal Endoscopy.” Cochrane Database of Systematic Reviews 6 (2012): CD008237.
  26. Moore, C. L., and J. A. Copel. “Point-of-Care Ultrasonography.” New England Journal of Medicine 364, no. 8 (2011): 749–57.
  27. Danelli, F. “Implementing Game Design in Gamification.” In T. Reiners and L. C. Wood (Editors), Gamification in Education and Business. New York: Springer, 2015, 67–79.
  28. Garris, R., R. Ahlers, and J. E. Driskell. “Games, Motivation, and Learning: A Research and Practice Model.” Simulation & Gaming 33, no. 4 (2002): 441–67.
  29. Neeli, B. K. “Gamification in the Enterprise: Differences from Consumer Market, Implications, and a Method to Manage Them.” In T. Reiners and L. C. Wood (Editors), Gamification in Education and Business. New York: Springer, 2015, 489–511.
  30. Thomas, P., and R. Macredie. “Games and the Design of Human‐Computer Interfaces.” Programmed Learning and Educational Technology 31, no. 2 (1994): 134–42.
  31. Wilson, K. A., W. L. Bedwell, E. H. Lazzara, E. Salas, C. S. Burke, J. L. Estock, K. L. Orvis, and C. Conkey. “Relationships between Game Attributes and Learning Outcomes Review and Research Proposals.” Simulation & Gaming 40, no. 2 (2009): 217–66.
  32. Isen, A. M., and J. Reeve. “The Influence of Positive Affect on Intrinsic and Extrinsic Motivation: Facilitating Enjoyment of Play, Responsible Work Behavior, and Self-Control.” Motivation and Emotion 29, no. 4 (2005): 295–323.
  33. Rieber, L. P. “Seriously Considering Play: Designing Interactive Learning Environments Based on the Blending of Microworlds, Simulations, and Games.” Educational Technology Research and Development 44, no. 2 (1996): 43–58.
  34. Deterding, S. “The Lens of Intrinsic Skill Atoms: A Method for Gameful Design.” Human–Computer Interaction 30, no. 3–4 (2015): 294–335.
  35. Hanus, M. D., and J. Fox. “Assessing the Effects of Gamification in the Classroom: A Longitudinal Study on Intrinsic Motivation, Social Comparison, Satisfaction, Effort, and Academic Performance.” Computers & Education 80 (2015): 152–61.
  36. Perryer, C., B. Scott-Ladd, and C. Leighton. “Gamification: Implications for Workplace Intrinsic Motivation in the 21st Century.” Asian Forum on Business Education Journal 5, no. 3 (2012): 371–81.
  37. Lee, M. K., C. M. Cheung, and Z. Chen. “Acceptance of Internet-based Learning Medium: The Role of Extrinsic and Intrinsic Motivation.” Information & Management 42, no. 8 (2005): 1095–1104.
  38. Ryan, R. M., and E. L. Deci. “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions.” Contemporary Educational Psychology 25, no. 1 (2000): 54–67.
  39. Ibid.
  40. Wilson, K. A., W. L. Bedwell, E. H. Lazzara, E. Salas, C. S. Burke, J. L. Estock, K. L. Orvis, and C. Conkey. “Relationships between Game Attributes and Learning Outcomes Review and Research Proposals.”
  41. Cheong, C., J. Filippou, and F. Cheong. “Towards the gamification of Learning: Investigating Student Perceptions of Game Elements.” Journal of Information Systems Education 25, no. 3 (2014): 233.
  42. Ryan, R. M., and E. L. Deci. “Facilitating and Hindering Motivation, Learning, and Well-Being in Schools: Research and Observations from Self-Determination Theory.” In K. R. Wentzel and D. B. Miele (Editors), Handbook of Motivation at School. 2nd ed. New York, NY: Routledge, 2016, 96–119.
  43. Hall, M. T., and J. E. Marshall. “Intrinsic and Extrinsic Motivation within the Context of Modern Education.” In E. Railean, G. Walker, A. Elçi, and L. Jackson (Editors), Handbook of Research on Applied Learning Theory and Design in Modern Education. Hershey, PA: IGI Global, 2016, 292–308.
  44. Skinner, E. A. “Engagement and Disaffection as Central to Processes of Motivational Resilience and Development.” In K. R. Wentzel and D. B. Miele (Editors), Handbook of Motivation at School. 2nd ed. New York, NY: Routledge, 2016, 145–68.
  45. Patrick, H., J. C. Turner, and A. D. Strati. “Classroom and School Influences on Student Motivation.” In K. R. Wenzel and G. B. Ramani (Editors), Handbook of Social Influences in School Contexts: Social-Emotional, Motivation, and Cognitive Outcomes. New York: Routledge, 2016: 241–57.
  46. Csikszentmihalyi, M. Finding Flow: The Psychology of Engagement with Everyday Life. New York: Basic Books, 1997.
  47. Seaborn, K., and D. I. Fels. “Gamification in Theory and Action: A Survey.” International Journal of Human-Computer Studies 74 (2015): 14–31.
  48. Hanus, M. D., and J. Fox. “Assessing the Effects of Gamification in the Classroom: A Longitudinal Study on Intrinsic Motivation, Social Comparison, Satisfaction, Effort, and Academic Performance.”
  49. Cruz, C., M. D. Hanus, and J. Fox. “The Need to Achieve: Players’ Perceptions and Uses of Extrinsic Meta-Game Reward Systems for Video Game Consoles.” Computers in Human Behavior (2015).
  50. Deci, E. L., and R. M. Ryan. “Optimizing Students’ Motivation in the Era of Testing and Pressure: A Self-Determination Theory Perspective.” In W. C. Liu, J. C. K. Wang, and R. M. Ryan (Editors), Building Autonomous Learners. New York: Springer, 2016, 9–29.
  51. Hamari, J., J. Koivisto and H. Sarsa. “Does Gamification Work?—A Literature Review of Empirical Studies on Gamification.” 2014 47th Hawaii International Conference on System Science (2014): 3025–34.
  52. Ibid.
  53. Ibid.
  54. Isen, A. M., and J. Reeve. “The Influence of Positive Affect on Intrinsic and Extrinsic Motivation: Facilitating Enjoyment of Play, Responsible Work Behavior, and Self-Control.”
  55. Lee, M. K., C. M. Cheung, and Z. Chen. “Acceptance of Internet-based Learning Medium: The Role of Extrinsic and Intrinsic Motivation.”
  56. Cruz, C., M. D. Hanus, and J. Fox. “The Need to Achieve: Players’ Perceptions and Uses of Extrinsic Meta-Game Reward Systems for Video Game Consoles.”
  57. Sanchez, G. PLS Path Modeling with R. Available at http://gastonsanchez.com/PLS_Path_Modeling_with_R.pdf.
  58. Gefen, D., and D. Straub. “A Practical Guide to Factorial Validity Using PLS-Graph: Tutorial and Annotated Example.” Communications of the Association for Information Systems 16 (2005): 91–109.
  59. Chin, W. W., B. L. Marcolin, and P. R. Newsted. “A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte Carlo Simulation Study and an Electronic-Mail Emotion/Adoption Study.” Information Systems Research 14, no. 2 (2003): 189–217.
  60. Gefen, D., D. Straub, and M. Boudreau. “Structural Equation Modeling and Regression: Guidelines for Research Practice.” Structural Equation Modeling 4 (2000).
  61. Chin, W. W. “Issues and Opinion on Structural Equation Modeling.” Management Information Systems Quarterly 22, no. 1 (1998): 7–16.
  62. Vinzi, V. E., L. Trinchera, and S. Amato. PLS Path Modeling: From Foundations to Recent Developments and Open Issues for Model Assessment and Improvement. New York: Springer, 2010.
  63. Sanchez, G. PLS Path Modeling with R.
  64. Cohen, J. “A Power Primer.” Psychological Bulletin 112, no. 1 (1992): 155.

Printer friendly version of this article.

Leave a Reply