동서대학교 동서대학교 디자인대학
홈 회원가입 일반대학원 디자인학과 한국연구재단 SDGs 센터 사이트맵 검색
아카이브학위논문박사논문(PhD Theses)

학위논문 Theses and Dissertations

NO.D.2022.08_10

‘게임화한 제2언어습득(Gamified-SLA)’시스템디자인 평가를 위한 사용성 휴리스틱 가이드라인 연구 Usability Heuristic Guidelines for the Evaluation of Gamified Second Language Acquisition (Gamified-SLA) System Design

  • Name : 웨이나/WEI NA
  • Info : 박사학위논문/Doctor's thesis/ 2022.08
  • Adviser : 장주영/Chang, Ju-young
192.168.95.160

초록

지난 20년간, 정보 기술과 모바일 기기의 발전은 보다 정교하고 현실감있는 사용자 친화 요소 등을 모두 갖춘 인터페이스를 탄생시켰다. 특히 디지털 미디어 산업에서 유래된 게이미피케이션의 개념은, ‘게임 요소를 비게임 환경에 활용한다’는 의미로 사용자의 참여를 유도하여 문제를 해결하는, 사용자에게 이전에 없던 체험 등을 제공해 주었다.
이러한 게이미피케이션의 가장 주목할 만한 장점은 사용자의 동기와 참여 욕구의 상승인데, 이는 교육의 관점에서 학생의 교육적 성과를 판가름하는 매우 중요한 조건이다. 이와 같은 게이미피케이션의 잠재적 효과는 학생뿐만 아니라 기획자와 언어 교육자에게도 흥미를 유발한다. 본 연구에서는 다양한 학술적 연구를 통해 제2언어습득 과정에서의 게이미피케이션 기술 응용 및 그 가치를 확인했으며, 이를 통해 학습자의 신체, 심리, 감정을 모두 포괄하는 참여의 유도를 실현하고자 했다.
제2언어습득은 모국어 외 다른 언어를 학습하는 것이다. 중국에서는 약 3억 명의 인구가 제2언어를 습득하고 있다. 교육 기술 및 네트워크 통신이 발전하면서 컴퓨터 보조 언어학습은 제2언어습득에서 핵심적인 분야가 되었으며, 다양한 학문에서 이를 연구하고 있다. 코로나의 영향을 받아 중국에서 인터넷+교육 기술의 반전이 가속화되었고, ‘게임화한 제2언어습득’ 시스템을 통한 학습자의 동기와 효율을 높이기 위한 시도를 적극적으로 하고 있다. 하나의 완벽한 게임을 디자인하는 것과 비교했을 때 게이미피케이션은 게임의 핵심 요소를 추출해 비게임 영역에 활용할 수 있어 더욱 쉽게 게임의 생산성을 재현할 수 있다.
게이미피케이션의 잠재적 효과를 제2언어습득에 보다 효과적으로 사용하기 위해서는 디자인 가이드라인과 평가 방법을 필수적으로 마련해야 한다. 이에 닐센(Nielsen)과 몰리쉬(Molich)가 개발한 사용성 휴리스틱을 대다수 사용자 인터페이스 디자인 가이드라인 및 평가에 적용해오고 있지만, 게이미피케이션의 제2언어습득을 위한 응용 시스템은 일반적인 교육과는 다르다. 따라서 특정 내용 및 분야를 기반으로 하는 사용성 휴리스틱에 관한 확장 연구를 통해 특정 사용성 문제를 식별 및 확인하여 디자인 효과를 개선하고 사용자 수요를 만족시켜야 할 것이다.
이를 위해 본 연구는 핵심적인 연구 목표 세 가지를 설정했다. 첫 번째 목표는 ‘게임화한 제2언어습득’ 시스템 디자인에 적용할 수 있는 종합적인 사용성 가이드라인을 마련하는 것이다. 다음으로, 개발된 사용성 가이드라인에 대한 전문가 검증을 진행한다. 그리고 마지막으로, 새로 개발된 사용성 휴리스틱 가이드라인을 활용하여 ‘게임화한 제2언어습득’ 시스템 관련 디자인에 대한 사용성 평가를 진행한다.
본 연구는 다양한 연구 방법을 통해 상술한 연구 목표를 이루었다. 먼저, 체계적 문헌고찰을 통해 ‘게임화한 제2언어습득’ 분야의 관련 연구를 상세히 살펴봤으며, 연구 현황 및 미흡한 부분 등을 파악하여 연구 범위를 설정했다. 이후 통합적 문헌고찰을 바탕으로 제2언어습득 시스템 및 게이미피케이션의 핵심 요소 등 관련 이론을 분석하고 통합적으로 정리했다. 이를 기반으로 ‘게임화한 제2언어습득’ 시스템 디자인 평가에 적용할 휴리스틱 가이드라인을 마련했다. 해당 가이드라인의 유효성 및 합리성을 검증하기 위해 전문가 6명을 대상으로 세 차례의 델파이 조사와 인터뷰를 진행했다. 또한, 117명의 사용자를 대상으로 전문가 6명의 조사와 인터뷰 결과에 대한 설문조사를 진행하는 등 두 차례의 검증 과정을 거쳤다. 마지막으로, 사용성 전문가 3명을 초청해 본 연구에서 개발한 사용성 가이드라인으로 두 개의 기존 ‘게임화한 제2언어습득’ 시스템의 디자인을 실제 평가 및 분석하는 과정을 거쳤다.
이상 본 논문은 체계적·통합적 문헌연구, 전문가 검증 및 휴리스틱 평가를 거쳐 최종적으로 ‘게임화한 제2언어습득’ 시스템디자인 평가를 위한 사용성 휴리스틱 가이드라인과 체크리스트를 도출하였는데, 연구의 성과는 다음과 같이 정리할 수 있다. 먼저, ‘게임화한 제2언어습득’ 시스템을 적용한 31가지 사용성 휴리스틱 가이드라인의 개발이다. 이 휴리스틱는 디자인 단계에 따라 설계 가이드라인을 제공할 수 있다. 둘째는, 휴리스틱 가이드라인을 기반으로 31가지 문항으로 구성된 더욱 직관적인 사용성 체크리스트를 도출한 것이다. 해당 체크리스트로 사용성 전문가는 시스템 디자인의 사용성 문제를 빠르게 식별할 수 있을 것이다. 셋째, 상술한 휴리스틱 가이드라인은 3단계 평가 모델로 요약할 수 있다. 1단계는 평가 차원, 2단계는 평가 지표, 3단계는 평가 세칙이다. 이러한 평가 모형은 ‘게임화한 제2언어습득’ 시스템 디자인의 통합형 모델을 평가하는 데 사용되며 전통적인 인터페이스의 사용성뿐만 아니라 제2언어습득 및 게이미피케이션 관련 요구사항을 포함한다. 따라서 재설계 의사 결정을 위한 정확한 가이드라인을 제공할 수 있을 것이다.
본 논문의 의의로서, 구축한 사용성 휴리스틱 가이드라인은 기존 사용성 평가 시스템을 효과적으로 보완할 수 있을 것으로 여겨진다. 해당 휴리스틱 평가 가이드라인은 ‘게임화한 제2언어습득’ 시스템 디자인에서 전문가가 사용성 장애를 식별하는 데 도움이 되며, 일종의 유연한 평가 방법으로 직접적으로 혹은 보조 도구로써 포커스 그룹 인터뷰, 사용자 관찰 등 평가 및 테스트를 돕는데 활용될 수 있을 것이다. 재설계 방안에 대한 정확한 가이드라인을 제시하기 때문에 이러한 피드백은 특히 유의미할 것으로 사료된다.

摘要

Abstract

The concept of gamification originated from the digital media industry, which refers to “the application of gaming elements to non-gaming contexts” to engage users and solve problems”. Gamification has already been successfully used in education, marketing, organizational, health and environmental initiatives to achieve desirable outcomes by influencing user behavior. Interest in applying gamification to education is increasing, given its capacity to capture and sustain students’ attention, which is a prerequisite for students’ success in educational environments. The gamified second language acquisition (gamified-SLA) systems emerge a significant development in recent years, therefore, reliable evaluation methods are required throughout the system design process. While general heuristic guidelines are suitable to evaluate most user interfaces, there is still a need to establish heuristic guidelines for gamified-SLA domains to ensure that their specific usability issues are identified. In this landscape, based on the existing usability heuristic evaluation method proposed by Nielsen and Molich, a renew usability heuristic guidelines was assumed.
Starting from usability evaluation, this study proposed a series of usability heuristic guidelines with three separate dimensions: interface usability heuristic, SLA usability heuristic and gamified usability heuristic through integrated literature review method. Among them, interface usability heuristic involves the requirement of system performance design according to the ten basic heuristic guidelines proposed by Nielsen and Molich, which includes core elements such as efficiency, effectiveness, ease of use, consistency, and intuitiveness. In view of the SLA usability heuristic, the research outcomes of a number of experts in related fields were sorted out and synthesized to explain how the second language was acquired and its internal key elements: meaning focus, quality, effect, learner fit and emotion. These 5 evaluation indicators cover 9 specific heuristic guidelines. For gamified usability heuristic, seven significant indicators that constitute a game were also obtained through a deeply analyzes to the theoretical and practical work of game and video game experts: autonomy, goals, rules, challenges, interaction, feedback and enjoyment which including 14 corresponding evaluation heuristic guidelines.
After the preliminary evaluation heuristic guidelines are determined, the Delphi method through three rounds of interviews with six experienced experts was conducted. Experts carefully reviewed each evaluation indicators and proposed amendments. As a result, 32 heuristic guidelines with instructive significance were synthesized.
Subsequently, the structural questionnaire with 117 responders was carried out for further validating the heuristic guidelines. All the respondents have a working background in HCI or design industries and are familiar with gamified second language acquisition tools. This ensured the objectivity and validity of the questionnaire.
To this point, the three-layer evaluation heuristic with 31 guidelines were synthesized and confirmed. The first level refers to the three evaluation dimension, the second level is the 17 evaluation indicators within each dimension, and the third level corresponds to the 31 concrete evaluation heuristic guidelines. The last  work  of this  study  is  an application experiment of the evaluation heuristic guidelines  developed in the preceding work. Two of the most reputed gamified-SLA systems in the Chinese application store were evaluated and analyzed. Based on 31 heuristic guidelines, three experts found the usability issues of the two applications after two walkthroughs, and pointed out the severity degree which provides guidance for further application optimization. At the same time, the three experts also confirmed that the newly developed usability heuristic guideline is helpful for quickly and comprehensively discovering the defects of system design, and has the value of popularization and application and will provide scientific guidance for the design of the gamified-SLA system in the future.
The novel usability heuristic guidelines are an effective supplement to the existing usability evaluation system. Simultaneously, it is a flexible evaluation method that can be directly used as an auxiliary tool to guide qualitative evaluation work for users, such as focus group interviews and user observations. The heuristic guidelines obtained in this study are able to identify usability obstacles in specific design aspects. This feedback is especially valuable as it provides precise input for making redesign decisions.

키워드

  • #게이미피케이션
  • # 게임화한 제2언어습득 시스템
  • # 사용성 평가
  • # 전문가 평가
  • # 휴리스틱 가이드라인


  • #Gamification
  • # Gamified second language acquisition system
  • # Usability evaluation
  • # Expert review
  • # Heuristic guidelines

참고문헌

[Book]

1. Albert, B, & Tom, T, Measuring the User Experience (Interactive Technologies): Collecting, Analyzing, and Presenting Usability Metrics, Newnes, 2013
2. Bogost, I, How to do things with videogames, U of Minnesota Press, 2011
3. Chapelle, C, Computer applications in second language acquisition: Foundations for teaching, testing, and research, Cambridge University Press, 2001
4. Crawford, C, The art of computer game design, McGraw-Hill Osborne Media, 1984
5. Csikszentmihalyi, M, Flow and the Foundations of Positive Psychology, Springer, 2014
6. Ericsson, K. A, & Herbert, A. S,  Protocol analysis: Verbal reports as data, the MIT Press, 1984
7. Gillham, B, Developing a questionnaire, A&C Black, 2008
8. Helmer, 0, Analysis of the future: the Delphi method, Rand Corporation, 1967
9. Huizinga, J, Homos Ludens, Chinese version, Guizhou Press, 2007
10. Kapp, K. M, The gamification of learning and instruction: game-based methods and strategies for training and education, John Wiley & Sons, 2012
11. Leaver, B. L, & Willis, J, Task-based instruction in foreign language education: Practices and programs, Georgetown University Press, 2004
12. McGonigal, J, Reality is broken. Why Games make us better and how they can change the world, Penguin Press, 2011
13. McLaughlin, B, Second language acquisition in childhood: Volume 2: School-age Children, Psychology Press, 2013
14. Saldaña, J, The coding manual for qualitative researchers (3rd ed.), Thousand Oaks, CA: Sage Publications, 2016
15. Zdarzil, H, Pädagogische Anthropologie, Chinese version, Shanghai Educational Press, 2001

[Academic Journal]

1. Abran, A, et al, Usability meanings and interpretations in ISO standards, Software quality journal, 2003
2. Akbari, Z, Current challenges in teaching/learning English for EFL learners: The case of junior high school and high school, Procedia-Social and Behavioral Sciences, 199, 2015
3. Alomair, Y, & Salah, H, A review of gamified techniques for foreign language learning, Journal of Educational Research and Reviews, 2019
4. Ardoiz, G. L, Gamification in English teaching in primary education, doctoral dissertation, University of Valladolid, Spain, 2017
5. Baldeón, J, et al, LEGA: a Learner-centered Gamification design framework, Proceedings of the XVII international conference on human computer interaction, 2016
6. Bevan, N, Usability is quality of use, Advances in Human Factors/Ergonomics, Elsevier, 1995
7. Bias, R. G, The pluralistic usability walkthrough: coordinated empathies, Usability inspection methods, John Wiley & Sons, Inc., New York, 1994
8. Bland, J. M, & Altman, D. G, Cronbach’s alpha,  BMJ, 314 (7080), 572, 1997
9. Bunchball, I, Gamification 101: An introduction to game dynamics, 2012
10. Casañ-Pitarch, R, An approach to digital game-based learning: Video-games principles and applications in foreign language learning, Journal of Language Teaching and Research, 2018
11. Chanier, T, & Selva, T, The ALEXIA system: the use of visual representations to enhance vocabulary learning, Computer Assisted Language Learning, 1998
12. Chin, J. P, et al, Development of an instrument measuring user satisfaction of the human-computer interface,  Proceedings of the SIGCHI conference on Human factors in computing systems, 1988
13. Chun, D. M, & Brandl, K. K, Beyond Form‐Based Drill and Practice: Meaning‐Enhancing CALL on the Macintosh, Foreign Language Annals, 1992
14. Clanton, C, An interpreted demonstration of computer game design, CHI 98 conference summary on Human factors in computing systems, 1998
15. Dalkey, N. C, The Delphi method: An experimental study of group opinion, 1969
16. Davis, F. D, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS quarterly, 1989
17. Davis, F, et al, Extrinsic and intrinsic motivation to use computers in the workplace 1, Journal of applied social psychology, 1992
18. Desurvire, H, & Wiberg, C, Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration, International conference on online communities and social computing, Springer, Berlin, Heidelberg, 2009
19. Deterding, S, et al, From game design elements to gamefulness: defining "gamification", In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments, 2011
20. Deterding, S, et al, Gamification. using game-design elements in non-gaming contexts, In CHI'11 extended abstracts on human factors in computing systems, 2011
21. Diaz, R. M, The intellectual power of bilingualism, 1984
22. Doughty, C, & Michael, L, Optimal psycholinguistic environments for distance foreign language learning, Language Learning & Technology, 2003
23. El-Masri, M, et al, A Design Science Approach To Gamify Education: From Games To Platforms, ECIS, 2015
24. Eltahir, M, et al, The Perspective of Students and Faculty Members on the Efficiency and Usability of E-Learning Courses at Ajman University: A Case Study, Journal of Technology and Science Education, 2019
25. Flavián, C, Guinalíu, M, & Gurrea, R, The role played by perceived usability, satisfaction and consumer trust on website loyalty, Information &management, 2006
26. Galdo, E. M, et al, A critical incident evaluation tool for software documentation, Ergonomics and human factors, Springer, New York, 1987
27. Garland, C. M, Gamification and implications for second language education: A meta analysis, 2015
28. Glassman, E. L, & Miller, R. C, Leveraging learners for teaching programming and hardware design at scale. In Proceedings of the 19th ACM conference on computer supported cooperative work and social computing companion, 2016
29. Gnauk, B, Dannecker, L, & Hahmann, M, Leveraging gamification in demand dispatch systems. In Proceedings of the 2012 Joint EDBT/ICDT workshops, 2012
30. Graves, K, The curriculum of second language teacher education,  The Cambridge guide to second language teacher education, 2009
31. Hamari, J, et al, Does gamification work? A literature review of empirical studies on gamification, In Proceedings of the 47th Hawaii International Conference on System Sciences, 2014
32. Hanus, M. D, & Fox, J, Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance, Computers & education, 2015
33. Harpur, P, & De Villiers, M. R, MUUX-E, a framework of criteria for evaluating the usability, user experience and educational features of m-learning environments, South African Computer Journal, 2015
34. Hermawati, S, & Lawson, G, Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Applied ergonomics, 2016
35. Hartson, H. R, et al, Criteria for evaluating usability evaluation methods, International journal of human-computer interaction, 2001
36. Hertzum, M, Images of usability, Journal of Human–Computer Interaction, 2010
37. Hollingsed, T, & David, G. N, Usability inspection methods after 15 years of research and practice, Proceedings of the 25th annual ACM international conference on Design of communication, 2007
38. Horwitz, E. K, Foreign and second language anxiety,  Language teaching,  2010
39. Huang, B, et al, Investigating the effects of gamification-enhanced flipped learning on undergraduate students’ behavioral and cognitive engagement, Interactive Learning Environments, 2019
40. Iosup, A, & Epema, D, An experience report on using gamification in technical higher education. In Proceedings of the 45th ACM technical symposium on Computer science education, 2014
41. Jean, G, & Daphnée, S, Grammar teaching and learning in L2: Necessary, but boring? Foreign language annals, 2011
42. Jeffries, R, & Desurvire, H, Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin, 1992
43. Jones, C, & Fortescue, S, Using computers in the language classroom. Addison-Wesley Longman Limited, 1987
44. Kahn, M, & Prail, A, Formal usability inspections, John Wiley & Sons, Inc., New York, 1994
45. Kang, C, Research on the application design of language education for preschool children based on the concept of gamification, dissertation, 2019
46. Kenning, M. M, Computer-assisted language learning, Language Teaching, 1990
47. Kirakowski, J, Corbett, M, & Sumi, M, The software usability measurement inventory, Br J Educ Technol, 1993
48. Korhonen, H, & Koivisto, E, Playability heuristics for mobile games, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, 2006
49. Krashen, S, Theory of second language acquisition, 2007
50. Law, E, et al, Understanding, scoping and defining user experience: A survey approach, Proceedings of the 27th International Conference on Human Factors in Computing Systems, 2009
51. Levy, M, Technologies in use for second language learning, The modern language journal, 2009
52. Levy, M, & Hubbard, P, Why call call “CALL”? Computer Assisted Language Learning, 2005
53. Levy, M, & Stockwell, G, CALL dimensions: Options and issues in computer-assisted language learning, Routledge, 2013
54. Lewis, J. R, IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human‐Computer Interaction, 1995
55. Lewis, J. R, & Sauro, J, Can I Leave This One Out? The Effect of Dropping an Item From the SUS, Journal of Usability Studies, 2017
56. Lewis, J, Measuring Perceived Usability: The CSUQ, SUS, and UNUX, International Journal of Human-Computer Interaction, 2018
57. Liu, I. F, Gamified mobile learning: effects on English learning in technical college students, Computer Assisted Language Learning, 2022
58. Liu, T. Y, Developing an English mobile learning attitude scale for adult learners, Journal of Educational Technology Systems, 2017
59. Mahfuzah, B. R, Gender and language: Communication apprehension in second language learning, Procedia-Social and Behavioral Sciences, 2014
60. Malone, T. W, Heuristics for designing enjoyable user interfaces: Lessons from computer games, Proceedings of the 1982 conference on Human factors in computing systems, 1982
61. Marache-Francisco, C, & Brangier, E, Process of gamification. Proceedings of the 6th Centric, 2013
62. McGee, M, et al, Understanding the Usability Construct: User-Perceived Usability, Human Factors & Ergonomics Society Annual Meeting Proceedings, 2004
63. Moher, D, et al, Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, Annals of internal medicine, 2009
64. Moher, D, et al, Preferred reporting items for systematic review and meta-analysis protocols 2015 statement, Systematic reviews, 2015
65. Muntean, C. I, Raising engagement in e-learning through gamification, In Proc. 6th international conference on virtual learning ICVL, 2011
66. Nielsen, J, Finding usability problems through heuristic evaluation, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992
67. Nielsen, J, & Mack, R, Usability inspection methods, John Wiley & Sons, Inc., New York, 1994
68. Nielsen, J, & Molich, R, Heuristic evaluation of user interfaces, Proceedings of the Conference on Human Factors in Computing Systems (CHI 90), Seattle, WA, April 1-5, 1990
69. Nielsen, J, & Phillips, V, Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared, Proceedings of the Conference on Human Factors in Computing System (CHI 93), Amsterdam, April 24-29, 1993
70. Norman, D. A, THE WAY I SEE IT Memory is more important than actuality, Interactions, 2009
71. Oja, M, & Riekki, J, Ubiquitous framework for creating and evaluating persuasive applications and games, In International Conference on Grid and Pervasive Computing Springer, Berlin, Heidelberg, 2011
72. Osherson, D. N, et al, Learning theory and natural language, Cognition, 1984
73. Papaloukas, S, Patriarcheas, K, & Xenos M, Usability assessment heuristics in new genre videogames, 13th Panhellenic Conference on Informatics, IEEE, 2009
74. Parmaxi, A, & Demetriou, A. A, Augmented reality in language learning: A state‐of‐the‐art review of 2014–2019, Journal of Computer Assisted Learning, 2020
75. Rafek, M. B, et al, Gender and language: Communication apprehension in second language learning, Procedia-Social and Behavioral Sciences, 2014
76. Rahimi, M, & Pourshahbaz, S, English as a Foreign Language Teachers' TPACK: Emerging Research and Opportunities: Emerging Research and Opportunities, IGI Global, 2018
77. Roopa, S, & Rani, M. S, Questionnaire designing for a survey, Journal of Indian Orthodontic Society, 2012
78. Rowe, G, & Wright, G, Expert opinions in forecasting: the role of the Delphi technique, Principles of forecasting, Springer, Boston, MA, 2001
79. Saha, D, & Mukherjee, A, Pervasive computing: a paradigm for the 21st century, Computer, 2003
80. Sauro, J, & Lewis, J. R, Quantifying the User Experience: Practical Statistics for User Research, Rapid Answers to Over 100 Examples and Exercises, Measuring Usibility LLC, 2012
81. Schaffhauser, D, Remote Learning Will Continue Growing over the Next Three Years. The Journal, 2020
82. Schlueter, A. L, Learning English for Young Learners “The Aspect of Intercultural Dimension, Diversity Arts and Media in the Early Foreign Language Education”, In Sixth of International Conference on English Language and Teaching, Atlantis Press, 2019
83. Shield, L, & Agnes K, Are language learning websites special? Towards a research agenda for discipline-specific usability, Journal of Educational Multimedia and Hypermedia, 2006
84. Shneiderman, B, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley Longman Publishing Co., Inc, 3 ed. Boston, 1997
85. Ssemugabi, S, & Ruth, D. V, A comparative study of two usability evaluation methods using a web-based e-learning application,  Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, 2007
86. Stege, L, et al, Serious games in education, International Journal of Computer Science in Sport, 2011
87. Taber, K. S, The use of Cronbach’s alpha when developing and reporting research instruments in science education, Research in science education, 2018
88. Tan, W, et al, Web evaluation: Heuristic evaluation vs. user testing, International Journal of Industrial Ergonomics, 2009
89. Thom, J, Millen, D, & DiMicco, J, Removing gamification from an enterprise sns, In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, 2012
90. Torraco, R, Writing integrative literature reviews: Guidelines and examples, Human resource development review, 2005
91. Vasilopoulos, G, et al, The practicality and relevance of peace in an EFL teacher training program: Applications and implications, Critical Inquiry in Language Studies, 2019
92. Veljković, M, M, Gamification in foreign language teaching: Do you Kahoot? Sinteza 2017-International Scientific Conference on Information Technology and Data Related Research, Singidunum University, 2017
93. Warschauer, M, & Healey, D, Computers and language learning: An overview, Language teaching, 1998
94. Warschauer, M, Motivational aspects of using computers for writing and communication, Second Language Teaching &Curriculum Center, University of Hawaii at Manoa, 1996
95. Wei, N, & Chang, J, Enhancing User Enjoyment in e-Learning System through Gamification-A Theoretical Framework for Designing an Enjoyment Journey From a Multi-disciplinary Perspective, 한국과학예술융합학회, 2021
96. Wei, N, & Chang, J, SUS-GSLA：a New Assessment Tool for Evaluating the Gamified SLA System Design, 한국과학예술융합학회, 2022
97. Wharton, C, et al, Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992
98. Whittemore, R, & Knafl, K, The integrative review: updated methodology, Journal of advanced nursing, 2005
99. Witt, M, Scheiner, C. W, & Robra-Bissantz, S, Gamification of online idea competitions: insights from an explorative case, In GI-Jahrestagung, 2011
100. Yudintseva, A, Game-enhanced second language vocabulary acquisition strategies: A systematic review, Open Journal of Social Sciences, 2015
101. Yürük, N, Edutainment: Using Kahoot! As a review activity in foreign language classrooms, Journal of Educational Technology and Online Learning, 2019
102. Rogers, Y, Sharp, H, & Preece, J, Interaction Design: Beyond Human-Computer Interaction, 3. ed., Hoboken: John Wiley & Sons, 2011
103. Zanakis, S. H, & Evans, J. R, Heuristic “optimization”: Why, when, and how to use it, Interfaces, 1981
104. Zarzycka-Piskorz, E, Kahoot it or not? Can games be motivating in learning grammar? Teaching English with Technology, 2016
105. Zhao, Y, & Lai, C, MMORPGs and foreign language education, Handbook of research on effective electronic gaming in education, 2009
106. Zichermann, G, & Christopher C,  Gamification by design: Implementing game mechanics in web and mobile apps, O'Reilly Media, Inc., 2011

[Website]

1. Guidelines for Developing Successful Games - https://www.gamedeveloper.com/production/guidelines-for-developing-successful-games
2. Internet World Stats. 2020. Top Ten Internet Languages in The World - Internet Statistics https://www.internetworldstats.com/stats7.htm
3. Speaking the Same Language About Research | Viget - https://www.viget.com/articles/speaking-the-same-language-about-research/
4. What is Heuristic Evaluation? | Interaction Design Foundation (IxDF) - https://www.interaction-design.org/literature/topics/heuristic-evaluation
5. 啟發式評估：您對轉換優化的啟發式評估的完整指南 - https://www.eskymobile.com/blog/heuristic-evaluation-your-complete-guide/

목록