Mednarodno inovativno poslovanje
Journal of Innovative Business and Management
1.01 Original scientific article = Izvirni znanstveni članek
* Assoc. Prof. Dr., DOBA Business School, Prešernova ulica 1, 2000 Maribor, Slovenija, and Bicero, d.o.o., Koropčeva ulica 1, 2313 Fram, Slovenija; tomislav.rozman@net.doba.si
** Prof. Dr., UVT, West University of Timisoara, Faculty of Economic and Business Administration, J. H. Pestalozzi 16, 300115 Timisoara, Romania; liliana.donath@e-uvt.ro
How to cite this paper = Kako citirati ta članek: Rozman, T. and Donath, L. (2019). The Current State of the Gamification in E-Learning: A Literature Review of Literature Reviews. Mednarodno inovativno poslovanje = Journal of Innovative Business and Management, 11 (3), 519. DOI: 10.32015/JIBM/2019-11 -3-2
© Copyrights are protected by = Avtorske pravice so zaščitene s Creative Commons Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0) / Creative Commons priznanje avtorstva-nekomercialno 4.0 mednarodno licenco (CC BY-NC 4.0)
Mednarodno inovativno poslovanje = Journal of
Innovative Business and Management
ISSN: 1855-6175
The Current State of the Gemification in E-Learning: A Literature Review of Literature Reviews
Tomislav Rozman* | Liliana Donath**
Abstract; The aim of this article is to present a systematic literature review of literature reviews (a meta-study) of gamification topics in e-learning with a purpose to present a high-level overview of the state of the development of the selected field. The systematic literature review of literature reviews (2010 -2019) was performed. The academic papers (literature reviews) from several academic databases such as DOAJ, ScienceDirect, Google Sholar and WOS were examined and filtered for further study. The selected articles were analysed for sub-topics (e.g. the efficiency of gamification mechanisms) and summarized. In last 5 years, the research of gamification in various fields (mobile apps, e-health, human resource management, business development, e-learning) is getting traction and the number of research articles and systematic reviews of research articles is increasing. After filtering literature review articles, we have found out that only a handful of them - 19, which summarizes 2631 studies, are directly related to gamification in e-learning. This study shows that the following gamification mechanisms in e-learning proved themselves efficient in almost all underlying studies, but there is no broad consensus on terminology and how to group of different gamification related concepts.
The research is limited to the following scholarly databases; DOAJ, Elsevier, Google Scholar, and WOS. We are aware that there is a possibility we missed some systematic literature reviews on this topic, which was indexed in other well-known databases or published using different terminology in the title or abstract. The article can serve as a starting point for the research for the scholars and especially post-graduate students who are getting familiar with the e-learning, gamification and performing systematic literature reviews. Also, developers of e-learning systems and course designers could benefit from this article by finding out which gamification mechanisms are most researched, most efficient and worth implementing. To our knowledge, this kind of meta-analysis of the literature reviews hasn't been done in the field of gamification in e-learning for the year 2019.
Keywords; gamification; e-learning; meta-analysis
Trenutno stanje igrifikacije v e-učenju: pregled literature o pregledih literature
Povzetek: Namen tega članka je predstaviti sistematičen pregled literature o pregledih literature (meta-študija) o igrifikacijskih temah v e-učenju z namenom predstaviti pregled na visoki ravni stanja razvoja izbranega področja. Opravljen je bil sistematičen pregled literature o pregledih literature (2010-2019). Študijski prispevki (pregledi literature) iz več akademskih baz podatkov, kot so
5
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
DOAJ, ScienceDirect, Google Sholar in WOS, so bili pregledani in filtrirani za nadaljnji študij. Izbrani članki so bili analizirani po podtemah (npr. učinkovitost mehanizmov igrifikacije) in povzeti. V zadnjih petih letih so bile opravljene raziskave o igrifikaciji na različnih področjih (mobilne aplikacije, e-zdravje, upravljanje s človeškimi viri, razvoj poslovanja, e-učenje), število raziskovalnih člankov in sistematičnih pregledov raziskovalnih člankov se povečuje. Po filtriranju preglednih člankov o literaturi smo ugotovili, da jih je le peščica (19), ki povzema 2.631 študij, neposredno povezanih z igrifikacijo v e-učenju. Ta študija kaže, da so se mehanizmi igrifikacije v e-učenju izkazali za učinkovite v skoraj vseh študijah, vendar ni širokega soglasja glede terminologije in načina razvrščanja različnih konceptov, povezanih z igrifikacijo.
Raziskava je omejena na naslednje znanstvene baze podatkov: DOAJ, Elsevier, Google Scholar in WOS. Zavedamo se, da obstaja možnost, da smo zgrešili nekaj sistematičnih pregledov literature na to temo, ki so bili indeksirani v drugih znanih bazah podatkov ali objavljeni z različnimi terminologi v naslovu ali izvlečku. Članek lahko pomaga znanstvenikom in še posebej podiplomskim študentom, ki se seznanjajo z e-učenjem, igrifikacijo in izvajanjem sistematičnih pregledov literature. Tudi razvijalci sistemov za e-učenje in oblikovalci predmetov bi lahko imeli koristi od tega članka, če bi ugotovili, kateri mehanizmi igrifikacije so najbolj raziskani, najbolj učinkoviti in jih je vredno implementirati. Kolikor nam je znano, tovrstna metaanaliza pregledov literature v letu 2019 še ni bila opravljena na področju igrifikacije v e-izobraževanju.
Ključne besede: igrifikacija; e-učenje; meta analiza 1 Introduction
imagine you are a developer of online (e-learning) courses who want to motivate attendees of online courses to complete them, improve their retention rate or improve their satisfaction with the e-learning process. Or, a researcher or a student who wants to explore the efficiency of gamification mechanisms.
You probably heard for example that PBL (Points-Badges-Leaderboards) mechanism is a must in e-learning process, but it is difficult to implement it to be effective. Or, sometimes PBLs can be even counterproductive. Moreover, a plethora of more subtle gamification mechanisms (such as achievements, transparency, goals, story, competition, collaboration, challenge, choice, role-playing, virtual goods etc.) was defined by the developers of gamified systems, which makes an additional burden on the course designer. Which ones are worth to implement into a course? Which mechanisms were already researched?
Within the article, we have answered these questions using the data we have gathered from the existing literature reviews. The rest of the article is structured as follows:
-	Firstly, the gamification related terminology is presented and the fields where it is currently used.
-	The second chapter presents the research methodology of this meta-study of the literature reviews. The article gathering and filtering process are presented. To get the high-level overview of this topic and the current trends, the systematic literature review of systematic literature reviews was performed. Various academic search engines (such as Google Scholar, DOAJ, Springer, WOS) were used.
-	The third chapter presents the extracted gamification designs, game mechanics and factors. The fourth chapter presents a list of suggestions for e-learning course designers. Lastly, the research is summarized, and some future research directions are discussed.
1.1 The terminology and related work
1.1.1 Gamification
Gamification is the application of game dynamics, game psychology and game mechanics to non-game situations and applications (Deterding and Dixon, 2011).
Gamification is the process of using game mechanics and game thinking in non-gaming contexts to engage users and to solve problems (Groh, 2012). Gamification is not the same thing as games, playful design, toys, as explained by
6
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
(Deterding et al., 2011; Kim, 2015). Gartner (Burke, 2014) defined it as "the use of game mechanics and experience design to digitally engage and motivate people to achieve their goals".
Gamification leverages game design, loyalty program design and behavioural economics to create the optimal context for behaviour change and successful outcomes. In this sense, gamified strategies have been implemented to attract, engage, activate and retain customers, but also in an educational footpath. It is used to influence and encourage specific behaviour of employees and customers (Prince, 2013; Prince et al., 2013; Dale, 2014).
1.1.2 Gamification in various contexts
Besides the learning context, gamification can be used and applied to various business and other fields (Burke, 2013), such as: services / marketing (Huotari, 2012; Huotari and Hamari, 2012), IT service management (Orta and Ruiz, 2012), risk management (Bajdor, 2011), training (Bellotti et al., 2013), e-learning (Raymer, 2011), crowdsourcing (Morschheuser, Hamari and Koivisto, 2016), project management (Aseriskis and Damasevicius, 2014), K-6 schools (simoes, Redondo and Vilas, 2013).
The correlation of usefulness of gamification with gender, age and similar has been researched by (Hamari, Koivisto and Sarsa, 2014; Koivisto and Hamari, 2014). Authors also argue and question the effectiveness of the gamification (Hamari, Koivisto and Sarsa, 2014). For example, (Kappen and Nacke, 2013) developed a design-centric model and analysis tool for gamification, which can be used as guidelines for the effective use of the gamification in business applications. But most important, gamification can also be considered as one of the most efficient growth-hacking strategies (Newhouse, 2014; Mapplinks, 2017).
The trend of using 'gamification' term started in the year 2010 and is stable since then as shown in [Figure 1].
1 Nov 2008	1 Feb 2012	1May2015	lAug2018
Figure 1: Search trend of the term 'gamification' over time (trends.google.com)
According to the Google Trends, the term 'gamification' is related to the search topics: learning, education, business, marketing and most related search queries are: 'gamification is' and 'gamification learning'. According to (Kasurinen and Knutas, 2018), there are 400 authors who publish about the gamification topic.
This chapter briefly presents the application of gamification in different fields (education, business development, human resource management, software development, ...).
This article focuses on gamification in education, specifically the effectiveness of gamification mechanisms in e-learning and their relation to the learners' motivation to attend and complete courses, course drop-out rate and satisfaction with the courses. Therefore, terms gamification in relation to learning is presented in the next chapter.
1.1.3 Gamification in e-learning
E-learning is defined according to (Rodrigues et al., 2019) as "E-learning is an innovative web-based system based on digital technologies and other forms of educational materials whose primary goal is to provide students with a personalized, learner-centred, open, enjoyable and interactive learning environment supporting and enhancing the learning processes."
The trend of using 'e-learning' term is not new - actually, the search interest is slowly declining for the last 15 years [Figure 2].
7
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
N:;ts
1 Jon 2004	1 Sep 2008	1 May 2013	1 Jan 2018
Figure 2: Search trend of the term 'e-iearning' over time (trends.google.com)
Gamification in education (Lee and Hammer, 2011) can be used effectively to help the learners apply their learning on-the-job through real-life situations in a controlled environment. The trend of using 'gamification in education' term is following the pattern of 'gamification' search trend - it appeared around 2010 and it is stable since then [Figure 3].
1 Nov 2008	1 Feb 2012	1 May 2015	1 Aug 2018
Figure 3: Search trend of the term 'gamification in education' over time (trends.google.com)
Gamification in e-learning is a set of methods and technology, which increase motivation and engagement to use and complete a course in the e-learning environment. Various authors have already proposed models to introduce gamification into the field of e-learning, for example in higher education institutions (Urh et al., 2015).
1		
m h « J	Aa A A i l Ä./V	
i n nu	/i/l n Aa/ VW	/
1 II 11 MM	ïï M v VV	h
1 Nov 2008	1 Feb 2012	1 May2015	lAug2018
Figure 4: Search trend of the term 'gamification in e-learning' over time (trends.google.com)
The trend of using 'gamification in e-learning' term is following the pattern of 'gamification' search trend - it appeared around 2012 and it is stable since then [Figure 4]. Based on the analysis of search trends, it is safe to assume the literature review should include articles from the period 2010-2019.
With this time limitation in mind, a systematic review of literature reviews was started, which is presented in the next chapter.
a
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
2 A systematic review of literature reviews
investigation of the trends in gamification in e-learning was performed by examining academic publications in peer-reviewed journals indexed in most influential academic databases. The focus was put on already performed literature reviews. The review of literature reviews was performed in Oct. and Nov. 2019 and included up-to-date articles published until that date.
The answers to the following research questions were sought:
RQ1: "Which gamification design principles proved themselves the most useful and efficient through research?" RQ2: "Which game mechanics proved themselves the most useful and efficient through research?" RQ3: "Which factors are influenced by gamification in e-learning?"
These three questions were answered by studying the existing literature (literature reviews). Firstly, the research methodology was set.
2.1 Research methodology
A systematic review of systematic literature reviews (Smith et al., 2011) is a method for search, quality assessment and summarization of already performed systematic literature reviews on the selected topic if there more than one exists. Systematic reviews are due to a large number of research articles required to bring the current research to a single place.
its purpose is to search for, review, check the quality and analyse other systematic literature reviews to create a single-entry point to the research of the selected topic. This kind of meta-study summarizes 19 systematic literature reviews, which summarize 2631 research studies [Figure 5].
1					i Meta-study			
. " .								
19		i Systematic literature review 1					Systematic literature review N	
i 1 i								
2,631	Research article 1			Research article n			Research article n+1	
Figure 5: Schematic overview of the meta-analysis
Searching for the appropriate research is usually overwhelming given the sheer amount of research materials to check through (Smith et al., 2011). The method of searching for appropriate literature in a meta-analysis is similar to systematic literature reviews. The difference is the inclusion of the special search keywords in article title search ('systematic' OR 'literature review') and special quality criteria for the selection and filtering of the articles.
Meta studies usually use the following exclusion criteria during the selection process:
1. Semantic criteria: the literature review must summarize the research on a selected topic: gamification AND e-learning,
9
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
2.	Comprehensiveness: number of research studies included in the literature review and
3.	Impact: citations of the article, journal impact factor and altmetrics.
In our study, the first two criteria were used. The research process as presented in [Figure 6] was followed.
Figure 6: Overview of the research process
Firstly, the following databases/search engines were selected: DOAJ, Google Scholar, Elsevier - Science Direct and WOS. The search process for each search engine varied a bit, which is presented in the next chapters.
2.2 The limitations of the research
The research is limited only to databases DOAJ, Science Direct and Google Scholar. The studies, where gamification is mentioned in conjunction with HR, e-health and similar topics were omitted. Also, non-English articles were not included. It is also possible that systematic literature reviews which do not have 'systematic' or 'literature review in the title or abstract were unintentionally missed.
2.3 DOAJ - Directory of open access journals
The first database that was used for our search was DOAJ. Surprisingly, the search returned a small number of results. Search details: keywords: "systematic literature review gamification" were used, search by titles, which returned 6 results [Table 1].
Table 1: Literature reviews from DOAJ
Reference
The title
Objectives of the study / no. Summary	of
of articles included	Research/results
How is this article related to the research topic
(Antonaci, Klemke The Effects of and Specht, 2019) Gamification in Online Learning Environments: A Systematic Literature Review
Mapping of game effects and key effects on the learners.
No. of articles included: 61 from ACM Digital Library, EBSCO, SAGE journals, ScienceDirect (Elsevier), Taylor & Francis Online, Wiley Online Library, Web of Science, Google Scholar, lEEEXplore, Springer Link
Application	of
gamification to MOOCs is still young field - there is a tendency to use gamification only as external rewards.
Strongly, the effects of the gamification to e-learning.
(Silva, Rodrigues and Leal, 2019a)
Gamification	in
management education: A systematic literature review
To describe gamification main themes, search for most relevant literature. No. of articles included: 244 from Scopus and WOS
Extraction of theoretical approaches	to
gamification & design of the conceptual model.
Partly - it relates to the teaching of management and extracts basic gamification terminology.
10
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
2.4 Elsevier - ScienceDirect
Next, Elsevier's - Science Direct database was used. The first search returned 473 results; therefore, some refining of the search terms was needed. Search details:
1.	Step (keyword definition)
Keywords: "systematic literature review gamification "e-learning"
2.	Step (selection of the database) Search database: Elsevier Search by titles & abstracts Number of results: 473
3.	Step (keyword refinement)
Keywords: "systematic literature review gamification "e-learning' Number of results: 72
4.	Step (title, abstract and content analysis)
Screening the titles, removing inappropriate titles. Removed: gamification in e-Health, Number of results: 6
Remained 6 articles were summarized in the following [Table].
Table 2: Literature reviews from Elsevier
Reference
The title
Objectives of the study / no. of articles included
Summary Research/results
of How is this article related to the research topic
(Kasurinen and Knutas, 2018)
Publication trends in gamification:	A
systematic mapping study
Publication trends in
gamification, to find the
most central papers about gamification
No. of articles included: 1164 from
ACM Digital library, IEEE Xplore, Web of Science and Google Scholar
Proof of concept studies & theoretical. 17,7% of analyzed articles (gamification) are focused on e-learning or MOOCs.
Identifies six major topics in which gamification is more relevant than in others.
Discussion on the main topics that involve gamification in the learning process
(Subhash and Cudney, 2018)
Gamified Learning in Higher Education: A Systematic Review of the Literature
Systematic literature review of game-based learning	systems,
frameworks that integrate game design elements, and various implementations of gamification in higher education
No. of articles included: 41 from Academic Search Complete, ACM Digital Library, Education Full Text, ASEM Digital Collection, IEEE Xplore, PsychINFO, and Scopus.
Most used gamification elements are points, badges, leaderboards, levels. Most apparent benefits	are
performance, engagement, attitude, motivation.
Strongly - it includes gamification elements in e-learning (for higher education)
(Rodrigues et al., 2019)
Tracking e-learning through published papers: A systematic
To review the literature on e-learning
No. of articles included: 99
Main elements of e-learning
review
Weekly - it extracts the concepts from the articles using words and phrases analysis
11
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
Reference
The title
Objectives of the study / no. of articles included
Summary Research/results
of How is this article related to the research topic
(Baptista and Oliveira, 2019)
Gamification and serious games: A literature meta-analysis and integrative model
To synthesize and integrate all the earlier literature and information available gamification and games
No. of articles included: 99
on serious
Weekly - it includes synthesis	of
gamification model
Attitude, enjoyment, and usefulness are the most relevant predictors of intention to use gamification. Intention, enjoyment,	and
usefulness
are the most relevant predictors of the brand attitude	towards
gamification
After a detailed reading of the articles from [Table], the following further filtering was made. The article, which was written by from (Subhash and Cudney, 2018) includes the answers to our research questions, therefore, it is included in our final analysis shown in [Table 2].
The comprehensive and detailed literature review of (Kasurinen and Knutas, 2018) explores and analyses the fields where the gamification is used. It does not reveal and present the details we're interested in: gamification design effectiveness, game elements and motivation factors, therefore it was not included in the final summary [Table 2].
The article which was written by (Rodrigues et al., 2019) analyses the frequency of words and concepts that appear in literature reviews and also does explore research questions which are in our interest (the efficiency of various gamification mechanics), included in the final summary [Table 2].
2.5 Google Scholar
Next, the Google Scholar search engine was used. The first search returned 17,900 results; therefore, heavy refining of the search terms was needed (advanced search). Search details:
-	Keywords: "systematic literature review gamification"
-	Search database: Google Scholar.
-	Search by titles, abstracts and full text.
-	Number of results: 17,900.
Google Scholar while one of the most comprehensive search engines, returns the highest number of hits, but unfortunately, there is a lot of irrelevant articles.
After studying the search results, the following exclusion criteria were defined:
-	Not directly related to e-learning.
-	Conference article.
-	Not reported the number of analysed articles.
Finally, the refined advanced search phrase was systematic "literature review" gamification e-learning-proceedings-conference-health which returned 181 articles. After further manual semantic review, 3 relevant articles remained.
Authors (Dicheva et al., 2015) proposed the following dimensions related to the context and use game elements: Context: Type of application, Context: Education level, Context: Academic subject, Implementation, Reported results from the evaluation. In the analysis by (Dicheva et al., 2015) it is evident that the following gamification design principles received the most attention (>5 research articles): Visible status, Social engagement, Goals/challenges, Rapid feedback, Freedom of Choice, Freedom to fail, Storyline/new identities. Moreover, the same analysis reports the following game mechanisms received the most attention by researchers: points, badges, levels, leader-boards. The most researched applications of the gamification include (ordered from most frequent, descending) Blended learning
12
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
courses, Courses w/out online support, MOOCs, e-learning sites and gamification support platforms. Gamification is implemented using (descending): Plugins for LMS / platforms, Gamification Apps, Courses w/out online support, gamification related platforms used. Most articles report a positive experience with evaluation (>18 articles), Mixed/recommended (7), Positive first impression (3). Negative experience is reported by only 1 article (Dicheva et al., 2015).
Table 3: Literature reviews from Google Scholar
Reference
The title
Objectives of the study / no. of articles included
Summary Research/results
of How is this article related to the research topic
(Dicheva et al., Gamification	in
2015)	education: A systematic
mapping study
To study the effects of using game elements in specific educational contexts
No. of articles included: 34
The categories include gamification design principles, game mechanics, the context of applying gamification (the type of
application, educational level, and academic subject), implementation, and evaluation. More substantial empirical research is needed to determine whether the both extrinsic and intrinsic motivation of the learners can be influenced	by
gamification
Somehow - it encompasses broader field -education, not only e-learning
(Looyestyn et al., 2017)
Does gamification increase engagement with online programs? A systematic review
To answer the question: "Are	gamification
strategies effective in increasing engagement in online programs?" No. of articles included: 15
12 of 15 studies found out the positive effect of gamification in online programmes
Somehow - it studies the effectiveness of gamification.
(Nortvig, Petersen A Literature Review of the To answer the question: Most contributing Not	directly
and Balle, 2018) Factors Influencing E- Which factors are found to factors:	educator mentioning
Learning and Blended influence e-learning and presence in online gamification, but Learning in Relation to blended learning in relation settings, interactions factors that affect Learning Outcome, to learning outcome, between students, online learning, Student Satisfaction and student satisfaction and teachers and content, which are similar to Engagement	engagement	in and	designed gamification & e-
collaboration in higher connections between learning. education and particularly online and offline in professional education? activities as well as No. of articles included: between campus-related 44	and practice-related
activities
The literature review of (Looyestyn et al., 2017) includes various contexts of gamification usage. 5 out of 15 studied articles explore the education context. This study explores 'the gamification features used' and it does not distinguish between gamification design and gamification elements/mechanisms/mechanics. Therefore 'gamification features' were regrouped into 2 groups. Most studies included the following gamification design principles: Goals, quests, challenges, social comparison and gamification mechanics: PBL, leader board, theme, reward, progress, avatar, bets, quizzes, achievements, feedback.
13
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
The next promising article, which was written by (Nortvig, Petersen and Balle, 2018) mentions factors such s learning outcome and student satisfaction, therefore it will be included into the final selection of articles.
2.6 WOS (Web of Science)
Lastly, the WOS search engine was used. We have searched for systematic literature review on gamification, 60 potential papers were surfaced out of which 7 were chosen that fit the purpose of the paper. The most relevant findings are given below in [Table ]:
Table 4: Literature reviews from WOS
Reference
The title
Objectives of the study / no. of articles included
Summary Research/results
of How is this article related to the research topic
(Mora et al., 2017)
Gamification:	a
systematic review of design frameworks
The importance of choosing a game design to help students surpass learning difficulties 40 papers out of 2314 were used
Effectiveness is rather difficult to attain because of verified game results
It stresses effectiveness gamification methods.
the of
(Bozkurt and A systematic review of Durak, 2018)	research: In pursuit of
homo ludens
Extensive research on patterns in gamification that relate to education 208 papers from Scopus, Eric, Web of Science, and Google Scholar)
Descriptive papers are prevalent; quantitative and researches follow; the new trends look at motivations, attitudes and behaviors in education
An	in-depth
gamification
mapping.
(Gundry and Deterding, 2019)
Validity Threats in Quantitative Data Collection with Games: A Narrative Survey
Discusses potential threats in gaming
Games are complex systems, that can impede predictable control, they can interact with non-gaming situations
Validity threats in data collection.
(Ortiz, Chiluiza and Valcke, 2016)
Gamification in higher education and stem: a systematic review of literature
Researches empirical studies about gamification STEM-related Higher Education
30 papers included
from WOS
How a combination of badges, leaderboards improves students' attendance	and
performance
Students' stimulus of goal orientation.
(Morschheuser al., 2017)
et Gamified crowdsourcing: Conceptualization, literature review, and future agenda
Conceptualization
gamification
crowdsourcing
of and
110 articles from Scopus
Demonstrate effectiveness gamification crowdsourcing
the It is strongly related of to the purpose of the in paper.
There are also other significant articles that deal with creating tools for programming teaching (Borges et al., 2018) or with motivational experiences in gamification (Rengifo et al., 2017). Another stream of literature questions the manner gamification is implemented in MOOCs (Ortega-Arranz et al., 2017) or endeavour to build a model of gamification in learning (Nazififard et al., 2019).
To be noted that most articles and papers published on WOS deal with the application of gamification in specific areas, i.e. informatics, engineering, medical sciences and care, advertising, nutrition, digital technologies, etc. Papers also address this topic specifically for different levels of schooling and education.
14
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
2.7 Miscellaneous articles - literature reviews
Affordance signalling performance is visible in articles like the one published by (Majuri, Koivisto and Hamari, 2018)) and accessed on ResearchGate. Using 128 articles collected from DBLP Computer Science Bibliography, and the AIS Electronic Library, ACM, IEEE, Springer, the authors conclude that gamification in education most often employs affordability and progression but immersion- oriented affordance is seldom used.
In the same line, (Hamari, Koivisto and Sarsa, 2014) using 24 articles from multiple databases (i.e. Scopus, Science Direct, EBSCO Host, Web of Science, ACM Digital library, AISel, Google Scholar, and ProQuest) show that gamification proves to be effective depending on circumstances and affordability.
(Junior et al., 2019) using data from WOS and Scopus demonstrates that in engineering education, gamification, when mainly corroborated with PBL enhances teaching skills.
(Kocakoyun and Ozdamli, 2018) using 313 studies conclude that gamification is mostly preferred in virtual environments, but the usage of gamification is not limited to technology and thus the method can be extended to all learning domains.
3 Gamification mechanisms in the reviewed literature
Within this chapter, the selection of the most promising articles in relation to our research questions are presented. For each article, the relation to our research questions is shown (columns 2,3,4) in [Table 2]. "-" char is used where the article does not answer to our research questions directly.
Table 2: How does the identified literature answer our research questions?
Reference	RQ1: "Which gamification design
principles proved themselves the most useful and efficient through research?
RQ2: "Which game	RQ3: "Which factors are mechanics/elements proved influenced by gamification in e-themselves the most useful and learning?" efficient through research?"
(Antonaci, -	Badges, Leaderboards, Points,	Performance, motivation,
Klemke and	Feedback, Challenges, Likes &	engagement, attitude towards
Specht, 2019)	social features, Communication	gamification, collaboration,
	channels, Narratives, Levels,	social awareness
	Progress bars, Teams, Agent,	
	Medals, Avatar, Trophies, Time	
	limit, Task, Virtual currency,	
	Personalizing features, Mission,	
	Replayability, Goal indicator,	
	Competition, Win state	
(Silva, Rodrigues -	-	Motivation, attitudes, flow,
and Leal, 2019b)		perceived learning.
(Subhash and Collaboration, Feedback, Freedom Cudney, 2018) to fail, Graphics, Narratives, Design (goals, rules, time-limit, clues, competition), real rewards, role play
Badges, Leaderboard, Levels points,
Attitude,	engagement,
enjoyment, competition, motivation, perceived learning, practical skills, retention, satisfaction,	student
performance
(Dicheva et al., Goals, challenges and quests, PBL, progress bars, levels, virtual 2015)	customization, progress, feedback, goods, currency, avatars, a
competition and cooperation, countdown clock accrual grading, access/unlocking content, freedom of choice,
15
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
Reference	RQ1: "Which gamification design
principles proved themselves the most useful and efficient through research?
RQ2: "Which game mechanics/elements proved themselves the most useful and efficient through research?"
RQ3: "Which factors are influenced by gamification in e-learning?"
freedom to fail, storytelling, new identities, onboarding, time restriction
(Looyestyn et al., 2017)
Goals, quests, comparison.
challenges, social
PBL, leaderboard, theme, reward, progress, avatar, bets, quizzes, achievements, feedback
Engagement (time spent, occasions visited, volume)
Educator presence in online settings, interactions between students, teachers and content, and designed connections between online and offline activities as well as between campus-related and practice-related activities
(Mora et al., -	According to the method and	Learning and attitudes
2017)	game	
(Ortiz, Chiluiza Goal orientation	Reward architecture: Badges,	Attendance and performance
and Valcke,	leaderboard	
2016)		
(Bozkurt and -	According to mage patterns and	Attitudes towards learning,
Durak, 2018)	environment	behaviours in education
(Kasurinen and -	Specific to domain	Learning
Knutas, 2018)		
(Subhash and -	According to game design	Improving learning
Cudney, 2018)	elements	performance in higher
		education
4 Discussion		
The number and the quality of found articles show that the field of gamification in various fields is increasing and the research field is maturing. Surprisingly, there is not a lot of studies - literature reviews related specifically to the application of gamification in e-learning, which is one of the 'natural' fields where gamification can be implemented.
During the review of the articles, the following issue was detected. The terminology related to the gamification is not consistent and does not encompass the same concepts. Within the articles, different terminology is used, such as Game elements, gamification constructs, gamification design, gamification mechanisms, game mechanisms, game design elements. We have classified them into two groups [Table 2]: gamification design principles and game mechanic/elements. The first group is related to conceptual course design and the second group is related to technical course design.
Studies about gamification affected factors show more consistency: the majority of them revolve around motivation, (learning) performance, attitude, engagement and social interactions. These are the primary reasons for the implementation of gamification concepts in the first place. Almost all studies showed that gamification design/game elements positively affect mentioned factors.
The most frequently researched concepts within the studied literature reviews are still PBL (points, badges, leaderboards/levels). This is probably due because these are the most simplistic and easy to implement game mechanic principles.
(Nortvig, Petersen and Balle, 2018)
16
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
What is even more interesting, the literature reviews which have been studied, are very recent - most of them are dated in 2017-2019. This is somehow understandable because the first studies, which are included in mentioned literature reviews emerged shortly after the year 2010.
4.1 Proposals for the e-learning course designers & teachers
Based on these findings and also practical experiences of the authors with the gamification of managerial online courses, the following proposals for e-learning course developers were prepared.
Proposal 1: When studying how to implement gamification in your e-learning environment and process, study the practitioner's resources such as (Marczewski, 2013) alongside the academic articles.
During our research of existing work, we have found out that practitioners and companies who already implemented gamification into the e-learning, classified and conceptualised the related terms and mechanisms in a more understandable and practical way than academic resources.
Proposal 2: When implementing gamification in your e-learning course, look beyond PBL, for example: Firstly, think about narrative and learner's experience process, set achievable goals, set the difficulty of the course in a way the learner falls into the 'flow' and incorporate 85% rule defined by (Wilson et al., 2019).
Some gamification principles (such as story or narrative) are difficult or impossible to implement within the LMS's functionalities or plugins, therefore think of them in the course design time.
Proposal 3: Before implementing gamification in your course, study, what do you want to achieve with it.
Is it a low drop-out rate? High satisfaction? High learning outcomes? Some factors are related, but some are opposite. Sometimes high satisfaction with the course is in contradiction with high learning yield. For example, a complex math course which requires high participant input and effort might not give a high 'recommend to a friend' score.
5 Conclusion
Within our research, we were interested in the state of the art in the field of gamification in e-learning. The main reason is that e-learning is gaining popularity, it is broadly available, affordable, but the participant drop-out and low motivation to complete still remains the issue.
Therefore, a meta-study was conducted using the latest literature reviews from various academic databases and search engines.
The final result is that only a handful of literature reviews specialises in gamification in e-learning, even less explore all its relevant fields: gamification design, game mechanics and motivational factors. Nevertheless, almost all studies report positive effects of the gamification mechanisms. Lastly, we have condensed all research findings into some proposals for the e-learning instructors and course designers.
We are aware of the limitations of this study (the possibility of missed relevant literature reviews or wrong interpretation).
The future research will point in the way of practical implementation of the researched gamification mechanisms, which is the only way to test their effectiveness in a real-life setting.
Acknowledgement
The article was co-financed by the project "Teaching and Learning for Sustainability - TeachSUS", Erasmus+ Programme -Strategic Partnership, Project No: 2018-1RO01-KA204-049253 (https://www.teachsus.eu/).
17
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
References
1.	Antonaci, A., Klemke, R. and Specht, M. (2019) 'The Effects of Gamification in Online Learning Environments: A Systematic Literature Review', Informatics. Multidisciplinary Digital Publishing Institute, 6(3), p. 32. doi: 10.3390/informatics6030032.
2.	Aseriskis, D. and Damasevicius, R. (2014) 'Gamification of a Project Management System', The Seventh International Conference on Advances in Computer-Human Interactions Gamification, (c), pp. 200-207. doi: 10.1145/2181037.2181040.
3.	Bajdor, P. (2011) 'the Gamification As a Tool To Improve Risk Management', Annales Universitatis Apulensis Series Oeconomica, 13(2), pp. 574-583. Available at: http://oeconomica.uab.ro/upload/lucrari/1320112/38.pdf.
4.	Baptista, G. and Oliveira, T. (2019) 'Gamification and serious games: A literature meta-analysis and integrative model', Computers in Human Behavior, 92, pp. 306-315. doi: https://doi.org/10.1016/j.chb.2018.11.030.
5.	Bellotti, F. et al. (2013) 'A gamified short course for promoting entrepreneurship among ICT engineering students', Proceedings - 2013 IEEE 13th International Conference on Advanced Learning Technologies, ICALT 2013, pp. 31-32. doi: 10.1109/ICALT.2013.14.
6.	Borges, R. P. et al. (2018) 'A systematic review of literature on methodologies, practices, and tools for programming teaching', IEEE Latin America Transactions. IEEE Computer Society, 16(5), pp. 1468-1475. doi: 10.1109/TLA.2018.8408443.
7.	Bozkurt, A. and Durak, G. (2018) 'A systematic review of gamification research: In pursuit of homo ludens', International Journal of Game-Based Learning, 8(3), pp. 15-33. doi: 10.4018/IJGBL.2018070102.
8.	Burke, B. (2014) Gartner Redefines Gamification, Gartner Blog Network. Available at: http://blogs.gartner.com/brian_burke/2014/04/04/gartner-redefines-gamification/.
9.	Burke, B. B. (2013) 'The Gamification of Business', Forbes, pp. 2013-2015. Available at: http://www.forbes.com/sites/gartnergroup/2013/01/21/the-gamification-of-business/.
10.	Dale, S. (2014) 'Gamification', Business Information Review, 31(2), pp. 82-90. doi: 10.1177/0266382114538350.
11.	Deterding, S. et al. (2011) 'From Game Design Elements to Gamefulness : Defining " Gamification "', in Envisioning Future Media Environments, pp. 9-11. doi: 10.1145/2181037.2181040.
12.	Deterding, S. and Dixon, D. (2011) 'Gamification : Using Game Design Elements in Non-Gaming Contexts', CHI2011: Conference on Human Factors in Computing Systems, pp. 5-8. doi: 10.1145/1979742.1979575.
13.	Dicheva, D. et al. (2015) 'Gamification in Education: A Systematic Mapping Study Darina', Educational Technology and Society, 18(3),	pp.	3-20.	Available	at: http://www.academia.edu/download/39683102/Gamification_in_Education_A_Systematic_M20151104-8546-1cs0c99.pdf (Accessed: 13 November 2019).
14.	Groh, F. (2012) 'Gamification: State of the Art Definition and Utilization', Research Trends in Media Informatics, pp. 39-46. doi: 10.1145/1979742.1979575.
15.	Gundry, D. and Deterding, S. (2019) 'Validity Threats in Quantitative Data Collection With Games: A Narrative Survey', Simulation & Gaming, 50(3), pp. 302-328. doi: 10.1177/1046878118805515.
16.	Hamari, J., Koivisto, J. and Sarsa, H. (2014) 'Does gamification work?--a literature review of empirical studies on gamification', System Sciences (HICSS), 2014 47th Hawaii International Conference on, pp. 3025-3034. doi: 10.1109/HICSS.2014.377.
17.	Huotari, K. (2012) 'Defining Gamification - A Service Marketing Perspective', Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, pp. 17-22. doi: 10.1145/2393132.2393137.
18.	Huotari, K. and Hamari, J. (2012) 'Defining gamification', Proceeding of the 16th International Academic MindTrek Conference on - MindTrek '12, p. 17. doi: 10.1145/2393132.2393137.
19.	Junior, E. et al. (2019) 'Systematic literature review of Gamification and Game-based Learning in the context of Problem and Project Based Learning approaches', in 11th International Symposium on Project Approaches in Engineering Education (PAEE) 16th Active Learning in Engineering Education Workshop (ALE). Hammamet, Tunisia.
20.	Kappen, D. L. and Nacke, L. E. (2013) 'The kaleidoscope of effective gamification: deconstructing gamification in business applications', Proceedings of the First International Conference on Gameful Design, Research, and Applications - Gamification '13, pp. 119-122. doi: 10.1145/2583008.2583029.
21.	Kasurinen, J. and Knutas, A. (2018) 'Publication trends in gamification: A systematic mapping study', Computer Science Review. Elsevier, 27, pp. 33-44. doi: 10.1016/J.C0SREV.2017.10.003.
22.	Kim, B. (2015) 'Understanding Gamification', Library Technology Reports, 51(2), p. 37. doi: http://dx.doi.org/10.5860/ltr.51n2.
23.	Kocakoyun, S. and Ozdamli, F. (2018) 'A Review of Research on Gamification Approach in Education', in Socialization - A Multidimensional Perspective. doi: 10.5772/intechopen.74131.
24.	Koivisto, J. and Hamari, J. (2014) 'Demographic differences in perceived benefits from gamification', Computers in Human Behavior, 35, pp. 179-188. doi: 10.1016/j.chb.2014.03.007.
25.	Lee, J. J. J. and Hammer, J. (2011) 'Gamification in Education: What , How , Why Bother?', Academic Exchange Quarterly, 15(2), pp. 1-5. Available at: http://dialnet.unirioja.es/servlet/articulo?codigo=3714308.
18
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2
Rozman, T., Donath, L. (2019). The Current State of the Gamification in E-Learning: A literature Review of Literature Reviews
26.	Looyestyn, J. et al. (2017) 'Does gamification increase engagement with online programs? A systematic review', PLoS ONE. Edited by F. Amblard, 12(3), p. e0173403. doi: 10.1371/journal.pone.0173403.
27.	Majuri, J., Koivisto, J. and Hamari, J. (2018) 'Gamification of education and learning: A review of empirical literature', in CEUR Workshop Proceedings, pp. 11-19. Available at: http://ceur-ws.org/Vol-2186/paper2.pdf. (Accessed: 19 November 2019).
28.	Mapplinks (2017) How To Use Gamification In Growth Hacking? Available at: http://mapplinks.com/use-gamification-growth-hacking/ (Accessed: 25 January 2018).
29.	Marczewski, A. (2013) Game Mechanics in Gamification. Available at: https://www.gamified.uk/2013/01/14/game-mechanics-in-gamification/ (Accessed: 20 November 2019).
30.	Mora, A. et al. (2017) 'Gamification: a systematic review of design frameworks', Journal of Computing in Higher Education. Springer New York LLC, 29(3), pp. 516-548. doi: 10.1007/s12528-017-9150-4.
31.	Morschheuser, B. et al. (2017) 'Gamified Crowdsourcing: Conceptualization, Literature Review, and Future Agenda', International Journal of Human-Computer Studies, 106, p. 43. doi: 10.1016/j.ijhcs.2017.04.005.
32.	Morschheuser, B., Hamari, J. and Koivisto, J. (2016) 'Gamification in crowdsourcing: A review', in Proceedings of the Annual Hawaii International Conference on System Sciences, pp. 4375-4384. doi: 10.1109/HICSS.2016.543.
33.	Nazififard, S. et al. (2019) 'A Model for Utilizing the Potential of Gamification in Learning', in 2019 IEEE 5th Conference on Knowledge Based Engineering and Innovation, KBEI2019, pp. 874-878. doi: 10.1109/KBEI.2019.8735001.
34.	Newhouse, K. (2014) Introducing the Growth Hacking Strategy of 'GAMIFICATION'... The HOTTEST Dental Web Marketing Strategy for 2014 and Beyond! - Get Clients Now - Dentaltown. Available at: http://www.dentaltown.com/blog/post/1129/introducing-the-growth-hacking-strategy-of-gamification-the-hottest-dental-web-marketing-strategy-for-2014-and-beyond#Comment (Accessed: 25 January 2018).
35.	Nortvig, A. M., Petersen, A. K. and Balle, S. H. (2018) 'A literature review of the factors influencing e-learning and blended learning in relation to learning outcome, student satisfaction and engagement', Electronic Journal of e-Learning, 16(1 ), pp. 4555. Available at: www.ejel.org (Accessed: 14 November 2019).
36.	Orta, E. and Ruiz, M. (2012) 'A Simulation and Gamification Approach for IT Service Management Improvement', Communications in Computer and Information Science, 290, pp. 295-301. doi: 10.1007/978-3-319-38980-6_7.
37.	Ortega-Arranz, A. et al. (2017) 'How gamification is being implemented in MOOCs? A systematic literature review', in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). doi: 10.1007/978-3-319-66610-5_40.
38.	Ortiz, M., Chiluiza, K. and Valcke, M. (2016) 'Gamification in higher education and stem: A systematic review of literature', in EDULEARN16Proceedings, pp. 6548-6558. doi: 10.21125/edulearn.2016.0422.
39.	Prince, J. et al. (2013) 'Gamification', Business & Information Systems Engineering, 5(4), pp. 275-278. doi: 10.1007/s12599-013-0273-5.
40.	Prince, J. D. (2013) 'Gamification', Journal of Electronic Resources in Medical Libraries, 10(3), pp. 162-169. doi: 10.1080/15424065.2013.820539.
41.	Raymer, R. (2011) 'Gamification: Using Game Mechanics to Enhance eLearning', eLearn, 2011(9), p. 3. doi: 10.1145/2025356.2031772.
42.	Rengifo, P. et al. (2017) 'Motivational experiences of gamifications: a systematic literature review', INNOVACION EDUCATIVA-MEXICO, 17(75), pp. 63-80.
43.	Rodrigues, H. et al. (2019) 'Tracking e-learning through published papers: A systematic review', Computers & Education, 136, pp. 87-98. doi: https://doi.org/10.1016Zj.compedu.2019.03.007.
44.	Silva, R. J. R. da, Rodrigues, R. G. and Leal, C. T. P. (2019a) 'Gamification in Management Education: A Systematic Literature Review', BAR - Brazilian Administration Review. Associaçao Nacional de Pos-Graduaçao e Pesquisa em Administraçao (ANPAD), 16(2), p. e180103. doi: 10.1590/1807-7692bar2019180103.
45.	Silva, R. J. R. da, Rodrigues, R. G. and Leal, C. T. P. (2019b) Gamification in Management Education: A Systematic Literature Review, BAR - Brazilian Administration Review. doi: 10.1590/1807-7692bar2019180103.
46.	Simöes, J., Redondo, R. D. and Vilas, A. F. (2013) 'A social gamification framework for a K-6 learning platform', Computers in Human Behavior, 29(2), pp. 345-353. doi: 10.1016/j.chb.2012.06.007.
47.	Smith, V. et al. (2011) 'Methodology in conducting a systematic review of systematic reviews of healthcare interventions', BMC Medical Research Methodology. BioMed Central, 11(1), p. 15. doi: 10.1186/1471 -2288-11-15.
48.	Subhash, S. and Cudney, E. A. (2018) 'Gamified learning in higher education: A systematic review of the literature', Computers in Human Behavior, 87, pp. 192-206. doi: https://doi.org/10.1016/j.chb.2018.05.028.
49.	Urh, M. et al. (2015) 'The Model for Introduction of Gamification into E-learning in Higher Education', Procedia - Social and Behavioral Sciences, 197, pp. 388-397. doi: https://doi.org/10.1016/j.sbspro.2015.07.154.
50.	Wilson, R. C. et al. (2019) 'The Eighty Five Percent Rule for optimal learning', Nature Communications, 10(1), p. 4646. doi: 10.1038/s41467-019-12552-4.
19
Mednarodno inovativno poslovanje = Journal of Innovative Business and Management 11(3), 5-19, DOi: 10.32015/JIBM/2019-11 -3-2