https://doi.org/10.31449/inf.v47i4.3401 Informatica 47 (2023) 487–500 487 A Framework for Evaluating Distance Learning of Environmental Science in Higher Education Using Multi-Criteria Group Decision Making Katerina Kabassi Department of Environment, Ionian University, Minotou Giannopoulou 26, Panagoula, 29100 Zakynthos, Greece E-mail: kkabassi@ionio.gr Keywords: e-Learning, environmental science, evaluation criteria, Fuzzy AHP Received: January 2, 2021 Purpose: Due to Covid-19, big changes took place in Universities around the world. Universities were asked in March 2020 within a short while to provide the whole of the available lessons using e-learning methods. Since the health crisis continues, e-learning has expanded on a variety of contexts and simultaneously has created an urgent need for designing, developing, and implementing a valid and reliable distance learning procedure. The validity and efficiency of the aforementioned procedure is a major issue that has to be tested at the end of the semester. Therefore, developing a valid framework to evaluate the e-learning process has become more important now than in the past due to the ongoing pandemic.Design/methodology/approach: The evaluation of the educational process is a multi-criteria problem that is based on the points of view of both instructors and students. In order to solve this multi- criteria problem of e-learning evaluation, a new framework for evaluating e-learning in Higher Education has been developed. This framework uses group decision-making with multiple criteria and is called ENVEL. This paper defines the set of evaluation criteria and uses the Fuzzy Analytic Hierarchy Process to prioritize criteria and help the decision-makers draw conclusions on the effectiveness and success of e-learning. Findings: The framework takes into account heterogeneous groups of students and professors, makes different calculations for these groups, and can extract useful conclusions by comparing the results of the different groups. The framework has been applied in the Department of Environmental Science at the Ionian University and conclusions have been made on its effectiveness and usage. Originality: Trying to focus on the evaluation of e-learning in a whole study program in Higher Education, and not only on single courses, the paper describes a novel framework for e-learning evaluation using multi-criteria decision-making with heterogeneous groups of users. This framework provides a formal way of combining different aspects of the evaluation of e-learning and collecting summative results. Povzetek: Raziskava uvaja okvir ENVEL za ocenjevanje e-učenja v visokem šolstvu z uporabo večkriterijskega skupinskega odločanja in metode Fuzzy AHP. 1 Introduction E-Learning has garnered increasing attention in Higher Education in the last few decades (Martín-Lara & Rico 2020, Njenga 2017, Otto & Becker 2018, Schieffer 2016). Several case studies for the application of e- learning in higher education have been reported (e.g. Sulčič & Lesjak 2009, Al‐Fadhli & Khalfan 2009, Bhadauria 2016; Sheikh & Gupta 2018). However, a lack of usage at the university level was clear (Mercader & Gairin 2020). Indeed, before the COVID-19 pandemic, e- learning was growing by approximately 15.4% yearly in educational institutions around the world without pressure on teachers, students, or institutions (Alqahtani & Rajkhan 2020). Since the health crisis continues, e- learning has expanded on a variety of contexts and simultaneously has created an urgent need for designing, developing, and implementing a valid and reliable distance learning procedure. The validity and efficiency of the aforementioned procedure is a major issue that has to be tested at the end of the semester. However, as Stöhr et al. (2020) report, previous studies have mainly focused on asynchronous online learning, rather than synchronous or mixed modes of online learning (Hrastinski 2008, Siemens et al. 2015). Furthermore, as Barteit et al. (2020) point out, the effectiveness of e-learning was mainly evaluated by comparing e-learning to other learning approaches such as traditional teaching methods or evaluating students' and teachers’ attitudes (Frehywot et al. 2013). Several systematic reviews and meta-studies on the effectiveness of e-Learning on single courses have been conducted (Wu & Hwang 2010, Noesgaard & Ørngreen 2015, Abdalla 2007, Liaw 2008, Haverila & Barkhi2009), but there is a lack of similar experiments that would evaluate e-learning adoption in a whole study program. Some of the studies on e-learning system evaluation focused on the technology-based components 488 Informatica 47 (2023) 487–500 K. Kabassi (Islas et al. 2007), others focused on the human factors (Liaw et al. 2007), and others are meta-reviews (Salter et al. 2014). Taking into account that most reports on e-learning mainly focused on evaluating single courses and not a whole study program and the fact that the effective adoption of e-learning can only be confirmed by evaluating the educational process, we notice that there is a great need for a tool that would evaluate e-learning adoption of a whole study program and not a single course. The effectiveness of e-learning depends on several factors and criteria (Harrati et al. 2016, Abuhlfaia & Quincey 2019, Alqahtani & Rajkhan 2020) and Jeong & González-Gómez (2020) highlight the necessity of determining those. As the evaluation of e-learning is affected by several factors and criteria that try to combine the points of view of different decision-makers, multi-criteria group decision-making may be found effective for designing a formal framework for e-learning evaluation. Indeed, MCDM has been used in the past for evaluating e- learning systems and applications (Mahdavi et al. 2008, Stecyk 2019, Çelikbilek & Adıgüzel Tüylü 2019, Alqahtani & Rajkhan 2020, Jeong & Gonzalez-Gomez 2020). However, these approaches did not focus on evaluating the e-learning of a whole study program. Therefore, this paper focuses on presenting a framework for evaluating e-learning of a study program in Higher education that is called ENVEL. Its name originates from the first application of the framework in the Department of Environment (ENVironment E- Learning evaluation). The framework defines the groups of decision-makers, the set of criteria, and the weights of their importance in the reasoning of the decision-makers while evaluating e-learning. The framework considers the instructors and students participating in the educational process as decision-makers and provides a formal way of combining different aspects of the evaluation of e-learning using Multi-Criteria Decision Making (MCDM) and collecting summative results. MCDM has evolved rapidly over the last decades (Zopounidis 2009) and different decision approaches have been proposed. These approaches differ in the way the objectives and alternative weights are determined (Mohamadali & Garibaldi 2011). The Analytic Hierarchy Process (Saaty 1980) is one of the most popular MCDM theories and has been used before for combining criteria for e-learning success, but for single courses or systems (Anis & Islam 2015, Vinogradova & Kliubas 2015, Jasim et al. 2018, Alqahtani & Rajkhan 2020). The AHP is chosen amongst other MCDM theories because it presents a formal way of quantifying the qualitative criteria of the alternatives, in this way removing the subjectivity of the result (Tiwari 2006). As Erensal et al. (2006) point out, the conventional AHP may not fully reflect a style of human thinking as users usually feel more confident in giving interval judgments rather than expressing their judgments in the form of single numeric values. The theory’s combination with the fuzzy theory resulted in Fuzzy AHP (FAHP) (Buckley 1985), which in comparison with other MCDM methods is considered by many researchers (e.g. Ramanayaka et al. 2019) as a more effective solution to solve MCDM related problems because of its powerful ability to deal with imprecise and uncertain data. Furthermore, the method’s ability to make decisions by making a pairwise comparison of uncertain, qualitative, and quantitative factors and also its ability to model expert opinion (Mulubrhan et al. 2014) is another important reason for its selection against other alternatives. As a result, FAHP has been used before for combining and prioritizing criteria in e-learning systems’ evaluation (Tai et al. 2011, Anggrainingsih et al. 2018, Lin 2010, Altun Turker et al. 2019, Naveed et al. 2020). Given the above advantages of FAHP, ENVEL uses the particular theory to prioritize criteria. The framework has been applied in the Department of Environmental Science at Ionian University for evaluating the e-learning conducted in the special circumstances that occurred during the spring semester of 2019-2020 due to the Coronavirus emergency. 14 professors and 98 students of the Department that took part in the e-learning participated in the evaluation experiment. The paper is organized as follows: Sections 2, 3, and 4 describe the framework ENVEL. More specifically, section 2 focuses on the criteria used in the evaluation process, section 3 on the prioritization of the criteria, and section 4 on the evaluation of the e-learning aspects. Section 5 presents how the Department of Environment at Ionian University turned to e-Learning during the spring semester of 2019-2020 and section 6 describes a case study, which involves the application of ENVEL in the specific department for the evaluation of the whole study program provided by e-learning methods. Section 7 includes a discussion of the results of the evaluation conducted using ENVEL and proposals that could improve the whole e-learning process. Finally, in the last section, the conclusions regarding the ENVEL framework are presented. 2 ENVEL: Defining criteria for e- learning evaluation Different MCDM theories and criteria have been used for evaluating e-learning systems. The most common approaches to evaluations of e-learning systems that use MCDM are presented in Table 1. Most of these frameworks use two levels of criteria and a combination of two MCDM models. However, the criteria used in these approaches mainly concern the technology used and the way that courses are designed for e-learning. Furthermore, these frameworks mainly focus on the A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 489 Table 1: The questions of the questionnaire and their connection with the criteria of ENVEL Mahdavi et al. 2008 Stecyk 2019 Çelikbilek & Adıgüzel Tüylü 2019 Alqahtani & Rajkhan 2020 Jeong & Gonzalez- Gomez 2020 Levels of criteria 2 levels 1 level 2 levels 1 level 2 levels No of criteria 1 st level – 4 criteria 2 nd level – 13 criteria 10 criteria 1 st level – 3 criteria 2 nd level – 19 criteria 10 criteria 1 st level – 4 criteria 2 nd level – 16 criteria What is evaluated Web-based E- Learning Systems e- learning course Components of e- learning systems e-learning approaches e-learning systems MCDM models AHP, Entropy and TOPSIS Approach PROME THEE II fuzzy F-DEMATEL analytic network process AHP&TOP SIS F-DEMATEL/MCDA method Table 2: The questions of the questionnaire and their connection with the criteria of ENVEL C1Functionality of the system. c11: Accessibility How easy was the access to the e-learning platform? c12: Response time The response/upload time was… c13: Reliability Was the system reliable (e.g. did you face connection problems, data loss, etc.)? c14: Easy to use/simplicity How simple was the usage of the system? c15: e-Learning Management The management of the educational material was… C2Quality of communication. c21: Quality of Synchronous Communication student-instructor The quality of Synchronous Communication student- instructor was… c22: Quality of Synchronous collaboration between students The tools for students’ collaboration were… c23:Students’ participation How was the students’ participation in comparison to the lessons in the classroom? The active participation of the students in the e- learning lessons was… C3 e-LearningReadiness. c31: e-LearningCulture Did you have previous experience in e-learning? In the past, were you in favor of e-learning? c32: e-Learning Support The quality of the technical support provided by the department was… c33: e-Learning Infrastructure The tools for synchronous and asynchronous e-learning provided by the department were… C4Quality of e-Learning. c41: Effectiveness How would you judge the effectiveness of e-learning? c42: Acceptability How do you judge the experience in e-learning during this semester? Would you like to continue e-learning after the end of the COVID-19 era? c43: Course design The design of synchronous and asynchronous lessons was… 490 Informatica 47 (2023) 487–500 K. Kabassi evaluation of one or two courses and not the evaluation of a whole study program. Criteria are also used in the cases where no MCDM model is applied. To determine the effectiveness of learner-controlled e-learning, research typically distinguishes between learning processes and learning outcomes (Alavi & Leidner 2001, Gupta & Bostrom 2009). However, Martinez-Caro (2018) argues that there is an absence of a solid and objective measure of learning, and concludes that perceived learning is positively related to satisfaction in e-learning courses. This point of view is in line with Richmond et al. (1987) who proposed the use of a subjective measure: the students’ perceived learning, which refers to the extent to which a student believes s/he has acquired specific skills. As a result, the best way of evaluating e-learning is to focus on the students’ perceived self-efficacy and perceived satisfaction (Liaw 2008). Another factor that is considered important while evaluating e-learning involves teacher-student interaction (Su et al. 2005, Harasim et al. 1995, Hartman et al. 1995, Elkaseh et al. 2016). Such interaction encourages learners to understand the content better (Su et al. 2005) and students who are shy or uncomfortable about participating in class discussions may prefer participation in online forums (Owston 1997). Additionally, the way the e-learning environment promotes collaboration and generally the interaction between students is of exceptional importance (Benbunan-Fich & Hiltz 2003, Arbaugh 2004) and considered as a critical component of quality education (Anderson 2001). In some cases, students have even expressed a preference for online dialogue over traditional classroom discussion (Clark 2001, Martinez-Caro 2011). A new way of interaction and learning, which requires a high level of learner control may result in students having negative attitudes or experiencing difficulties (Chou & Liu 2005). Arbaugh’s (2008) study confirms that students’ prior experience with e-learning can positively affect their implementation. This is in line with the work of Marks et al. (2005), which suggests that students with experience in e-learning courses may perform better in other e-learning courses. As a result, the prior experience may influence the current e-learning experience. Therefore, both of these criteria are to be taken into account while evaluating the e-Learning experience. Other factors that have been reported to be taken into account in studies that measure the students’ perceived satisfaction in e-learning settings involve service quality, system quality, content quality or e-learning quality, learner perspective/attractiveness, instructor attitude, supportive issues, etc. (Aguti et al. 2014, Reiser 2001, Tseng et al. 2011, Salter et al. 2014). Some researchers claim that the effectiveness of e- learning depends on demographic factors such as age, gender, etc. (Islam et al. 2011), while others argue against these hypotheses (Marks et al. 2005, Martinez- Caro 2018). Therefore, such criteria were not taken into account at all. Given the above, the criteria used within the ENVEL framework for evaluating the e-Learning process are the following: C1: Functionality of the system. In this category, all criteria are related to the quality of the system. c11: Accessibility. The system makes learning materials easily accessible. c12: Response time. The waiting time for loading learning materials is reasonable. c13: Reliability. The e-Learning system provides the right solution to learner requests. c14: Ease of use/simplicity. The user interface should be simple and easy to use. c15: e-Learning Management. The easiness in designing the e-Learning Process. Figure 1: Steps of the ENVEL. C2: Learner Attractiveness. All the criteria are related to the Learner’s Attractiveness. c21: Quality of Synchronous Communication with the instructor. Quality of synchronous communication with the instructor. c22: Quality of Synchronous Communication with students. Quality of synchronous communication among students. c23:Students’ participation. Quality of students’ participation in the distance lesson. A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 491 C3 e-Learning Readiness. In this category, all criteria are related to the way the instructors integrated e- learning. c3.1: e-learning Culture. Beliefs and attitudes towards e-learning. c3.2: e-Learning Support. Staff mentoring and support in providing e-learning. c3.3: e-Learning Infrastructure. Tools provided (recording, blackboard, scheduling, etc.). Especially in the case of a department with laboratories and activities on the field, e-Learning Infrastructure may also involve the tools used for capturing and reproducing the functionality and atmosphere of laboratories and/or activities on the field. C4: Quality of e-Learning. In this category, all criteria are related to the quality of e-Learning c41: Effectiveness. The general evaluation of the effectiveness of the current e-Learning experience. c42: Acceptability. The acceptability of the current e-learning experience. c43: Course design. The course has been structured correctly. In order to implement the evaluation experiment and estimate the values of the criteria, a questionnaire was designed. Table 2 presents the questions of the questionnaire and their connection with the criteria for the evaluation of the e-learning experience. After the criteria have been defined and the questionnaire has been designed, ENVEL consists of 11 main steps that are presented in Figure 1. Steps 1-4 are presented in Section 3 and do not have to be repeated every time ENVEL is implemented. Steps 5-11 are presented in section 4 and the first four are obligatory to run in every evaluation experiment while the last three are only implemented if criteria with low scores occur. 3 ENVEL: prioritize evaluation criteria According to ENVEL, the e-learning experience is measured using specific evaluation criteria. However, these criteria are not equally important in the evaluation process. For this purpose, ENVEL uses the Fuzzy Analytic Hierarchy Process (FAHP) (Buckley 1985). The steps of the theory for the criteria prioritization are the following: 1. Form the groups of decision-makers A1-A2: Two sets of decision-makers (DMs) that involve human experts and students are set. All professors and students should have experience in e-learning so that they can make decisions on the importance of the criteria. The appropriate choice of experts is of great importance because only in this way the framework would give reliable and valid results. These groups are called A1 and A2, respectively. As a result, group A1 contained three professors. One was an expert in e-learning, one in pedagogy, and one in education and didactics. Group A2 was comprised of 6 students who had previous experience in e-learning. Both groups of DMs are considered homogeneous and, therefore, no degrees of reliability (or importance) were determined. 2. Construct a fuzzy judgment matrix: To scale the relative importance of the criteria, a fuzzy judgment matrix should be constructed. More specifically, a comparison matrix is formed so that the criteria of the same level are pair-wise compared. Each evaluator is asked to express the relative importance of two criteria at the same level using linguistic terms, which are then transformed into triangular numbers (Table 3). As a result, each evaluator completes a matrix for comparing C1-C4, one for c11-c15, one for c21-c23, one for c31- c33, and finally one for c41-c43. This procedure is done for both professors and students. Table 3: The linguistic variables and the corresponding triangular fuzzy numbers Linguistic variables Triangular fuzzy numbers Equally important (1,1,1) Intermediate 2 (1,2,3) Weakly important (2,3,4) Intermediate 4 (3,4,5) Strongly more important (4,5,6) Intermediate 6 (5,6,7) Very strongly more important (6,7,8) Intermediate 8 (7,8,9) Absolutely more important (9,9,9) According to Buckley (1985), a fuzzy judgment matrix can be defined as: 𝑅 ̅ 𝑘 = [𝑟 ̃ 𝑖𝑗 ] 𝑘 , where 𝑅 ̅ 𝑘 is a fuzzy judgment matrix of evaluator k, 𝑟 ̃ 𝑖𝑗 𝑘 the fuzzy assessments between criterion i and criterion j of evaluator k, 𝑟 ̃ 𝑖𝑗 𝑘 = (𝑙 𝑖𝑗 𝑘 , 𝑚 𝑖𝑗 𝑘 , 𝑢 𝑖𝑗 𝑘 ), n is the number of the 𝑟 ̃ 𝑖𝑗 𝑘 = (1,1,1), when 𝑖 = 𝑗 and 𝑟 ̃ 𝑖𝑗 𝑘 = 1/𝑟 ̃ 𝑖𝑗 𝑘 , 𝑖 , 𝑗 = 1,2, … , 𝑛 . For example, the matrix for comparing C1-C4 has been completed by an expert in education and didactics as presented in Table 4. Table 4: The 𝑅 ̅ 1 completed by the environmental education expert C1 C2 C3 C4 C 1 1 1 1 1 2 3 1 1 1 /3 /2 1 C 2 /3 /2 1 1 1 1 /3 /2 1 /4 /3 /2 C 3 1 1 1 1 2 3 1 1 1 /3 /2 1 C 4 1 2 3 2 3 4 1 2 3 1 1 1 Each DM completes all five matrices and the final values of each matrix are calculated taking into account the geometric mean of the corresponding values of each matrix’s cell in the respective matrices. As a result, the final matrices are built. For example, Tables 5-9 present those tables for the professors. Quite similar are the respective tables for the students. 492 Informatica 47 (2023) 487–500 K. Kabassi C1 C2 C3 C4 C1 1 1 1 1.26 2.29 3.30 1.00 1.00 1.00 0.30 0.44 0.79 C2 0.30 0.44 0.79 1 1 1 0.33 0.50 1.00 0.22 0.13 0.44 C3 1.00 1.00 1.00 1.00 2.00 3.00 1 1 1 0.30 0.44 0.79 C4 1.26 2.29 3.30 2.29 7.42 4.64 1.26 2.29 3.30 1 1 1 Table 6: Matrix for the pair-wise comparison of the sub-criteria of C1 c11 c12 c13 c14 c15 c11 1 1 1 1 2 3 0.25 0.33 0.50 0.26 0.35 0.55 0.33 0.50 1 c12 0.33 0.50 1 1 1 1 0.20 0.25 0.33 0.33 0.50 1.00 0.33 0.50 1 c13 2.00 3 4 3 4 5 1 1 1 1.59 2.62 3.63 2 3 4 c14 1.82 2.88 3.91 1 2 3 0.28 0.38 0.63 1 1 1 1 1 c15 1 2 3 1 2 3 0.25 0.33 0.50 1 1 1 1 1 1 c21 c22 c23 c21 1 1 1 1 2 3 0.25 0.33 0.50 c22 0.33 0.50 1 1 1 1 0.20 0.25 0.33 c23 2 3 4 3 4 5 1 1 1 Table 8: Matrix for the pair-wise comparison of the sub- criteria of C3 c31 c32 c33 c31 1 1 1 0 0.33 0.50 0.33 0.50 1 c32 2 3 4 1 1 1 1 1 1 c33 1 2 3 1 1 1 1 1 1 Table 9: Matrix for the pair-wise comparison of the sub- criteria of C4 c41 c42 c43 c41 1 1 1 2 3 4 1 2 3 c42 0.25 0.33 0.50 1 1 1 1 2 3 c43 0.33 0.50 1 0.33 0.50 1 1 1 1 3. Fuzzy weights 𝒘̃ 𝒊 are calculated. The geometric mean of the fuzzy comparison value of the attribute 𝑖 to each attribute can be found as 𝑟 ̃ 𝑖 = [∏ 𝑝 ̃ 𝑖𝑗 𝑛 𝑗 =1 ] 1 𝑛 , 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖 then the fuzzy weight 𝑤 ̃ 𝑖 of the 𝑖 𝑡 ℎ attribute indicated by a triangular fuzzy number is calculated as 𝑤 ̃ 𝑖 = 𝑟 ̃ 𝑖 × [∑ 𝑟 ̃ 𝑗 𝑛 𝑗 =1 ] −1 = (𝑤 𝑖 𝑙 , 𝑤 𝑖 𝑚 , 𝑤 𝑖 𝑢 ) 4. Undertake defuzzification. Finally, the fuzzy priority weights are converted into crisp values by using the center of area method as follows 𝑤 𝑖 = 𝑤 ̃ 𝑖 ∑ 𝑤 ̃ 𝑗 𝑛 𝑗 =1 = 𝑤 𝑖 𝑙 + 𝑤 𝑖 𝑚 + 𝑤 𝑖 𝑢 ∑ 𝑤 ̃ 𝑗 𝑛 𝑗 =1 Table 10: Weights of the criteria for professors and students in the department of environment Weight for professors Weight for students c11: Accessibility 0.118 0.170 c12: Response time 0.096 0.126 c13: Reliability 0.412 0.432 c14: Easy to use/simplicity. 0.197 0.220 c15: e-Learning Management 0.177 0.052 c21: Quality of Synchronous Communication student- instructor 0.241 0.241 c22: Quality of Synchronous collaboration between students 0.145 0.145 c23:Students’ participation 0.613 0.613 c31: e-LearningCulture 0.226 0.221 c32: e-Learning Support 0.334 0.337 c33: e-Learning Infrastructure 0.440 0.442 c41: Effectiveness 0.528 0.613 c42: Acceptability 0.261 0.241 c43: Course design 0.211 0.145 As a result, for each criterion, the final weights are calculated for the professors and students. These weights of the criteria are presented in Table 10. This process revealed that for both professors and students, the most important criterion of the first level is ‘Quality of Learning’, followed by the ‘Functionality of Learning’ and ‘Readiness’. Within the sub-criteria of ‘Functionality Table 5: Matrix for the pair-wise comparison of the fourcriteria of the first level. Table 7: Matrix for the pair-wise comparison of the sub-criteria of C2 A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 493 of the system’, the ‘Reliability’ of the system was considered by far the most important criterion. Regarding ‘Quality of communication’, the weights of the criteria were the same for professors and students. The sub- criterion ‘Students’ participation’ was considered far more important than the other two. As far as ‘e-Learning Readiness’, for both groups, the infrastructure was considered important and the support in the e-Learning process was followed. Finally, concerning the ‘Quality of e-Learning’, ‘Effectiveness’ is considered much more important than the other two criteria. Steps 1-4 are not essential to be repeated during the application of ENVEL. Researchers may use the weights presented in Table 10. However, if other researchers feel that the nature of the study program they evaluate may influence the weights of importance of the criteria, then steps 1-4 have to be repeated to calculate new weights. 4 ENVEL: Evaluating e-learning aspects Steps 5-8 have to be repeated every time ENVEL is implemented, while steps 9-11 may be optionally implemented if low values on criteria have occurred: 5. Form B1 and B2 committees of DMs. The members of the committees are the professors (B1) and the students (B2) participating in the survey. B1 should be a heterogeneous group of professors. This means that the group should contain professors with different perceptions of e-learning and different levels of skills in e-learning and computer usage. Ideally, they should cover different subjects of the study program being evaluated. Similarly, B2 should contain students who have different skills and perceptions of e-learning. 6. Determine the degree of reliability (or importance) of the DMs. Since the evaluation is a problem under group decision-making conditions, the reliability of the DMs should be determined. If the degrees of importance of DMs are equal, then the group of decision-makers is deemed a homogenous group (Chou et al. 2008). Otherwise, the group is deemed a heterogeneous group. If the groups are considered heterogeneous, then it is proposed that the DMs that have previous experience in e-learning can have slightly better reliability than the others as they have experience on the subject. The degrees of importance of DMs are 𝐼 𝑡 , where t is the DMs,𝐼 𝑡 ∈ [0,1] and ∑ 𝐼 𝑡 𝑘 𝑡 =1 ∈ [0,1]. 7. Calculate the values of criteria. The calculation of the values of the criteria is made by the heterogeneous groups of DMs. All the questions of the questionnaire used the five Likert scale for their answers, except for the one question of c31 used in step 6 of ENVEL and one of the two questions related to c42 (Would you like to continue e-learning after the end of the COVID-19 era?), in which the answers were three (yes, no, only for the lessons that e-learning seems appropriate). The answer to each question is the value of the criterion corresponding to that question. The answers to the questionnaires are collected and we make the following estimations: in the case of the criteria that have only one question assigned to them, the mean of all answers to each question. in the case of criteria c31 and c42, the values of the criteria are acquired only by the one question that uses five Likert scale answers. in the case of criterion c23, which involves students’ participation and has two possible answers, we calculate the mean of each question and then take the mean of these two values. The values of the sub-criteria are within the range [1,5]. Those values and the values of 𝐼 𝑡 are used for calculating the mean which is assigned a value of the criterion c1-c4 and can be further used to conclude about the e-learning application. As a result, if we suppose the k members of the group of DMs had previous experience in e-learning and m DMs hadn’t had any experience in e- learning then 𝑟 𝑗 = ∑ 𝐼 𝑖 𝑐 𝑖𝑗 𝑘 𝑖 =1 𝑘 + ∑ 𝐼 𝑖 𝑐 𝑖𝑗 𝑚 𝑖 =1 𝑚 . 8. Calculate the final value of each main criterion. The aggregation of the weights and performance values is calculated as follows: 𝑐 𝑖 = ∑ 𝑤 𝑗 𝑟 𝑖𝑗 𝑛 𝑗 =1 This value is used for characterizing the application of e-learning. Following the study by Linjawi & Alfadda (2018), the scale was set as follows: - Low score: if the final value ranged from 1 to <3. - Acceptable/moderate score: if the final value ranged from 3 to <4. High score: if the final value ranged from 4 to 5. 9. Definition of the criteria for interviews. This step is implemented for defining the criteria that are characterized as a low score. For the criteria that are characterized as a low score, a set of interviews is performed to find out: - how severe are the problems related to this criterion - the exact nature of the problems that were encountered during e-learning implementation. 10. Definition of the group of DMs D1. As soon as the problems are identified a new group of decision- makers is formed. These decision-makers should have specialization in the study program that is evaluated and have experience in e-learning. 11. Analyze results and decide on improvements. The group of decision-makers D1 should decide on the improvements that have to be implemented to ameliorate the e-learning process in the department. 5 The Department of environment turns to e-learning The Department of Environment runs three different study programs related to the Environment, Conservation, and Technologies of the Environment. These study programs involve several different courses such as physics, chemistry, ecology, protected area management, geographical information systems, databases, waste management, renewable energy sources, etc. The courses are implemented using theoretical 494 Informatica 47 (2023) 487–500 K. Kabassi lectures, laboratories, and practice exercises, which in some cases take place on the field. In Greek Universities, information and communication technologies were mainly used in classrooms or in the form of asynchronous e-learning using Learning Management Systems for uploading additional educational material. The department had been using blended learning for the last decade in some courses but its usage depended only on the professor responsible for each course. More specifically, professors had been using e-class, a Learning Management System for uploading assignments, notes, announcements, etc. However, synchronous e-learning was not allowed in Greek Universities, except for the Hellenic Open University. During the Coronavirus emergency, all Greek Universities were asked to reorganize the educational process and provide all courses remotely. Synchronous e- learning was suddenly not only accepted but was considered mandatory. This is especially challenging for a department of Environmental Science that implements theoretical lectures, laboratories, and practice on the field. All professors, irrelevant whether they were in favor of e-learning or not, and if they had experience in e- learning or not, were asked to re-organize their courses and provide synchronous e-learning. More specifically, no more than 64% of the professors had previous experience in e-learning, synchronous or asynchronous. Another obstacle to the implementation was the fact that not all professors were in favor of e-learning. 21% of the professors were not or were very little in favor of e- learning. Another 21% of the professors were moderately in favor of e-learning and only 57% supported techniques of distance education. Despite this fact, all professors successfully transformed their courses and the department managed to provide all courses by distance. After two months of e-Learning implementation, a questionnaire was developed and teachers and students were asked to answer it voluntarily. As a result, 14 professors of various subjects related to the Environment, Conservation, and Environmental Technology, and 98 students of these subjects participated in the study. All of them had been actively attending the e-learning courses. 6 Case Study: Application of ENVEL in the department of environment Steps 1-4 are implemented once and could be used as-is by other researchers that apply ENVEL. However, steps 5-8 should be implemented in each evaluation experiment. In this section, we present the implementation of steps 5-8 of the ENVEL for the evaluation of e-learning in the Department of Environment at Ionian University. Table 11: Results of the evaluation of professors and students in the department of environment Professor value Student value Weighted value for professors Weighted value for students c11: Accessibility 4.456 3.596 0.526 0.611 C1 prof=4.27 C1 st=3.16 c12: Response time 4.484 3.267 0.430 0.412 c13: Reliability 4.092 2.983 1.686 1.289 c14: Easy to use/simplicity. 4.516 3.211 0.890 0.706 c15: e-Learning Management 4.185 2.760 0.741 0.144 c21: Quality of Synchronous Communication student-instructor 3.936 3.035 0.949 0.731 C2 prof=3.63 C2 st=3.08 c22: Quality of Synchronous collaboration between students 3.723 2.799 0.540 0.406 c23:Students’ participation 3.496 3.162 2.143 1.938 c31: e-LearningCulture. 3.388 2.629 0.766 0.581 C1 prof=4.22 C1 st=2.88 c32: e-Learning Support 4.484 2.925 1.498 0.986 c33: e-Learning Infrastructure. 4.452 2.985 1.959 1.319 c41: Effectiveness 3.724 2.741 1.966 1.680 C1 prof=4.02 C1 st=2.81 c42: Acceptability 4.300 2.876 1.122 0.693 c43: Course design 4.392 2.993 0.927 0.434 A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 495 5. Form a committee of DMs for professors and one committee of DMs for students. During the implementation of ENVEL, the questionnaires were given to the members of B1 (14 professors) and the members of B2(94 students). All members of groups B1 and B2 participated in the e-learning and voluntarily became members of the groups after ensuring that these groups were heterogeneous regarding their perceptions and skills in e-learning. 6. Determine the degree of reliability (or importance) of the DMs. The question of c31 (Did you have previous experience in e-learning?), which the students and the professors could only answer yes or no, can be used to determine the degree of importance of each DM. If we consider that all DMs that have experience have importance ω exp = 1 and those that don’t have experience have importance ω non−exp = 0.85. Then, the degree of reliability is calculated as I κ = ω κ ∑ ω ι 2 i=1 . As a result,I exp = 0.54 for DMs with experience in e-learning and I non−exp = 0.46 for DMs for users with no experience in e-learning. 7. Calculate the values of the criteria. Taking into account the mean values of criteria and the reliability of the members of the group we calculate the values of all criteria. All these values are presented in Table 10. 8. Calculate the final value of each main criterion. Using the values of sub-criteria that were calculated in step 7 and the weights of the sub-criteria and applying the weighted sum we estimate the values of the criteria. From the analysis of the values of the criteria, one can easily conclude the different aspects of e-learning in a whole study program in Higher Education. 7 Discussion on the results of the case study and proposed improvements ENVEL has been applied for evaluating e-learning in the Department of Environment at the Ionian University. ENVEL, similar to the frameworks of Mahdavi et al. (2008), Çelikbilek & Adıgüzel Tüylü (2019) and Jeong & Gonzalez-Gomez (2020), has two levels of criteria. Our proposed approach has 4 criteria in the first level and 14 criteria in the second level. ENVEL, unlike Mahdavi et al. (2008) and Alqahtani & Rajkhan 2020 that use AHP, uses Fuzzy AHP, which is considered more friendly to professors and students due to the linguistic terms that are used. However, the main difference between our framework and the frameworks presented in Table 1 (Mahdavi et al. 2008, Stecyk 2019, Çelikbilek & Adıgüzel Tüylü 2019, Alqahtani & Rajkhan 2020, Jeong& Gonzalez-Gomez 2020) is that ENVEL is used for the evaluation of a whole study program implemented by distance and not specific e-learning courses and/or systems. This is the main reason why it was important to develop a new framework instead of using one of the existing ones presented in Table 1. The criteria used in those frameworks do not correspond to the aspects of a whole study program. The results of the evaluation revealed that the whole e-learning experience was rated as mediocre and, although it was considered satisfactory as a solution to an emergency, it needs improvements if it is to be implemented again. Furthermore, both instructors and students agreed that face-to-face education is more effective, especially for a subject like environmental science that needs laboratories and practical exercises in the field. One of the main conclusions involves the culture of users in e-Learning. Indeed, the results show that, although 78% of the participants were not in favor of e- learning, 40% considered the current implementation of e-learning very effective and the other 32% characterized it as of medium effectiveness. The evaluation experiment revealed significant deviations in the views of students and professors. Taking into account how successful the implementation of e-learning was, the values assigned to the sub-criteria were 4.092-4.516 by the professors and much lower by the students (2.760-3.596). These values have occurred by computing the weighted value of the responses. Regarding the factor of communication, again the values of the criteria assigned by the professors were much better than those assigned by students. However, what seemed rather disappointing was the deviation in the values in the last two main criteria. For example, as far as the readiness for the e-Learning implementation was concerned the value of the criterion for the professors was 4.22 while the corresponding value for the students was 2.88. 83% of the professors and the students stated that they had no previous experience in e-learning. Furthermore, only 28% of the participants were in favor of e-Learning, before that semester. Taking into account the low experience of professors in e-learning and the low acceptability of e- learning in general, before the semester of the evaluation, the support and the infrastructure provided by the university were considered quite satisfactory. A rather important criterion for evaluating the whole process is the ‘Quality of e-Learning’. This criterion is mainly influenced by the sub-criterion ‘c41- Effectiveness’ and then ‘c42: Acceptability’ and ‘c43: Course design’. The deviation of the values given by students and professors in this criterion is high. For example, the three sub-criteria c41, c42, c43 were rated 3.724-4.392 by professors and 2.741-2.993 by students. Notably, most of the criteria were rated above 4 by the professor, and therefore, they were considered a high score, except for the criterion ‘Learner Attractiveness’ which was considered mediocre. However, the values of the criteria provided by the students were much lower and were considered a medium score for the functionality of the system and the learner attractiveness and a low score for the readiness of e-learning and its quality. This shows a significant deviation between the professors’ and students’ views and shows that there is much more to be done to improve aspects of e-learning before it can be fully and more effectively implemented. Since criteria, c3 and c4 are rated with low scores, steps 9-11 of the ENVEL should be implemented. 496 Informatica 47 (2023) 487–500 K. Kabassi 9. Definition of the criteria for interviews. The criteria that have low scores are c3 and c4. 10. Definition of the group of DMs D1. Two human experts from the Department of Environment were selected to perform the interviews with random professors and students who participated in the experiment. More specifically, one expert in e-learning and one in environmental education performed the interviews. Both experts had served as head of the department in the past. 11. Analyze results and decide on improvements. After interviews with the DMs of D1 were performed, the main problems occurred because students were not familiar with the particular method and had problems adjusting to e-learning. Many students also encountered technical problems as they didn’t have the equipment to connect from home and had to follow the e-learning courses from their mobile phones. The low value of quality of learning was an expected problem, as professors had designed the courses for on-site learning. The complaints mainly involved laboratories and courses that were supposed to be implemented in situ and affected the effectiveness of e-learning. The group of decision-makers D1, after performing the interviews, decided on the improvements that have to be implemented to ameliorate the e-learning process and make teaching and learning more effective. The following improvements were proposed: - Organizing seminars and webinars about e- learning. These seminars would help users get informed on the platforms used and the e-learning process, in general. - Updating Opencourses. The main platform that was used for uploading the course material, encountered several problems, which could be addressed with a simple update. - Re-organizing laboratories to be better implemented by distance or postpone them until on-site learning is applicable. - Purchasing an e-learning platform. In this way, the courses could last longer and have more functionality. - Giving suggestions to professors about their course design, to cover the e-learning needs in a better and more effective way. 8 Conclusions In general, face-to-face education may be preferable and more easily carried out than distance education, even if it is synchronous. However, in cases like the corona virus emergency synchronous e-learning was the only solution. This paper presented a framework for e-learning evaluation called ENVEL. ENVEL runs the evaluation experiment under group decision-making conditions using heterogeneous groups of students and professors. Combining the views of heterogeneous groups of people may provide a broader view of the implementation of e- learning. For this purpose, the framework suggests that the sample of users participating in the experiment should involve both students and professors, experienced and non-experienced users in e-learning, and people with different views on e-learning. The framework applies multi-criteria decision- making in order to combine the different aspects of e- learning and collect summative results on the e-learning. These results show the effectiveness and success of the e- learning implementation of a whole semester in Higher Education. Furthermore, the particular framework presents an analysis of the criteria that are taken into account in the evaluation of e-learning as well as an estimation of their importance in the evaluation process. The analysis of the values of the criteria provides a useful tool to conclude the specific aspects that need to be addressed to improve e-learning centrally. Therefore, the framework could be used in departments in higher education to draw conclusions and schedule the required changes to improve e-learning application in the whole study program. The weights of the criteria also reveal the priority of the related improvements. For example, the corrections in aspects that are assigned to criteria with higher weight should be addressed in higher priority. For the estimation of the weights of criteria, FAHP is used. The selection of FAHP over the other MCDM theories lies in the fact that it has a very well-defined process for calculating the weights of criteria in comparison to other theories such as TOPSIS (Hwang & Yoon 1981), VIKOR (Opricovic, 1998), MAUT (Vincke 1992), SAW (Hwang & Yoon 1981; Chou et al. 2008), etc. Moreover, this process instead of asking experts to assign a weight to each criterion, allows them to make pairwise comparisons, and, therefore, was better than other methods. As a result, this process results in capturing better expert reasoning. This is also an advantage of AHP. However, the quantification in numbers may be difficult for some people. Another advantage of FAHP is that it supports decision-makers to assign linguistic variables in the form of numeric values to express their judgment and can incorporate incomplete, unobtainable, and unquantifiable information into the decision model in a fuzzy environment (Ramanayaka et al.2019, Nagpal et al. 2015, Sivarji et al. 2013, Chang et al. 2008, Sadeghi et al. 2012, Chen et al. 2015). It is among our plans, to implement this framework for e-Learning evaluation in other similar departments of Higher Education and prove its effectiveness for e- Learning evaluation in Higher Education in general. Furthermore, the same experiment is going to be repeated in the next semester after taking corrective actions in the implementation of e-learning. A second evaluation experiment is going to show if the results of the evaluation are going to change and in what way. References [1] Abdalla, I. (2007). Evaluating effectiveness of e- blackboard system using TAM framework: A structural analysis approach. AACE Journal, 15(3), pp. 279-287. [2] Abuhlfaia K., de Quincey E. (2019). Evaluating the Usability of an e-Learning Platform within Higher A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 497 Education from a Student Perspective. ICEEL 2019: Proceedings of the 2019 3rd International Conference on Education and E-Learning, pp. 1–7 https://doi.org/10.1145/3371647.3371661 [3] Aguti, B., Wills, G.B., Walters, R.J. (2014). An Evaluation of the Factors that Impact on the Effectiveness of Blended E-Learning within Universities. International Conference on Information Society, pp. 117-121. https://doi.org/10.1109/i-society.2014.7009023 [4] Alavi, M., & Leidner, D. E. (2001). Technology- mediated learning: A call for greater depth and breadth of research. Information Systems Research, 12(1), pp. 1–10. https://doi.org/10.1287/isre.12.1.1.9720 [5] Al-Fadhli, S. & Khalfan, A. (2009). Developing critical thinking in e-learning environment: Kuwait University as a case study. Assessment & Evaluation in Higher Education, 34(5), pp. 529- 536, DOI: 10.1080/02602930802117032 [6] Alqahtani A.Y., Rajkhan A.A. (2020) E-Learning Critical Success Factors during the COVID-19 Pandemic: A Comprehensive Analysis of E- Learning Managerial Perspectives. Educational Sciences, 10, 216. https://doi.org/10.3390/educsci10090216 [7] Altun Turker Y., Baynal K., Turker T. (2019) The Evaluation of Learning Management Systems by Using Fuzzy AHP, Fuzzy TOPSIS and an Integrated Method: A Case Study, Turkish Online Journal of Distance Education, 20(2), pp. 195-218, DOI: 10.17718/tojde.557864 [8] Anderson, T. (2001). The hidden curriculum in distance education. An update view, Change 33, pp. 28-35. https://doi.org/10.1080/00091380109601824 [9] Anggrainingsih, R. ,Umam, M. Z. ,Setiadi, H. (2018) Determining e-learning success factor in higher education based on user perspective using Fuzzy AHP. MATEC Web of Conferences 154, 03011, https://doi.org/10.1051/matecconf/201815403011 [10] Anis A. & Islam R. (2015) The application of analytic hierarchy process in higher-learning institutions: a literature review. Journal International Business and Entrepreneurship Development, 8(2). https://doi.org/10.1504/jibed.2015.070446 [11] Arbaugh, J. B. (2004). Learning to learn online: A study of perceptual changes between multiple online course experiences. The Internet and Higher Education, 7, pp. 169-182. https://doi.org/10.1016/j.iheduc.2004.06.001 [12] Arbaugh, J. B. (2008). Does the community of inquiry framework predict outcomes in online MBA courses? International Review of Research in Open and Distance Learning, 9(2). https://doi.org/10.19173/irrodl.v9i2.490 [13] Barteit S, Guzek D, Jahn A, Bärnighausen T, Jorge MM, Neuhann F. (2020) Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Computer Education. 145, 103726. doi: 10.1016/j.compedu.2019.103726. PMID: 32565611; PMCID: PMC7291921. [14] Benbunan-Fich, R. & Hiltz, S. R. (2003) Mediators of the effectiveness of online courses, IEEE Transactions on Professional Communication, 46, pp. 298 312. https://doi.org/10.1109/tpc.2003.819639 [15] Bhadauria R. (2016) E-Learning – A boon for Indian Higher Education System. International Journal of Engineering Technology, Management and Applied Sciences. 4(2), pp. 122-128, ISSN 2349-4476 [16] Buckley, J.J. (1985). Fuzzy hierarchical analysis. Fuzzy Sets and Systems, 17(3), pp. 233-247. https://doi.org/10.1016/0165-0114(85)90090-9 [17] Çelikbilek Y. & Adıgüzel Tüylü A.N. (2019) Prioritizing the components of e-learning systems by using fuzzy DEMATEL and ANP, Interactive Learning Environments, DOI: 10.1080/10494820.2019.1655065 [18] Chang, C. W., Wu, C. R., & Lin, H. L. (2008) Integrating fuzzy theory and hierarchy concepts to evaluate software quality, Software Quality Journal, 16(2), pp. 263–76. https://doi.org/10.1007/s11219-007-9035-2 [19] Chen, J.-F., Hsieh, H.-N., & Do, Q. H. (2015) Evaluating teaching performance based on fuzzy AHP and comprehensive evaluation approach, Applied Soft Computing, 28, pp. 100–108. https://doi.org/10.1016/j.asoc.2014.11.050 [20] Chou, S.Y., Chang, Y.H., Shen, C.Y. (2008) A fuzzy simple additive weighting system under group decision-making for facility location selection with objective/subjective attributes. European Journal of Operational Research, 189, pp. 132–145. https://doi.org/10.1016/j.ejor.2007.05.006 [21] Chou, S.-W., & Liu, C.-H. (2005) Learning effectiveness in a web-based virtual learning environment: A learner control perspective, Journal of Computer Assistive Learning, 21, pp. 65-76 https://doi.org/10.1111/j.1365-2729.2005.00114.x [22] Clark, L. J. (2001). Web-based teaching: A new educational paradigm, Intercom, 48, pp. 20-23. [23] Elkaseh A.M., Wong K.W. and Fung C.C.(2016) Perceived ease of use and perceived usefulness of social media for e-learning in Libyan higher education: a structural equation modeling analysis. International Journal of Information and Education Technology, 6(3), pp. 192-199. https://doi.org/10.7763/ijiet.2016.v6.683 [24] Erensal, Y. C., Öncan, T., &Demircan, M. L. (2006). Determining key capabilities in technology management using fuzzy analytic hierarchy process: A case study of Turkey, Information Sciences, 176(18), pp. 2755–70. https://doi.org/10.1016/j.ins.2005.11.004 [25] Frehywot, S., Vovides, Y., Talib, Z. et al. (2013) E- learning in medical education in resource constrained low- and middle-income 498 Informatica 47 (2023) 487–500 K. Kabassi countries. Human Resources Health 11(4). https://doi.org/10.1186/1478-4491-11-4 [26] Gupta, S., & Bostrom, R.P. (2009). Technology- Mediated Learning: A Comprehensive Theoretical Model. Journal of the Association for Information Systems, 10(9), pp. 686-714. https://doi.org/10.17705/1jais.00207 [27] Harasim, L., Hiltz, S. R., Teles, L., &Turoff, M. (1995) Learning networks: A field guide to teaching and learning online. The MIT Press, Cambridge, MA. [28] Harrati, N., Bouchrika, I., Tari, A., and Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computer Human Behavior, 61, pp. 463--471. DOI= https://doi.org/10.1016/j.chb.2016.03.051. [29] Hartman, K., Neuwirth, C. M., Kiesler, S., Sproull, L., Cochran, C., Palmquist, M., &Zubrow, D. (1995) Patterns of social interaction and learning to write: Some effects of network technologies. In: Z. L. Berge and M. P. Collins, Editors. Computer mediated communication and the online classroom, Vol. 2. Hampton Press, Inc, Cresskill, NJ, pp 4778. https://doi.org/10.1177/0741088391008001005 [30] Haverila, M., &Barkhi, R. (2009). The Influence of Experience, Ability and Interest on eLearning Effectiveness. European Journal of Open, Distance and E-Learning, https://www.eurodl.org/materials/contrib/2009/Hav erila_Barkhi.pdf [31] Hrastinski, S. (2008) Asynchronous and synchronous e-learning. Educause Quarterly, 31(4), pp. 51-55. [32] Hwang, C.L., Yoon,K. (1981) Multiple Attribute Decision Making: Methods and Applications. Lecture Notes in Economics and Mathematical Systems, 186. [33] Islam, A. Md., Rahim, N.A.N., Liang, T.C., &Momtaz, H. (2011) Effect of Demographic Factors on E-Learning Effectiveness in A Higher Learning Institution in Malaysia. International Education Studies. 4(1), pp. 112-121. https://doi.org/10.5539/ies.v4n1p112 [34] Islas, E., Perez, M., Rodriguez, G., Paredes, I., Avila, I., & Mendoza, M. (2007), E-learning tools evaluation and roadmap development for an electrical utility, Journal of Theoretical and Applied Electronic Commerce Research, 2(1), pp. 63-75. https://doi.org/10.3390/jtaer2010006 [35] Jasim, M. H., Maznah M. K., Nizal M. S. I. (2018) Evaluation of E-Learning Approaches Using AHPTOPSIS Technique. Journal of Telecommunication, Electronic and Computer Engineering, 10 (1-10). pp. 7-10. ISSN 2180-1843. [36] Jeong J. & González-Gómez D. (2020). Assessment of sustainability science education criteria in online- learning through fuzzy-operational and multi- decision analysis and professional survey, Heliyon, Volume 6, Issue 8, August 2020, e04706, https://doi.org/10.1016/j.heliyon.2020.e04706 [37] Liaw, S.S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51, pp.864–873 https://doi.org/10.1016/j.compedu.2007.09.005 [38] Liaw, S. S., Huang, H. M., & Chen, G. D. (2007). Surveying instructor and learner attitudes toward e- learning, Computers & Education. https://doi.org/10.1016/j.compedu.2006.01.001 [39] Lin, H.-F. (2010) An application of fuzzy AHP for evaluating course website quality. Computers & Education, 54, 877–888 https://doi.org/10.1016/j.compedu.2009.09.017 [40] Linjawi, AI, Alfadda, LS. (2018) Students’ perception, attitudes, and readiness toward online learning in dental education in Saudi Arabia: a cohort study. Advances in Medical Education and Practice, 9, pp. 855-863. https://doi.org/10.2147/AMEP.S175395 [41] Mahdavi, I., Fazlollahtabar, H., Heidarzade, A., Mahdavi-Amiri N. and Rooshan, Y.I. (2008). A Heuristic Methodology for Multi-Criteria Evaluation of Web-Based E-Learning Systems Based on User Satisfaction. Journal of Applied Sciences, 8, pp. 4603-4609. DOI: 10.3923/jas.2008.4603.4609 [42] Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A structural equation model of predictor for effective online learning, Journal of Management Education, 29, pp. 531-563. https://doi.org/10.1177/1052562904271199 [43] Martinez-Caro, E. (2011). Factors Affecting Effectiveness in E-Learning: An Analysis in Production Management Courses. Computer Applications in Engineering Education, 19(3), pp. 572 – 581. DOI: 10.1002/cae.20337 [44] Martín-Lara, M.Á., Rico, N. (2020) Education for Sustainable Energy: Comparison of Different Types of E-Learning Activities. Energies 2020, 13, 4022. https://doi.org/10.3390/en13154022 [45] Mercader, C., Gairín, J. (2020) University teachers’ perception of barriers to the use of digital technologies: The importance of the academic discipline. International Journal of Educational Technology in Higher Education, 17, 4. https://doi.org/10.1186/s41239-020-0182-x [46] Mohamadali, N.A., & Garibaldi, J. (2011). Comparing user acceptance factors between research software and medical software using AHP and Fuzzy AHP. The 11th Workshop on Computational Intelligence, 7 - 9 September 2011, Kilburn Building. [47] Mulubrhan, F., Akmar, A., Mokhtar, A. & Muhammad, M. (2014). Comparative Analysis between Fuzzy and Traditional Analytical Hierarchy Process. MATEC Web of Conferences 13. https://doi.org/10.1051/matecconf/20141301006 [48] Nagpal, R., Mehrotra, D. Bhatia, P. K., & Sharma, A. (2015) FAHP approach to rank educational websites on usability, International Journal of A Framework for Evaluating Distance Learning of Environmental… Informatica 47 (2023) 487–500 499 Computing and Digital Systems, 4(4), pp. 251–260. https://doi.org/10.12785/ijcds/040404 [49] Naveed, Q.N., Qureshi, M.R.N., Tairan, N., Mohammad, A., Shaikh, A., Alsayed, A.O., et al. (2020) Evaluating critical success factors in implementing E-learning system using multi- criteria decisionmaking. PLoS ONE 15(5): e0231465. https://doi. org/10.1371/journal.pone.0231465 [50] Njenga, J.K. (2017) E-Learning in Higher Education. Encyclopedia of International Higher Education Systems and Institutions; Springer: Maine, The Netherlands, pp. 1–6. https://doi.org/10.1007/978-94-017-9553-1_323-1 [51] Noesgaard, S. S., & Ørngreen, R. (2015) The Effectiveness of E-Learning: An Explorative and Integrative Review of the Definitions, Methodologies and Factors that Promote e-Learning Effectiveness. The Electronic Journal of eLearning, 13(4), 278-290. [52] Opricovic, S. (1998). Multicriteria optimization of civil engineering systems. Belgrade: Faculty of Civil Engineering [53] Otto, D., Becker, S. (2018) E-Learning and Sustainable Development. Encyclopedia of Sustainability in Higher Education; Springer: Cham, Switzerland, p. 8. https://doi.org/10.1007/978-3-319-63951-2_211-1 [54] Owston, R. (1997) The World Wide Web: A technology to enhance teaching and learning? Educational Researcher, 26(2). https://doi.org/10.2307/1176036 [55] Ramanayaka, K. H., Chen, X., & Shi, B. (2019). UNSCALE: A Fuzzy-based Multi-criteria Usability Evaluation Framework for Measuring and Evaluating Library Websites, IETE Technical Review, 36(4), 412 - 431, 10.1080/02564602.2018.1498032 [56] Reiser, R. (2001). A History of Instructional Design & Technology: Part 1: A History of Instructional Media. Educational Research & Development, 49 (1), pp. 53-64. https://doi.org/10.1007/bf02504506 [57] Richmond, V. P., McCroskey, J. C., Kearney, P. & Plax, T. G. (1987) Power in classroom VII: Linking behaviour alteration techniques to cognitive learning, Communication Education, 36, 1-12. https://doi.org/10.1080/03634528709378636 [58] Saaty, T. (1980). The analytic hierarchy process. New York, McGraw-Hill. [59] Sadeghi, M., aliRashidzadeh, M., &aliSoukhakian, M. (2012). Using analytic network process in a group decision making for supplier selection, Informatica, 23(4), pp. 621–43. https://doi.org/10.15388/informatica.2012.378 [60] Salter, S.M., Karia, A., Sanfilippo, F.M., Clifford, R.M. (2014). Effectiveness of E-learning in Pharmacy Education. American Journal of Pharmaceutical Education, 78(4), Article 83. [61] Schieffer, L. (2016) The Benefits and Barriers of Virtual Collaboration Among Online Adjuncts. Journal of Instructional Research, 5, 109–125. https://doi.org/10.9743/jir.2016.11 [62] Sheikh, T. H., Gupta R. (2018). E-Learning for Higher Education. A Case Study. Journal of Advanced Research in Dynamical & Control Systems, 10(4), ISSN 1943-023X [63] Siemens, G., Gasevic D. & Dawson S. (2015). Preparing for the digital university: A review of the history and current state of distance, blended and online learning. Athabasca: Athabasca University. DOI: 10.13140/RG.2.1.3515.8483 [64] Sivaji, A., Abdullah, M. R., Downe, A. G. & Ahmad, W. F. W. (2013) Hybrid Usability Methodology: Integrating Heuristic Evaluation with Laboratory Testing across the Software Development Lifecycle, 10th International Conference on Information Technology: New Generations, pp. 375–83. https://doi.org/10.1109/itng.2013.60 [65] Stecyk A. (2019) Application of PROMETHEE II method for e-learning course selection, e-mentor, 1(78), 39-45. DOI: 10.15219/em80.1398 [66] Stöhr C., Demaziere C., Adawi T. (2020) The polarizing effect of the online flipped classroom. Computers & Education, 147, 103789 https://doi.org/10.1016/j.compedu.2019.103789 [67] Su, B., Bonk, C. J., Magjuka, R.J., Liu, X. & Lee, S. (2005) The importance on Interaction in web- based education: A program level case study of online MBA courses, Journal of Interactive Online Learning, 4, 119. [68] Sulčič, V., & Lesjak, D. (2009). E-Learning and Study Effectiveness, Journal of Computer Information Systems, 49(3), 40-47. [69] Tai D.W.S., Chen JL., Wang R. (2011) Research on Fuzzy Linguistic Evaluation in e-Learning Using AHP and TOPSIS Based on Web Resources. In: Lin S., Huang X. (eds) Advances in Computer Science, Environment, Ecoinformatics, and Education. CSEE 2011. Communications in Computer and Information Science, 217. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3- 642-23339-5_69 [70] Tiwari, N. (2006). Using the Analytic Hierarchy Process (AHP) to identify Performance Scenarios for Enterprise Application, Computer Measurement Group, Measure It, 4, 3. [71] Tseng, M.-L., & Lin, R.-J., Chen, H.-P. (2011). Evaluating the effectiveness of e-learning system in uncertainty. Industrial Management & Data Systems, 111(6), pp. 869-889. https://doi.org/10.1108/02635571111144955 [72] Vincke, P. (1992) Multi criteria Decision-Aid. Wiley, Chichester. https://doi.org/10.1002/mcda.4020030208 [73] I. Vinogradova, R. Kliukas (2015) Methodology for Evaluating the Quality of Distance Learning Courses in Consecutive Stages, Procedia - Social and Behavioral Sciences, 191, pp. 1583-1589, https://doi.org/10.1016/j.sbspro.2015.04.364. 500 Informatica 47 (2023) 487–500 K. Kabassi [74] Wu, W. & Hwang, L.-Y. (2010). The effectiveness of e-learning for blended courses in colleges: A multi-level empirical study. International Journal of Electronic Business Management, 8(4), pp. 312- 322. [75] Zopounidis, C. (2009). Knowledge-based multi- criteria decision support. European journal of operational research, 195, pp. 827-828. https://doi.org/10.1016/j.ejor.2007.11.026