Tina Rutar Leban, Ph.D., Pedagogical Institute, Slovenia Vladimir Milekšič, National Education Institute Slovenia What Is in There for Us in the e-Schoolbag An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Final report What Is in There for Us in the e-Schoolbag An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools - Final report Authors: Tina Rutar Leban, Ph.D., Pedagogical Institute, Slovenia Vladimir Milekšič, National Education Institute Slovenia Editor: Amela Sambolić Beganović Reviewers: Mira Hedžet Krkač, Andreja Čuk, Mr. sc. Translations: Ensitra prevajanje, Brigita Vogrinec s.p. Publisher: National Education Institute Slovenia For the publisher: Vinko Logaj, Ph.D. Design: Suzana Kogoj Published online: http://www.zrss.si/pdf/e-school-bag-evaluation-study.pdf Ljubljana, 2016 The operation is partly financed by the European Union through the European Regional Development Fund and the Ministry of Education, Science and Sport of Slovenia. The operation is performed under the Operational Programme for Strengthening Regional Development Potentials 2007-2013, development priorities: Economic Developmental Infrastructure; priorities: Information Society. © Zavod RS za šolstvo, 2016 CIP - Kataložni zapis o publikaciji Narodna in univerzitetna knjižnica, Ljubljana 37.091.64:004(0.034.2) 37.018.43:004(0.034.2) RUTAR Leban, Tina What is in there for us in the e-schoolbag [Elektronski vir] : an evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools : final report / Tina Rutar Leban, Vladimir Milekšič ; [translations Ensitra prevajanje]. - El. knjiga. - Ljubljana : National Education Institute Slovenia, 2016 Način dostopa (URL): http://www.zrss.si/pdf/e-school-bag-evaluation-study.pdf ISBN 978-961-03-0329-9 (pdf) 1. Milekšič, Vladimir 283066880 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Final report Tina Rutar Leban, Ph.D., Pedagogical Institute, Slovenia Vladimir Milekšič, National Education Institute Slovenia TABLE OF CONTENTS INTRODUCTION 1 AND THE PURPOSE OF THE EVALUATION . 7 2. METHOD 10 2.1 Sample 10 2.2 Instruments 10 2.3 Evaluation process 12 3. RESULTS 14 3.1 The results of the analysis of the data collected with the questionnaire for teachers 14 3.2 Results of the analysis of data obtained through the observation of lessons 24 3.3 The results of the analysis of data collected by group interviews (focus groups) with the teachers 26 3.4 The results of the analysis of data collected using the questionnaire for pupils/students 28 3.5 The results of the analysis of the data collected using the questionnaire for parents 30 3.6 The results of monitoring the effects of the use of interactive elements of e-textbooks during lessons 32 4 CONCLUSIONS . 38 5 SOURCES . 42 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Final report 1. INTRODUCTION AND THE PURPOSE OF THE EVALUATION The project e-School Bag was carried out by the National Education Institute of Slovenia in consortium partnership with Arnes. The e-School project was partly financed by the European Union through the European Regional Development Fund and the Ministry of Education, Science and Sport of Slovenia. The operation was carried out within the framework of the Operational Programme for Strengthening Regional Development Potentials 2007-2013, developmental priorities: Economic Development Infrastructure; priority orientations: Information Society. Objectives of the e-School Bag project: • Developing modern e-content in the Slovenian language (development of a mini-mum of 15 e-textbooks for social science classes in the 8th and 9th grade of primary school and the 1st grade of secondary school, which cover the entire curriculum for the individual subject in a given class or grade) and of e-services, • Establishing an appropriate infrastructure at participating schools, • Providing support in the use of the e-content and e-services developed within the project in the educational process of the pilot project. • Evaluating the effects of the use of e-content and e-services in the educational process. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 8 Introduction and the purpose of the evaluation The purpose of the evaluation study was to monitor and evaluate the testing and the use of e-content (with an emphasis on e-textbooks) and e-services in primary schools and secondary schools. We carried out the testing, monitoring and the evaluation of the e-content and e-services (with a focus on e-textbooks, which were developed within the E-textbooks projects with a focus on natural sciences1 and the e-School Bag2) for the teaching and learning within two pilot projects: (1) The introduction and the use of e-content and e-services and (2) Testing of e-content and e-services. The pilot projects started in the 2013/2014 school year and were completed in the 2014/2015 school year. In June 2013, we chose 14 schools to participate in the pilot project Introduction and use of e-content and e-services (10 primary schools and 4 secondary schools). Each school was able to stand as a candidate with only one department from the 4th to the 9th grade of primary school and the 1st or 2nd grade of secondary school. We provided all teachers and pupils/students, with the help of our consortium partner, with tablet computers for the purpose of testing the e-content and the e-services. The term PUPIL/STUDENT occurs everywhere in the document. A pupil is a child visiting a primary school and a student is a child visiting a secondary school for general education in Slovenia (a gymnasium). With the help of ARNES, a suitable infrastructure was also put in place at every participating school (optical connections, wireless network, etc.). In January 2014, we chose 44 schools to participate in the pilot project Testing of e-content and e-services (34 primary schools and 10 secondary schools). Schools were able to with at least one department from the 4th to 9th grade of primary school or 1st to 3rd grade of secondary school and at least one teacher who would test and evaluate the e-content and e-services within the class that was participating. Schools that applied to participate in the pilot project ensured that all teachers and pupils/students in the participating department had mobile devices or computers during class for the purpose of testing and evaluating the e-content and e-services. A total of 58 primary and secondary schools3, as well as 286 teachers, were chosen to participate in both pilot projects. In compliance with the goals of the pilot project, the participants of the project planned, implemented, monitored and evaluated the developed e-content (e-textbooks) and e-services as well as the lessons using this e-content and e-services on different e-devices. This report represents the final report on the results of the pilot projects in the second year of implementation (2014/2015 school year). The results of the first year of the two projects, presented in the interim report after the first year of the project, served for the development, testing and revision of the evaluation instruments that were then used in the second year of the projects’ implementation. Based on the analysis of the data collected during the second year of the implementation of the projects, 1 In the E-textbooks project, interactive e-textbooks were created for natural sciences and mathematics in primary schools (2011-2014), which were confirmed by the Expert Council for General Education of Slovenia, for the following subjects: Physics, Chemistry, Mathematics, Natural Sciences, Natural Sciences and Engineering of the 2nd and 3rd triennium of primary school and of secondary school. 2 Within the e-School Bag project, we developed interactive e-textbooks for the 8th and 9th grade of primary school and the 1st year of secondary school for the following subjects: Slovene and English as the first foreign language, German as the second foreign language, Geography, Music and Visual Arts. 3 List of participating schools: http://projekt.sio.si/e-solska-torba/pilotna-projekta/ An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Introduction and the purpose of the evaluation 9 we identified the advantages and disadvantages of the e-content and e-services, the advantages and disadvantages of their use in the teaching process and the impact/effects of e-textbooks on the learning of the pupil/student as well as the impact/effects of the e-textbooks on the teaching by the teacher. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 10 Method 2. METHOD 2.1 Sample 123 teachers (43%), 995 pupils/students and 242 parents participated in the evaluation in the second year of the pilot project. In filling out the evaluation questionnaire, different teachers from different subjects were involved, who used either the e-textbook or a different form of e-content, as the e-textbooks were created at the same time as the implementation of the pilot projects. The primary school pupils (N = 826) and the secondary schools students (N = 169) that were included in the evaluation used the e-textbook in class for at least one of the subjects. In the questionnaire they were asked to choose one subject in which they used the e-textbook and answer questions in connection with the lessons of the selected subject. Parents that were involved in the evaluation had children either in primary (N = 178) or secondary school (N = 64). 614 children were involved in monitoring the introduction of the e-textbooks into the classes, which also included a written examination (for a detailed description see chapter 2.3 Course of the Evaluation), 429 of these were primary school pupils (36 pupils from the 7th grade, 172 pupils from the 8th grade and 221 pupils from the 9th grade) and 185 students from the first grade of secondary school4. Examinations were used in four subjects (Mathematics, Slovene, Geography and Chemistry)5, in 24 departments. 24 teachers participated, which represents 8% of all participating teachers. 99 lessons were monitored; this represents 35% of all participants in the pilot projects. 105 teachers were involved in the focus groups, which represents 37% of all participants. These were carried out in ten focus groups. 2.2 Instruments We developed the following instruments for the purpose of the evaluation of the project: an Evaluation questionnaire for teachers, an Evaluation questionnaire for pupils/students, an Evaluation questionnaire for parents, an Evaluation scale for the observers 4 The results of the written examination include the results of only those pupils/students who attended both the first and the second test. 5 In these subjects, the number of teachers or pupils/students still enabled a valid comparison An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Method 11 of classes, Written tests for testing the acquired knowledge and a Semi-structured interview for the execution of focus groups with teachers. Below, each instrument is described in brief. Evaluation questionnaire for teachers The questionnaire was divided into four sections: the use of the e-textbook, the suit-ability of individual e-learning units, the effects of the use of the e-textbook during the lessons and the use of e-services during the lessons. In total, the questionnaire included 57 items to which teachers had to respond with the help of a scale with three, four or five levels. For a better understanding, the scales are described in more detail in the chapter Results, for each individual question. Evaluation questionnaire for pupils/students The questionnaire was divided into four sections: basic information about the pupil/ student, the use of the e-textbook, the effects of the use of the e-textbook during lessons and the use of e-services during lessons. In total, the questionnaire included 53 items to which the pupils/students had to respond with the help of a scale with three, four or five levels. For a better understanding, the scales are described in more detail in the chapter Results, for each individual question. Evaluation questionnaire for pupils The questionnaire had two sections: the use of the e-textbook at home and the use of ICT6 at home. The questionnaire contained 19 items to which the parents had to answer either on a five-level or three-level scale. For a better understanding, the scales are described in more detail in the chapter Results, for each individual question. Semi-structured interview for the performance of the focus groups with teachers The interview consisted of five questions that were set as starting points for discussion. In addition to the main questions, the person performing the interview had sub-questions available, which he/she could use to help himself/herself encourage an open discussion with the teachers. The semi-structured interviews in the groups of teachers who taught the same subject, were conducted by counsellors from the National Education Institute of Slovenia. Grading scale for the observers of the lessons The grading scale was constructed by the counsellors from the National Education Institute of Slovenia, who were also the ones observing and using the scale as an instrument to make note of the observations. The scale included nine items related to the e-happening during the lessons. The items were related to the use of ICT in studying new materials, an examination of prior knowledge, the development of communication and collaboration skills of pupils, the individualisation and differentiation of lessons, the consolidation and assessment of knowledge, etc. The observers evaluated the items on a five-level scale (0-activity could not be assessed, 1- not true, 2- partially true, 3-mostly true, 4-absolutely true). 6 The term ICT signifies modern information and communications equipment, which includes hardware devices (computer, laptop, tablet, phone, digital camera, etc.), the software used on these devices, and the World Wide Web (the internet). An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 12 Method Written tests for the examination of acquired knowledge in monitoring the effects of the use of the interactive elements of the e-textbooks in class The purpose of the examination was to analyse the effects of the independent acquisition of the same and new content from different media (regular printed textbooks and interactive e-textbooks) in Mathematics, Chemistry, Slovene and Geography in the 8th and 9th grade of primary school and the first year of secondary school. Teachers that participated in the pilot project and taught the same subject in at least two departments of the same grade were included in the examination. Within the framework of this part of the evaluation, specific tests were prepared for four of the subjects that were monitored, which helped us examine, for each subject, how much the pupils/students had learnt on their own with the help of the interactive e-textbooks or the normal printed textbooks. Written tests and instructions for the evaluation were constructed by the counsellors from the National Education Institute of Slovenia. In the selection of the learning content and the preparation of the tasks for the examination of knowledge, we considered the following criteria: • the content of the learning units must be identical in both sources (e-textbook and p-textbook (printed textbook)). • he selected learning unit in the e-textbook included interactive elements which should be the key advantage of e-textbooks used during lessons (multimedia, animation, 3D computer generated models, video footage of experiments, etc.). • he chosen content in the e-textbook should be presented in such a way that it enabled the pupils/students more independent, dynamic and active learning with the multimedia and interactive elements that were included. • we avoided content that had very complex notions and concepts, because we did not want the effect of learning to be more dependent on the difficulty of the content rather than on the medium of learning. • the chosen content of the learning units should demand as little prior knowledge of the pupil/student as possible, so that the pupils/students with less prior knowledge would not have a smaller chance to successfully learn the content. • the chosen content, due to its »artificial« placement in the teaching process should be able to be dealt with as independently as possible from the content being currently taught. • the tasks in the written tests should only test knowledge that was included in both learning units of the e-textbook and the classical textbook. • each test included tasks from three taxonomic levels in an appropriate relation (level I. Knowledge and understanding, II. Understanding and application, III. Analysis, synthesis, etc.). 2.3 Evaluation process Pupils/students and teachers began using the e-textbooks at the beginning of the 2013/2014 school year. Both pilot projects were continuously monitored. The project monitoring in the second year was performed on three levels: An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Method 13 • obtaining the opinions of the participants with the help of the evaluation questionnaires and interviews • classroom observations • monitoring the effects of using the interactive elements of the e-textbooks in obtaining knowledge through written tests 2.3.1 The monitoring process through questionnaires, classroom observations and interviews All participants answered the questionnaires in April 2015. The collective interviews with teachers were also carried out in April 2015. Classroom observations were performed from October 2014 to April 2015. 2.3.2 The monitoring process of the effects of the use of interactive elements of e-textbooks in acquiring knowledge Monitoring was conducted in two phases. First phase: During the same school day, the teacher enabled/offered/provided the pupils/students in two different departments of the same grade to individually study the same content, which was new to them, from two different media (the pupils/students from one department studied from the printed textbook and the other from an e-textbook or with the help of select interactive elements from the e-textbook). At the end of the hour, the pupils demonstrated their acquired knowledge by taking a written test with tasks. After the end of the school hour, the written tests (tests with tasks) were evaluated by the teacher and/or counsellor, according to uniform evaluation instructions. Second phase: After approximately 2 weeks, the teacher repeated the procedure. This time they used different content, and they also changed the medium. The pupils from class x that used the e-textbook for the first time, used the regular one this time and vice versa. At the end of the hour, the pupils demonstrated their acquired knowledge again by taking a written test. After the end of the school hour, the written tests (tests with tasks) were evaluated by the teacher and/or counsellor. The results from both phases were compared and then sent to the counsellors at the ZRSŠ (National Education Institute of Slovenia). The pupils’ written tests showed their acquired knowledge on three different taxonomic levels. Based on this data, we analysed the effects of the use of the interactive elements of the e-textbooks on independent learning. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 14 Results 3. RESULTS 3.1 The results of the analysis of the data collected with the questionnaire for teachers In this chapter, we have presented the results of the analysis of the data collected by using the questionnaire, which was completed by the teachers involved in the project. The results are presented according to thematic sets, which were included in the questionnaire. A. The use of the e-textbook First, we asked the teachers to evaluate how often they use the e-textbook during lessons. The following table shows the answers of all the teachers as a percentage as well as the answers of teachers in regard to the subject they teach. Table 1: A1 How often do you use the e-textbook during lessons of your subject? Rarely Sometimes Often Very often Never (up to 20% (21-40% (41-60% (more than 61% of the lessons) of the lessons) of the lessons) of the lessons) All teachers (N=123) 5.7 22.0 44.7 17.9 9.8 Mathematics (N=39) 2.6 17.9 51.3 15.4 12.8 English (N=15) 6.7 46.7 33.3 13.3 0 Physics (N=13) 7.7 15.4 30.8 23.1 23.1 Chemistry (N=11) 0 18.2 63.6 9.1 9.1 Natural sciences (N=11) 0 0 54.5 45.5 0 Slovene (N=9) 0 22.2 44.4 22.2 11.1 10 Dr. Sonja Pečjak: Psihološka perspektiva e-učenja, Vzgoja in izobraževanje št. 2–3/2015 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 15 The results of the answers of all teachers regarding the use of e-textbooks during lessons show that the frequency of use “sometimes” (44.7%) and “rarely” (22.0%) pre-vails. In regard to individual subjects, e-textbooks are most often used by teachers of mathematics (sometimes 51.3% frequently 15.4% and very often 12.8%), physics (sometimes 33.3%, often 23.1% and very often 23 1%) and chemistry (sometimes 63.3%, often 9.1% and very often 9.1%) Starting with the assumption that the textbook is intended for the pupils independent acquisition, deepening and consolidation of knowledge and the fact that e-textbooks are designed so that certain interactive parts of the e-textbooks are intended for the teachers use in lessons7, and considering the number of such elements in e-textbooks, we would expect more answers from teachers to be in the categories of “rarely” and “sometimes”. The results of the questionnaire for teachers confirm this assumption. The answers of teachers are concentrated in the category of “sometimes” (44.7%) and “rarely” (22.0%). This lets us conclude that the teachers have recognised the useful-ness of the interactive content in the e-textbooks and advantageously used it during their lessons. In regard to the use of e-textbooks in certain subjects, in order to perform a correct interpretation it is necessary to consider the fact that most of the e-textbooks were created for mathematics, chemistry and physics, as well as the fact that the nature of these three subjects was such that the understanding requires processes that are easiest to explain by means of interactive elements. The results of the questionnaire also confirmed these findings, since the frequency of use was highest for textbooks of mathematics, physics and chemistry. From these results, it can be summarized that the interactive elements in e-textbooks are also useful to the teacher during lessons, especially in subjects that “require” interactivity, simulations, etc. We then asked the teachers about the types of usage of the e-textbooks in the classroom. This is described on next page. 7 Slovenian i-textbooks (http://www.zrss.si/pdf/slovenian-i-textbooks.pdf, Ljubljana, 2014) An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 16 Results Table 2: A2 How often does the following happen during lessons of your subject: Rarely Sometimes Often Very often Never (up to 20% (21-40% (41-60% of the of the of the (more than 61% lessons) lessons) lessons) of the lessons) The e-textbook is only used by you (during explanations, you use elements of the e-textbook in 26.0 35.0 25.2 12.2 1.6 order to present the topic to the pupils/ students) The e-textbook is used by you as well as the pupils/ students (during your explanation, you play the elements 11.4 26.0 35.0 20.3 7.3 of the e-textbook, at the same time, the pupils/students follow the e-textbook on their device) The pupils/students use the e-textbook independently (for 9.8 25.2 33.3 21.1 10.6 individual work, work in pairs or groups) According to the answers of the teachers, the e-textbook was most often used independently by pupils (sometimes 35.0%, frequently 21.1% and very often 10.6%), as well as during the teachers explanation, together with the teacher (sometimes 35.0%, often 20.3%, and very often 7.3%). It was rare that teachers used the e-textbook by them-selves merely as an aid for the explanation, without the pupils’ active participation. The use of textbooks (and e-textbooks) during lessons largely depended on the teachers’ general didactic orientation in the context of promoting or not promoting the independent work of pupils when acquiring new knowledge. Without comparative data on the usage of the classic textbook, it is difficult to interpret these results, accept for the finding that e-learning units, which are defined and designed as “a stimulating, friendly and produc-tive environment for independent learning that stimulates the development of relational learning (Skemp, 1978) with the development of a connected cognitive schemata or with strong emphasis on understanding as an essential component of conceptual knowledge (Hiebert, 1986)”,8 also successfully fulfil this function during lessons. The results indicate that the use of the e-textbook or the interactive units contributes to independent work of pupils and promotes interaction between teachers and pupils. E-textbooks thereby contribute to the assertion of the didactic principle of the activity of the pupils. 8 Slovenian i-textbooks (http://www.zrss.si/pdf/slovenian-i-textbooks.pdf) An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 17 Table 3: A3 How often did pupils use the e-textbook for the purposes of your subject (for schoolwork or homework) to do one of the following things: Rarely Sometimes Often Very often Never (up to 20% (21-40% (41-60% (more than 61% of the lessons) of the lessons) of the lessons) of the lessons) a) Practice skills and procedures 11.4 31.7 39.0 17.1 0.8 b) Analyse data or information 13.0 34.1 30.1 18.7 4.1 c) Observe simulations 14.6 19.5 27.6 29.3 8.9 g) Do their homework 26.0 30.9 26.8 13.0 3.3 h) Look for information 12.2 22.8 32.5 20.3 12.2 i) Work independently (study the topic 12.2 26.0 32.5 21.1 8.1 independently) The teachers responded that during lessons, pupils most commonly used the e-textbook to observe simulations (often 29.3% and very often 8.9%), search for information (often 20.3% and very often 12.2%) and for independent work (often 21.1% and very often 8.1%) and the least for homework assignments (often 13.3% and very often 3.3% or never 26.0% and rarely 30.9%). Even though the information that e-textbooks were most commonly used during lessons to observe simulations and search for information is expected and logical from the point of view of economy (it is easier and quicker to look at a simulation than to perform a test or an experiment by yourself, it is easier and quicker to obtain information from an e-textbook, where it is concentrated in one spot, than it is to search for it using different sources), we cannot interpret such use as nothing but positive. Namely, a real threat exists that all or most of the trials and experiments in the classroom could be replaced by simulations or interactive elements of the e-textbook. This consequently negatively affects the field of acquisition of skills and competences required by experimental work. In the same way, in the search for information, one might limit oneself to only one source (the textbook). This negatively affects the field of acquiring skills and competences for searching for information using different sources. On the other hand, the very rare use of the e-textbook for homework is a very surpris-ing result, considering the fact that the textbook is primarily intended for the pupils/ students independent work (mainly at home). However, considering the fact that most schools did not allow pupils to take the tablets home in order to ensure the security of the tablets, this result is simultaneously expected. The e-textbooks could only be used for homework by those pupils/students that had the necessary equipment at home as well. Another reason may lie in the low number or in some e-textbooks the absence of exercises for the consolidation of knowledge and repetition (collection of exercises) or in the types of exercises that the teachers required the pupils/students to perform. However, we were unable to answer this question on the basis of our data. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 18 Results B. Adequacy of individual e-learning units In the second part of the questionnaire, we asked the teachers to assess the adequacy of the individual learning units of the e-textbook. The table below presents the answers of the teachers who evaluated the individual elements of the learning units of the e-textbook on a five-level scale (1-inadequate, 2-mostly adequate, 3-partially adequate, 4-adequate and 5-completely adequate). The following table presents the average responses of the teachers. Table 4: B1 evaluation of the adequacy of the e-textbook. M SD structure of the textbook 3.72 1.01 professional adequacy and correctness 3.87 1.07 consideration of general didactic principles 3.80 1.05 consideration of subject specific didactic principles 3.77 1.05 inclusion of multimedia elements (multi-sensory learning) 3.67 1.14 compliance with learning objectives, standards of knowledge in the curriculum 3.99 1.13 feedback to pupils/students 3.74 1.17 consideration of the development stage of the pupil/student 3.70 1.05 individualisation and differentiation of content, exercises 3.41 1.12 the possibility of adaptation to pupils/students with special needs 2.98 1.10 consideration of the cultural environment of the pupils/students 3.28 1.13 uniform exterior format and user image 3.70 1.15 age-appropriate language 3.76 1.03 clear and comprehensible text 3.80 1.08 M - arithmetic mean; SD - standard deviation; The adequacy of the learning units was evaluated the worst regarding the possibility of adaptation to pupils/students with special needs (M = 2.98), followed by the consideration of the cultural environment of pupils (M = 3.28) and the possibility of individualisation and differentiation (M = 3.41). All other values were above 3.5, which meant that they were approaching the grade “adequate”. The best values were regarding the professional adequacy and correctness (M = 3.87), consideration of didactic principles (M = 3.80), and the clarity and comprehensibility of the text (3.80). These values were expected, considering the fact that all e-textbooks underwent formal procedures during their process of formation (counsellors, reviewers, editors) and the process of certification by the Expert Council for General Education of Slovenia. Despite the fact that one might expect e-textbooks to be able to be adapted more to pupils/students with special needs (this is enabled by the technology itself, at least for certain categories of pupils/students with special needs), we cannot confirm this assumption, because we do not have the relevant comparative data for classic textbooks. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 19 C. The effects of the use of e-textbooks during lessons In the third part of the questionnaire, the teachers evaluated the effects of the use of e-textbooks during lessons. We asked them to evaluate to what extent the use of the e-textbook, during lessons of their subject, was reflected in various aspects. They compared lessons with the use of the e-textbook with lessons in which they used a classic textbook. Table 5: C1 the effects of the use of the e-textbook during lessons Lower than with The same as with Greater than with a classic textbook a classic textbook a classic textbook motivation of pupils/ students for school work 7.3 30.1 62.6 activity of pupils/ students in class 5.7 33.3 61.0 independence of pupils/ students in school work 5.7 46.3 48.0 attention of pupils/ students in class 19.5 46.3 34.1 motivation of pupils/ students to read 22.0 48.8 29.3 collaborative learning of pupils/students 9.8 52.8 37.4 The following table presents the percentage of teachers that selected a certain value for a certain effect. According to the teachers’ answers, the e-textbook positively affected the pupils’/students’ motivation for school work (62.6% higher than with classic textbooks), the activity of pupils/students during lessons (61.0% higher than with classic textbooks) and the independence of pupils/students in their school work (48.0% higher than with classic textbooks). According to the answers from the teachers it did not have any effect (either positive or negative) on the attention of pupils/students during lessons (46.3% the same as with classic textbooks), the motivation of pupils/students to read (48.8% the same as with classic textbooks) and their collaborativelearning (52.8% the same as with classic textbooks). The increased motivation, activity and independence of the pupils/students during lessons could partially be attributed to the introduction of new technologies into the teaching process and partly to the e-textbook (interactivity). We can probably find the interpretation for the seemingly conflicting results (positive effect on general motivation and no impact on reading motivation) in “studies of the last twenty years on elearning, which show that people better understand and more easily memorise a text if it is written on a piece of paper than a computer screen”, this is also true for modern “digital” generations. This is probably the reason why “modern research shows that people give priority to reading texts on paper”9. Based on these results it is difficult to find unambiguous answers, but we can conclude that the use of e-textbooks and technology has its advantages and disadvantages in various activities of pupils/students in various stages of lessons. 9 Dr. Sonja Pečjak: Psihološka perspektiva e-učenja, Vzgoja in izobraževanje Issue 2-3/2015 (Psychological Perspective of e-Learning, Upbringing and Education) An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 20 Results We also asked the teachers to evaluate to what extent the use of the e-textbook, during lessons of their subject, supported the development of different knowledge, skills and competences of the pupil/knowledge. The teachers answered the questions using a five-level scale (1-does not support, 2-supports a little 3-partially supports, 4- supports 5-greatly supports). The following table presents the average responses of the teachers. Table 6: C2 The effects of the use of e-textbooks on the pupils/students knowledge, skills, competences. M SD extraction of elements from a whole, analysing, abstracting 3.31 0.94 causal and consequential explanation (inductive, deductive, by analogy, etc.) 3.40 0.94 interpreting 3.40 0.94 the use of procedures, models and theories 3.27 1.02 the creation of something new 3.27 1.10 evaluation 3.36 1.13 formation of views 3.15 1.08 argumentation 3.22 1.08 expressing an opinion 3.11 1.10 presentation of a topic to others (oral) 3.24 1.03 a conversation, a discussion on various opinions and arguments 3.11 1.07 the development of teamwork and cooperation 3.20 1.09 M - arithmetic mean; SD - standard deviation; On average, the teachers estimated that the use of the e-textbook during lessons partially supported the development of the skills, competences and knowledge stated in the table, most of all regarding the causal and consequential explanation (inductive, deductive, by analogy) and interpreting (M = 3.40) and least regarding the expres-sion of opinions and a conversation, a discussion on various opinions and arguments (M = 3.11). The development of these skills, competencies and knowledge depended on the teachers’ general didactic attitude. The e-textbook, to the extent that it was used by the teachers during lessons, with its simulations, multimedia, animations, 3D computer generated models, video clips or interactivity in general can contribute the most to causal and consequential explanation and interpreting. The results also confirm this to a certain extent. We then asked the teachers to assess how often the pupils/students use ICT for different activities during lessons. They compared the lessons with the use of the e-textbook with the lessons in which they used a classic textbook. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 21 Table 7: C3 The effect of using an e-textbook on the general use of ICT of pupils/students. How often do pupils/students, Not as often As often More often for the purposes of your subject as with than with than with (in school or at home) use ICT a classic a classic a classic for the following activities: textbook textbook textbook a) searching for information on the internet 7.3 41.5 51.2 b) practising skills and procedures 17.1 52.0 30.9 c) taking tests or handing in homework 15.4 48.0 36.6 d) writing or editing of texts, papers or essays, whereby text editors are used 14.6 53.7 31.7 e) analysing data or information 8.9 59.3 31.7 f) creating a multimedia presentation (using audio or video) 9.8 35.0 53.3 g) accessing sources using the online classroom or the class website 8.1 39.8 52.0 h) collaboration with classmates via e-mail, video-conference, etc. 14.6 52.8 32.5 i) collaboration with classmates or adults outside of class (e.g., with students 21.1 60.2 18.7 from other schools, tutors, etc.) The following table presents the percentage of teachers that selected a certain value for a certain effect. According to the responses of the teachers, the use of the e-textbook did not significantly increase the use of ICT during lessons except when searching for information on the internet (51.2% more often than when using the classic textbook), creating a multimedia presentation (53.3% more often than when using the classic textbook) and accessing sources through the online classroom or the class website (52.0% more often than when using the classic textbook). We can probably ascribe the more frequent searching for information on the internet, creation of multimedia presentations and access to sources on the online classroom to the availability of IC technology of all pupils and students (all pupils/students within the pilot project were equipped with computers and tablets) and not to the use of e-textbooks. This was also indicated by the teachers’ answers for the activities that were more connected to the use of textbooks (e-textbooks as well as classic textbooks), where there were no significant differences in the use of e-textbooks, like taking tests or handing in homework (48% the same with both textbooks), practising skills and procedures (52% the same with both textbooks) and analysing data or information (59.3% the same with both books). At the end of the questionnaire, we asked the teachers to assess the effect of the use of e-learning materials and the e-textbook on their teaching. Like with the previous questions, they compared the lessons with the use of the e-textbook with the lessons in which they used a classic textbook. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 22 Results Table 8: C4 The effect of the use of the e-textbook on the teacher’s way of teaching. Lower that with The same as with Greater than with a classic textbook a classic textbook a classic textbook the teacher’s autonomy in teaching 5.7 79.7 14.6 the diversity of the used didactic strategies 5.7 26.8 67.5 in teaching individualisation and differentiation 12.2 33.3 54.5 of the lessons the possibility of examining and consolidating 8.1 47.2 44.7 knowledge the activity of the teacher when teaching 13.8 54.5 31.7 collaborative learning of pupils/students 10.6 44.7 44.7 The following table presents the percentage of teachers that selected a certain value for a certain effect. The teachers estimated that with the introduction of the e-textbook, they used a number of different didactic strategies (67.5% agree with the answer that the diversity was greater than when using the classic textbook), that such a means of working enabled them greater individualisation and differentiation of lessons (54.5% agree with the answer that the individualisation and differentiation was greater than when using a classic textbook). Most teachers, however, were of the opinion that, due to the introduction of the e-textbook, the autonomy of the teacher was not increased. According to the opinion of most teachers, the activity of the teacher during lessons did not change. The collaborative learning of pupils and students and the examination and consolidation stayed at the same level as when using a classic textbook. Taking into account the fact that the technology in the e-textbook provides a variety of additional approaches (with simulations, multimedia, animations, 3D computer generated models, video clips of experiments or interactivity in general) and that it allows for real-time feedback for pupils/students (regarding the questions and collections of tasks), answers about the positive effects on the diversity of didactic strategies and consequently also on the individualisation and differentiation were expected. We were surprised regarding the result of the examination and consolidation of knowledge, on which the e-textbook, according to the evaluation of the teachers, supposedly did not have an effect, despite the additional options (at least in some e-textbooks, which have extensive collections of exercises, such as in mathematics) enabled by the technology of e-textbooks. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 23 D. The use of e-services during lessons In the last part of the questionnaire, we asked the teachers to evaluate the effects of the use of e-services during lessons. The table below presents the teachers’ answers which evaluated the given statements according to a four-level scale (1-never, 2-rarely, 3-sometimes, 4-often, 5-very often). The following table presents the average responses of the teachers. Table 9: D1 How often do you use the following e-services during lessons of your subject? N M SD Vote casting (e.g. Kliker at http://kliker.sio.si) 123 2,06 0,99 Online surveys (e.g. at http://ankete.sio.si) 123 2,08 0,96 Online classroom (e.g. their own class’s website, or at http://skupnost.sio.si) 123 3,38 1,27 Video conferencing (e.g. at http://vox.arnes.si) 123 1,37 0,66 E-Portfolio (e.g. at http://listovnik.sio.si) 123 1,39 0,71 Loading and watching videos (e.g. at http://video.arnes.si) 123 2,34 1,32 Sending large files (e.g. at http://filesender.arnes.si) 123 1,83 1,15 Saving documents (e.g. at http://mapa.arnes.si) 123 1,98 1,27 Portal SIO (http://www.sio.si) 123 2,71 1,13 Websites (e.g. at http://arnes.si) 123 3,14 1,31 Of these e-services, the teachers sometimes used the online classroom ( M = 3.38) and websites ( M = 3.14), while all other e-services were rarely used. Online classrooms are not a novelty tied to the e-School Bag project. Their use stems from the e-school project, the same applies to the use of a website. The interpretation of these results for the duration of the pilot projects was not acceptable due to the fact that the e-services were developed only during the course of the project and that the participating teachers were informed of each after the development was finished (usually through ICT lessons, which were carried out long-distance), that time ran out for a planned and thorough in-tegration of this service into the didactics of individual subjects and that consequently, the use of e-services during the pilot was left to the creativity of the individual teachers participating. Last but not least, when comparing the use of e-textbooks and e-services, a methodological problem also arises in defining the categories used (1-never, 2-rarely, 3-sometimes, 4-often,5-very often). Using the same categories presumes a comparable frequency of the use of e-textbooks and e-services, which, considering the nature of the individual e-services (for instance Filesender, video conferencing), was not true. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 24 Results 3.2 Results of the analysis of data obtained through the observation of lessons An observation scale was used for the observation of lessons. The scale was developed by the counsellors from the National Education Institute of Slovenia. We included data from 99 observations of lessons in the analysis. The observations were conducted in April 2015 (at the end of the second year of the project). The counsellors from the National Education Institute of Slovenia performed the observations for several subjects and in various departments of the primary and secondary schools that took part in the project. In this chapter of results, we have presented a review of the obtained data. Table 10: The number of observations per subject at the first or the second observation of lessons Subject No. of observations in April 2015 English 10 Biology 1 Native and Civic Culture and Ethics 1 Society 0 Physics 8 Geography 7 Music 6 Home Economics 1 Chemistry 11 Visual Arts 1 Mathematics 28 Natural Sciences 6 Natural Sciences and Engineering 3 German 3 Slovene 8 Engineering and Technology 0 History 5 Total 99 As can be seen from the table above, the majority of observations were made in classes of Mathematics, Chemistry and English. When interpreting the results, we need to consider that the observations were not carried out to the same extent for all subjects. Below, we present a summary table of the values given by the evaluators, divided by items, which, in the rating scale, are related to e-textbooks, e-services and e-content during lessons. The table presents the answers of the evaluators in per cent, for all subjects together according to the individual values. In the far right hand corner of the table, the answers of the evaluators are presented in arithmetic means. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 25 The observation scale was filled in by the counsellor for each school lesson. They rated the individual items on a scale from 0 to 4 where the numbers indicate the following: 0 – it was not possible to observe the activities (e.g. there was no evaluation during observed hours) 1 – not true 2 – partially true 3 – mostly true 4 – completely true Table 11: E-textbooks, e-content, e-materials during lessons. 0 1 2 3 4 April 2015 M 1. In dealing with new content/materials, the teacher used interactive elements from 7.6 6.3 24.7 41.1 20.3 2.70 the e-textbook and other e-materials. 2. For examining the necessary prior knowledge, they used e-services appropriately. 12.2 18.6 43.6 16.7 9.0 1.69 3. For the active participation of the pupils/ students, the teacher included information 8.2 4.1 16.4 47.9 23.3 / technology appropriately. 4. When developing communication and collaborative skills, the teacher encouraged the pupils/students to appropriately use 8.1 31.7 36.6 16.8 6.8 1.44 the different e-materials and e-services. 5. Individualisation/differentiation took place with the support of information technology. 12.0 28.5 40.5 13.3 5.7 1.55 6. When consolidating the acquired knowledge, the pupils/students appropriately used 8.4 9.0 54.8 23.2 4.5 1.84 the e-textbook and e-materials. 7. The teacher prepared the students for independent work and studying with 6.0 5.4 63.3 18.7 6.6 1.93 the e-textbook and e-materials. 8. The teacher uses the e-textbook and e-materials for examinations. 11.9 19.5 45.9 13.8 8.8 1.69 9. The teacher appropriately used e-services for grading. 58.9 0 4.1 23.3 12.3 / The observations in the lessons with regard to e-materials and e-services have shown that the teachers, when dealing with new content/materials, most often used interactive elements from the e-textbook and other e-materials (M=2.70) sometimes they also prepared the students for independent work and studying with the help of the e-textbook and e-materials (M=1.93) and for consolidating the acquired knowledge (M=1 784). Even independent observers confirmed the findings of the questionnaire for teachers, that teachers, during lessons, most of all recognised the value of interactive elements and used them often. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 26 Results When reviewing the assessments of the evaluators in the field of e-textbooks, e-content and e-materials during lessons, we could see that the teachers, according to the assessment of the evaluators, still had a lot of room to use e-content, mostly in the field of the individualisation and differentiation of lessons (M=1.55) and also in the development of communication and the collaborative skills of pupils and students (M=1.44). In these areas of lessons - for the development of these competences of pupils and students - it would be possible to use ICT even more systematically or more often. 3.3 The results of the analysis of data collected by group interviews (focus groups) with the teachers In this chapter, we have presented the results of the analysis of the data collected with the group interview in the form of a focus group with teachers participating in the project. The group interview was partially structured, involving five questions in the form of brief descriptions for discussion and the possibility that the participants could add content at the end that may have been omitted during the interview. The interviews were carried out in focus groups in which teachers were divided according to the subject that they teach. In accordance with the method of content analysis, we extracted relevant content units from the records of the interviews and evaluated them quantitatively depending on the incidence throughout all interviews. The following two tables present the results of the analysis. Table 12: Advantages of the e-textbook and e-content in teaching. f Greater motivation 8 Displaying information, interactivity (videos, animations, etc.) 7 Use in all phases of lessons 6 Individualisation/differentiation of lessons 5 The development of critical thinking 5 Quick access to information 4 The possibility of independent and collaborative learning 4 Variation of lessons 3 Increased activity 3 Actualisation of lessons 3 Immediate feedback 3 Greater creativity of the teacher 3 Always at hand (classic textbooks at home), a lighter bag 3 Access to a repetition of the explanation of the content, a video clip of your own work 3 The development of spatial conception 3 Development of the ability to learn how to learn 3 Communication skills (also skills as a performer, recording, presenting, etc.) 3 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 27 Better results of learning in some of the content 1 Less time spent on certain activities (for example blackboard drawings, etc.) 1 Pupils/students who have difficulty with writing (learning disorders) can type more easily 1 Fast examination of prior knowledge 1 Independent learning from own mistakes 1 Encouragement of communication in a foreign language 1 Improvement of self-esteem of the pupils/students that are not as good in school as others. 1 High-quality materials in terms of content and didactics 1 Opportunity to develop different forms of lessons 1 The possibility of flipped learning 1 Encouragement of self-evaluation 1 Encouragement of divergent thinking 1 Exercises for examination and consolidation 1 Development of criticality towards sources used 1 From the analysis of the group interviews, it is clear that teachers see the advantage of the e-textbook over the classic textbook most often in the increased motivation of pupils/students to work, the interactivity of the e-textbook, its effective use in all phases of the lessons, the greater possibility of individualisation and differentiation of lessons and greater support in the development of the critical thinking of pupils and students. Table 13: Disadvantages of the e-textbook and e-content in teaching. f The teacher needs more time to prepare for the lessons. 8 Greater difficulty in supervising the work of all pupils/students 4 Technical difficulties, errors, etc. 4 The lack of knowledge of ICT amongst teachers, didactics of teaching with ICT 3 They reduce the concentration of pupils with lower learning abilities 3 Similar types of exercises 3 Too much time in front of the screen 2 To easy exercises, lower taxonomy 2 Easy access to the solution of the exercises 2 More time is spent on studying the subject matter 2 Not suitable for assessing knowledge 2 Scant feedback for the teacher on the work of the individual pupil/student 2 When examining knowledge, it is more difficult to use questions of a higher taxonomic level 2 Useless applications 2 They do not allow the addition of footnotes, notes 1 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 28 Results Does not have a search engine 1 Less recorded content in notebooks 1 Pupils/students no longer use any other sources, just the e-textbook. 1 The pupils/students are sloppy in their work 1 Not all children are equally skilled at typing 1 Inferior functional literacy of the pupils/students 1 As a disadvantage of the introduction of the e-textbook into the lessons, teachers most often state that they spend more time preparing for lessons, that it is harder to supervise the work of the pupils/students during lessons, that there are frequent technical difficulties when using the e-textbook or e-content and that teachers lack knowledge in the field of ICT. 3.4 The results of the analysis of data col ected using the questionnaire for pupils/students In this part of the report, we have presented the results of the analysis of the data collected using the questionnaire, which was completed by the pupils/students involved in the project. During the second collecting of data, this questionnaire was completed by 955 pupils and students (April 2015). The results are presented according to the thematic sets of questions that were included in the questionnaire. A. The use of the e-textbook We asked the pupils/students about the types of usage of the e-textbooks in the classroom. Table 14: A2 How often does the following happen during lessons of the subject: Never Rarely Sometimes Often Very often (up to 20% (21-40% (41-60% (more than of the of the of the 61% of the lessons) lessons) lessons) lessons) The e-textbook is only used by the teacher (during explanations, they use elements of the e-textbook 28.7 35.5 21.7 8.6 5.4 in order to present the topic to the pupils/students) The e-textbook is used by the teacher as well as us, the pupils/students (during the explanation, the teacher plays the elements of the e-textbook, 15.0 25.9 25.2 21.9 12.0 at the same time, we, the pupils/ students, follow the e-textbook on our device) We, the pupils/students, use the e-text- book independently (for individual 14.3 28.8 25.2 22.3 9.3 work, work in pairs or groups) An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 29 The responses of the pupils and students indicate that the e-textbook was only rarely used during lessons in such a way that it is actively used only by the teacher for their explanation (28.7% never and 35.5% rarely), whereas the pupils/students would not be active. More often it is used by the pupils/students during the teacher’s explanation, together with the teacher (sometimes 25.9%, often 25.2% and very often 12.0%) and independently by pupils and students (sometimes 25.5%, often 22.3% and very often 9.3%). If we compare this data with the data from the questionnaire for teachers, we see similar relationships: just like the teachers (see table 2), the pupils/students also reported that they are active when using an e-textbook during lessons. We then asked the pupils/students about types of usage of the e-textbooks in the classroom. Table 15: A3 How often did pupils/students use the e-textbook for the purposes of the subject (for schoolwork or homework) to do one of the following things: Never Rarely Sometimes Often Very often (up to 20% (21-40% (41-60% (more than of the of the of the 61% of the lessons) lessons) lessons) lessons) a) practice skills and procedures 17.6 37.3 26.8 12.9 5.4 b) analyse data or information 20.3 34.8 26.8 13.7 4.4 c) observe simulations 29.3 29.6 24.0 11.9 5.1 g) do homework 28.8 25.4 21.1 13.2 11.5 h) search for information 10.3 26.8 27.0 24.5 11.4 i) work independently (study the topic independently) 14.2 29.4 26.6 18.2 11.6 Pupils and students estimated that during lessons, they most commonly used the e-textbook to search for information (sometimes 27.0%, often 24.5% and very often 11.4%) and work independently (sometimes 26.6%, often 18.2% and very often 11.6%). According to their opinion, they used it most seldom for observing simulations (never 29.3% and rarely 29.6%), do homework (never 28.8% and rarely 25.4%) or analyse data or information (never 20.3% and rarely 34.8%). The responses of the pupils and students were quite consistent with the responses of the teachers (see Table 3), they only differ significantly in the item observation of simulations where the students and pupils assessed that this occurred more rarely than with the teachers. It is most likely that this is connected with the difference in the understanding of the concept simulation between teachers and pupils and students. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 30 Results B. The effects of the use of e-textbooks during lessons At the end of the questionnaire, we wanted to know how pupils and students evaluated the effects of the use of the e-textbook during lessons. We asked them to compare lessons with and without the e-textbook at every given criteria. Table 16: The effects of the use of the e-textbook during lessons Lower The same Greater that with as with than with a classic a classic a classic textbook textbook textbook I like using the e-textbook during lessons. 29.8 32.1 38.1 I prefer to get to know new content by myself using the e-textbook. 35.0 33.9 31.2 During lessons, I find it easier to follow using the e-textbook. 33.9 41.5 24.6 I like to read the contents of the e-textbook. 30.6 39.3 30.2 I like to solve exercises in the e-textbook. 24.2 34.4 41.4 I like to do my homework from the e-textbook. 37.8 35.4 26.8 Using the e-textbook, I find it easier to understand the contents that are 29.7 43.8 26.4 discussed in school. I like learning from the e-textbook at home. 50.5 29.6 19.9 The following table presents the percentage of pupils and students that selected a certain value for a certain effect. The table shows that pupils and students primarily found the positive effects of the use of the e-textbook in the form of greater motivation for the use of the e-textbook (I like using the textbook during lessons 38.1% more than with the use of the classic textbook) for solving exercises from the e-textbook (I like solving exercises from the e-textbook 41.4% more than with the use of the classic textbook). For learning new content, doing homework and learning at home, they preferred using the classic textbook (which may be connected to the fact that some pupils/students did not have the appropriate equipment at home). In other activities (following lessons, reading the contents of the textbook, understanding the discussed content) most pupils/students assessed that there was no difference in comparison with the use of the e-textbook. 3.5 The results of the analysis of the data collected using the questionnaire for parents In the next part of the chapter on results, we have presented the results of the data analysis, which was gathered during the second collecting of data, with the help of a questionnaire developed for parents. The questionnaire was taken by 242 parents. We first asked the parents to answer the question, to what extent the use of the e-textbook at home reflected on the work done by their child for school, at home. They compared the child’s work at home with the use of the e-textbook with work done at home using the classic textbook in the form of a book. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 31 A. The use of the e-textbook at home We first asked the parents to answer the question, to what extent the use of the e-textbook reflected on the work done by their child for school, at home. They compared the child’s work at home, with the use of the e-textbook with work done at home, using the classic textbook in the form of a book. Table 17: The effects of the use of the e-textbook at home Lower The same Greater that with as with than with a classic a classic a classic textbook textbook textbook The child likes to read the e-textbook at home. 50.4 31.8 17.8 The child likes do their homework from the e-textbook. 41.7 40.9 17.4 The child likes to learn from the e-textbook at home. 45.0 18.4 16.5 The child is independent when working with the e-textbook. 17.4 47.1 35.5 When using the e-textbook, the child is concentrated and focused on their work. 28.9 46.7 24.4 The child is motivated for schoolwork. 19.0 54.1 26.9 The following table presents the percentage of parents that selected a certain value for a certain effect. Regarding the use of the e-textbook at home, the parents estimated that pupils/students in general use the e-textbook less than the classic textbook, except with regard to the last three items (independence at work, concentration and focus and motivation on school work), where there were no differences in the use of the e-textbook and the classic textbook. The results of the responses of parents were consistent with the responses of pupils/ students (in doing homework and the analysis of data and information at home) or lesser or the same use of e-textbooks at home for work done for school. At first glance, the results show that the e-textbook did not fulfil its function in regard to the work of pupils/students done for school at home, however, it is hard to make any conclusions regarding the reasons based on the current data. A possible interpretation might lie in the fact that the pupils/students did not take the equipment (tablets, computers) home with them, they left it at school, which might mean that at home they only had (or didn’t even have this) IC technology, which they otherwise had at their disposal anyway. B. The use of ICT at home In the second part of the questionnaire, we asked the parents to evaluate the extent to which the use of the e-textbooks reflects on the use of information and communication technology (ICT) in their home or how this is reflected on the family relations with An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 32 Results the child. Parents answered to the statements using a five-level scale (1 - I completely disagree, 2 - I disagree, 3 - I partially agree, 4 - I agree, 5 - I completely agree). The following table presents the average responses of parents. Table 18: The effect of the use of the e-textbook on home life April 2015 (after the second year of the use of e-textbooks) Now that the child is using the e-textbook: M SD a) they spend more time at home on the computer/tablet 3.05 1.17 b) they spend more time at home using social networks (Facebook, Twitter, etc.) using the tablet/computer 2.61 1.20 c) we have to supervise them so that they do not use the computer/tablet too much 2.78 1.32 d) we have to monitor the content that the child searches for on the internet 2.62 1.33 e) they have gained a lot of knowledge in the field of ICT competences 3.36 1.02 f) we as parents are learning to use the tablet as well 2.98 1.28 g) it is easier for them to start on their homework 2.83 1.13 h) they are more independent when doing their homework for school 2.98 1.15 i) they remember the subject matter better 2.84 1.12 M - arithmetic mean; SD - standard deviation; The responses of the parents to the questions on the use of ICT at home show that parents see the effects of the introduction of the e-textbook mainly in the acquisition of ICT competences (M=3.36), but partially also in the increased use of the tablet and in the greater independence in doing homework for school. On average, they do not report additional difficulties in limiting the child’s access to a computer or about the introduction of additional supervision over the content to which the child has access to, which would mainly occur because of the introduction of the e-textbook for lessons. The results show that the introduction of e-textbooks increased the use of the existing IC technology, which the pupil/student already had at their disposal (they did not take the tablets from school home), for doing their school homework. 3.6 The results of monitoring the effects of the use of interactive elements of e-textbooks during lessons 3.6.1 The results of the basic analysis of the data collected Based on the collected results of the pupils and students in the course of their independent work with the e-textbook or the printed textbook, we performed a paired sample t-test with the help of which we compared the work result of each pupil/student when working with the e-textbook with the results of the same pupil/student working An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 33 using a printed textbook. In doing so, we should highlight the fact that the content of the learning units and consequently the exercises, with which we compared the pupils/ students knowledge in working with one or the other media, were different. When creating the exercises, the teachers and the counsellors from the National Education Institute of Slovenia paid attention to making the exercises of both media comparable in difficulty. We should nonetheless consider this data when interpreting the results. Table 19: Basic statistics of the comparison. M N SD SE e-textbook 53.81 527 25.26 1.10 Pair c-textbook (classic) 51.36 527 25.21 1.10 As can be seen from the table above, the results of the pupils and students working with e-textbooks are, on average, higher by 2 percentage points than during work using a printed textbook. The results vary considerably among pupils and students (SD values are very high), which is probably due to the fact that the overall sample covered four different subjects and three different classes. Table 20: The correlation between the results of the pupils’/students’ work using the e-textbook and their work using the printed textbook. N r p Pair e-textbook & c-textbook 527 0.272 0.000 The correlation between the result of the individual pupil or student regarding their work using the e-textbook and their work using the printed textbook is positive (although low), but statistically significant. The result shows that the pupils or students that achieved a better result working with a printed textbook also achieved a better result with the e-textbook and vice versa. Table 21: Results of the paired sample t-test. 95% t df P M SD SE confidence interval Pair e-textbook - c-textbook 2.444 30.459 1.327 -0.162 5.051 1.842 526 0.066 The results of the paired sample t-test show that the difference between the results of working with the e-textbook in comparison with working with a printed textbook are not statistically significant ( t =1.842, p>0.05). Below we have presented the results of the comparison with the paired sample t-test separately for each subject. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 34 Results 3.6.2 The results of the basic analysis of the data collected in Chemistry Table 22: Basic statistics of the comparison for Chemistry. M N SD SE e-textbook 56.43 232 20.793 1.365 Pair c-textbook (classic) 54.85 232 20.563 1.350 As can be seen from Table 22, the results of the pupils and students working with e-textbooks in Chemistry are on average higher by 2 percentage points than when working with a classic textbook. The results vary considerably among pupils and students (SD values are very high), which is probably due to the fact that the overall sample covered three different classes (8th and 9th grade of primary school and the 1st grade of secondary school). Table 23: The correlation between the results of the pupils’/students’ work with the e-textbook and the work with the printed textbook in Chemistry. N r p Pair e-textbook & c-textbook 232 0.276 0.000 The correlation between the result of the individual pupil or student regarding their work using the e-textbook and their work using the printed textbook in Chemistry was positive, although low, but statistically significant. The result showed that the pupils or students that achieved a better result working with a printed textbook also achieved a better result with the e-textbook and vice versa. Table 24: Results of the paired sample t-test in Chemistry. 95% confidence t df P M SD SE interval e-textbook - Pair 1.580 24.876 1.633 -1.638 4.798 0.967 231 0.334 c-textbook Results of the paired sample t-test showed that the difference between the results regarding their work using the e-textbook and their work using the printed textbook in Chemistry are not statistically significant ( t =0.967, p>0.05). 3.6.3 The results of the basic analysis of the data collected in Mathematics Table 25: Basic statistics of the comparison for Mathematics. M N SD SE e-textbook 41.35 143 25.713 2.150 Pair c-textbook (classic) 41.79 143 28.097 2.350 An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 35 As can be seen from the table above, the results of pupils and students working with e-textbooks were, on average, the same as when working with a printed textbook in Mathematics. The results vary considerably among pupils and students (SD values are very high), which is probably due to the fact that the overall sample covered three different classes (8th and 9th grade of primary school and the 1st grade of secondary school). Table 26: The correlation between the results of the pupils’/students’ work using the e-textbook and their work using the printed textbook in Mathematics. N r p Pair e-textbook & c-textbook 143 0.291 0.000 The correlation between the result regarding their work using the e-textbook and their work using the printed textbook in Mathematics was positive, although low, but statistically significant. The result showed that the pupils or students that achieved a better result working with a printed textbook also achieved a better result using the e-textbook and vice versa. Table 27: Results of the paired sample t-test in Mathematics. t df P M SD SE 95% confidence interval Pair e-textbook - c-textbook -0.436 32.102 2.684 -5.743 4.871 -0.162 142 0.871 Results of the paired sample t-test show that the difference between the results of the work using the e-textbook in comparison with the work using a printed textbook in Mathematics are not statistically significant ( t =-0.162, p>0.05). 3.6.4 The results of the basic analysis of the data collected in Slovene lessons Table 28: Basic statistics of the comparison for Slovene lessons. M N SD SE e-textbook 57.79 102 29.730 2.944 Pair c-textbook (classic) 51.38 102 27.394 2.712 As can be seen in Table 28, the results of pupils and students working with e-textbooks in Slovene lessons were on average higher by 6 percentage points than when working with a printed textbook. The results varied considerably among pupils and students (SD values are very high), which is probably due to the fact that the overall sample covered three different classes (8th and 9th grade of primary school and the 1st grade of secondary school). An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 36 Results Table 29: The correlation between the results of the pupils’/students’ work using the e-textbook and their work using the printed textbook in Slovene lessons. N r p Pair e-textbook & c-textbook 102 -0.057 0.570 The correlation between the results of the individual pupil’s or student’s work using the e-textbook and their work using the printed textbook in Slovene lessons was negative, very low, but not statistically significant. Table 30: Results of the paired sample t-test in Slovene lessons. t df P M SD SE 95% confidence interval Pair e-textbook - c-textbook 6.409 41.559 4.115 -1.754 14.572 1.557 101 0.123 Results of the paired sample t-test showed that the difference between the results of the work with the e-textbook in comparison with the work with a printed textbook in Slovene lessons were not statistically significant ( t =1.557, p>0.05). 3.6.5 The results of the basic analysis of the data collected in Geography Table 31: Basic statistics of the comparison in Geography. M N SD SE e-textbook 69.11 50 17.761 2.512 Pair c-textbook (classic) 62.51 50 23.298 3.295 As can be seen from the table above, the results of pupils and students working with e-textbooks in Geography were on average higher by 7 percentage points than when working with a printed textbook. The results varied considerably among pupils and students (SD values are very high), which is probably due to the fact that the overall sample covered three different classes (8th and 9th grade of primary school and the 1st grade of secondary school). Table 32: The correlation between the results of the pupils’/students’ work with the e-textbook and their work with the printed textbook in Geography. N r p Pair e-textbook & c-textbook 50 0.533 0.000 The correlation between the results of the individual pupil or student when the work with the e-textbook and the work with the printed textbook in Geography is positive, medium low and statistically significant. The result showed that the pupils or students that achieved a better result working with a printed textbook also achieved a better result with the e-textbook and vice versa. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Results 37 Table 33: Results of the paired sample t-test in Geography. t df p M SD SE 95% confidence interval Pair e-textbook - c-textbook 6.604 20.425 2.889 0.799 12.409 2.286 49 0.027 Results of the paired sample t-test show that the difference between the results of the work with the e-textbook in comparison with the work with a printed textbook in Geography were not statistically significant ( t= 2.286, p<0.05). In Geography, the pupils and students achieved a significantly better result when doing exercises with the use of the interactive elements in the e-textbook than when using the classic printed textbook. The results of monitoring the effects of the use of interactive elements in e-textbooks during lessons indicated a trend of the better acquisition of knowledge through interactive elements of e-textbooks. In most cases, the results of the pupils and students were higher by a few percentage points in cases where pupils used e-textbooks, the difference was however statistically significant only in Geography. Given the smaller and selectively chosen sample, these, however, do not represent representative and valid conclusions. The results nevertheless suggest the way for further development of e-textbooks and other e-content for schools. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 38 Conclusions 4. CONCLUSIONS This final report presents the results of the monitoring of the project of introducing e-textbooks into lessons in primary and secondary schools. The participants of the evaluation were pupils, students, their teachers and parents. We asked all participants of the project to submit their observations, opinions, and evaluations on the introduction of e-textbooks into lessons in different ways. All participants completed an evaluation questionnaire, which was developed specifically for evaluation purposes. The questionnaire was prepared in three versions - for the teachers, for the pupils/students and for the parents. In addition to the questionnaires, we carried out the evaluation using the observations of lessons. These were carried out by counsellors from the National Education Institute of Slovenia. They recorded their observations using a special scale that was developed specifically for this evaluation. In the context of the evaluation of the project, we also monitored the effects of the use of interactive elements of the e-textbooks on the knowledge of pupils/students. In addition to the quantitative methods of data collection, we used a qualitative method, whereby we encouraged teachers to share their personal experiences in connection with the e-textbook with us, using a semi-structured interview. We thereby compiled an extensive database that we analysed in accordance with the possibilities given by the data, with the intention of col-lating the results. Below, we have presented the main conclusions that can be derived from the analysis of the data. The teachers involved in the project mostly assessed the introduction of e-textbooks to lessons in a positive way. They highlighted the positive effects of the use of e-textbooks regarding the work of the pupils and students, as well as in their own work - teaching. They assessed the interactive elements in the e-textbooks as useful during lessons, especially in subjects that “call” for interactivity, simulations, etc. The use of the e-textbook or interactive units contributed to the individual work of the pupils/ students and promoted interaction between teachers and pupils/students, which contributed to the assertion of the didactic principle of the activities of pupils/students. According to the teachers, the use of e-textbooks during lessons had the most positive effect on the motivation of pupils/students for doing schoolwork, their activity and independence during lessons. They did not perceive significant changes in the attention of the pupils/students during lessons, the motivation for reading and collaborative learning. The increased motivation, activity and independence of pupils/students during lessons can partially be attributed to the introduction of new technologies into the teaching process and partly to the e-textbook (interactivity). An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Conclusions 39 Most teachers assessed that the introduction of the e-textbook enabled them to use many different didactic strategies and that this method of work allowed for greater individualisation and differentiation of lessons. Most of them also believed that, due to the introduction of the e-textbook, the autonomy of the teacher has not increased. According to the opinion of most teachers, the activity of the teacher during lessons has not changed. The collaborative learning of pupils and students and the examination and consolidation stays at the same level as when using a classic textbook. Taking into account the fact that technology in the e-textbook provides a variety of additional approaches (with simulations, multimedia, animations, 3D computer generated models, video clips of experiments or interactivity in general) and that it allows for real-time feedback for pupils/students (with questions and collections of tasks), answers about the positive effects on the diversity of didactic strategies and consequently also on the individualisation and differentiation are expected. The results of the observations of lessons in regard to the use of e-content and e-services showed that the e-textbook is most often used when dealing with new content/ materials, for independent work and learning with the e-textbook and e-materials and for the consolidation of the acquired knowledge. At the same time, the results showed that teachers, according to the assessment of the graders, still have a lot of room for increasing the use of the e-textbook, mostly in the field of individualisation and differentiation of lessons and also in the development of the communication and collaborative skills of the pupils and students. In these areas of lessons - for the development of these competences of pupils and students - it would be possible to use ICT even more systematically or more often. From the analysis of the group interviews, it is clear that teachers see the advantage of the e-textbook over the classic textbook most often in the increased motivation of pupils/students to work, the interactivity of the e-textbook, its effective use in all phases of the lessons, the greater possibility of individualisation and differentiation of lessons and greater support in the development of critical thinking of pupils and students. As a disadvantage of the introduction of the e-textbook into the lessons, the teachers most often state that they spend more time preparing for lessons, that it is harder to supervise the work of the pupils/students during lessons, that there are frequent technical difficulties when using the e-textbook or e-content and that teachers lack knowledge in the field of ICT. Like teachers, pupils and students also noticed their own increased activity during lessons when using the e-textbook in comparison with the use of the classic textbook. Pupils and students estimated that they use the e-textbook during lessons most often to search for information, work independently and more seldom to observe simulations, do homework and analyse data and information. The responses of the pupils and students were quite consistent with the responses of the teachers, they significantly differ only regarding the observation of simulations where the students and pupils assessed that this occurred more rarely than teachers did. It is most likely that this is connected with a difference in the understanding of the concept simulation between teachers or pupils and students. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 40 Conclusions The pupils and students perceived the positive effects of using an e-textbook primarily in the form of increased motivation for the use of the e-textbook in general and for solving exercises from the e-textbook. For getting to know new content, doing homework and for learning at home, they still prefer the use of a classic textbook. In other activities (following lessons, reading the contents of the textbook, understanding the discussed content), most pupils/students assessed that there was no difference in the preference of the use of the e-textbook or the classic textbook. The interpretation of these results is difficult given the fact that the IC equipment, which was necessary in the pilot, was only available to the participating pupils/students in school. Regarding the use of the e-textbook at home, parents assessed that pupils/students generally used the e-textbook less than or as much as the classic textbook. These replies were consistent with the replies of pupils/students (for doing homework and for analysing data and information at home) or the less frequent or the same use of the e-textbooks for doing homework for school. Thereby, at first glance, the results of the domestic use of the e-textbooks show that the e-textbook does not fulfil its function in regard to pupils’/students’ schoolwork done at home, but the fact that pupils/students were not allowed to take the equipment (tablets, computers) home with them means that they only had IC technology at home (or not even that), which was otherwise at their disposal anyway. We can therefore not make such interpretations. The results of monitoring the effects of the use of interactive elements of e-textbooks during lessons indicated a trend of better acquisition of knowledge through interactive elements of e-textbooks. In most cases, the results of pupils and students were higher by a few percentage points in cases where pupils used e-textbooks, the difference was however statistically significant only in the subject of Geography. Given the smaller and selectively chosen sample, we cannot, however, describe these as representative and valid conclusions. The results nevertheless suggest the way for the further development of e-textbooks and other e-content for schools. All projects that were carried out in the field of the introduction of e-content and e-services into schools were development-oriented, which means that they were primarily focused on the development and testing of content and services during lessons. We did, only to a lesser extent, also examin the effects of e-content and e-services on the knowledge and the learning of the students within the framework of these pilot projects. The sample of schools, teachers, parents and pupils/students that were involved in the pilot projects was not representative for Slovenia, therefore the results of the evaluation of the projects cannot be generalised for the entire population of Slovenian pupils/students. The data collected gives us only rough information, based on which we can further develop e-content and e-services. Their actual effects on the knowledge and learning could only be verified using a longitudinal study, covering a representative sample of Slovenian schools that would last at least 5 years, in which the pupils/students and teachers would be monitored. The results of the evaluation of these projects must therefore be understood primarily in terms of collecting the views and opinions of teachers, parents, pupils and students on proven e-content and e-services. In this phase, the results confirmed the findings of other research carried out in this field, that “a critical reflection of the teacher, as a rule, An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools Conclusions 41 results in the finding that students do not need a computer at all times in all subjects, nor for all the contents of the individual subject in order to gain knowledge”10 and, at the same time, indicate the direction that the further development of e-textbooks is not a copy of traditional textbooks (the digitization of traditional textbooks), but the appropriate use of technology for the achievement of new opportunities/added value, such as interactive elements of e-textbooks, access to information and the like. 10 Dr. Sonja Pečjak: Psihološka perspektiva e-učenja, Vzgoja in izobraževanje Issue No. 2-3/2015 (Psychological Perspective of e-learning, Upbringing and education) An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools 42 Sources 5. SOURCES Rutar Leban, T. (2014). Evalvacijska študija pilotnih projektov Uvajanje in uporaba e-vsebin in e-storitev ter Preizkušanje e-vsebin in e-storitev v osnovnih šolah in gimnazijah: vmesno poročilo. Ljubljana: Pedagogical Institute. E-šolska torba, Projektna dokumentacija (2012), Interno gradivo. (E-School Bag, Project documentation (2012), Internal materials.) Povabilo k sodelovanju pri izvedbi pilotnega projekta Uporabe in uvajanja e-vsebin in e-storitev (2013), interno gradivo. (An invitation to participate in the pilot project Use and introduction of e-content and e-services (2013), internal materials.) Povabilo k sodelovanju pri izvedbi pilotnega projekta Preizkušanje e-vsebin in e-storitev (2013), Interno gradivo. (An invitation to participate in the pilot project Testing of e-content and e-services (2013), internal material.) Slovenski i-učbeniki ((http://www.zrss.si/pdf/slovenski-i-ucbeniki.pdf, Ljubljana, 2014)) (Slovenian i-textbooks (http://www.zrss.si/pdf/slovenian-i-textbooks.pdf) Pečjak, S. Psihološka perspektiva e-učenja, Vzgoja in izobraževanje, let. 46, no. 2– 3 Kreuh, N., Sambolić Beganović, A. Na poti k e-kompetentni šoli prek e-šolstva, e-učbenikov in e-šolske torbe. Vzgoja in izobraževanje, let. 46, no. 2– 3, page 29– 35. Hiebert, J. (1986). Conceptual and Procedural Knowledge: The Case of Mathematics. London: Routlege. Skemp, R. R. (1978). Rational understanding and instrumental understanding. Arithmetic Teacher, 26(3), page 9–15. An evaluation study of the pilot projects of the use of e-textbooks in primary schools and secondary schools