122 Sodobna pedagogika/Journal of Contemporary Educational Studies
Klaudija Šterman Ivančič
The effects of the use of modern technologies 
on learning achievement in different upper-
secondary education programmes: The PISA 
2022 study
Abstract: The rapid development of digital technology has had significant effects on the design of 
the educational environment and the integration of digital resources into the teaching and learning 
process. It has not only changed the way students interact with educational content but also the way 
teachers teach. Using data from the PISA 2022 study and a representative sample of 5,591 students 
aged 15, the aim was to examine the extent to which students from different upper-secondary edu-
cation programmes in Slovenia report the availability and use of digital resources for learning and 
leisure both in and out of school, the use of digital resources by teachers in the classroom and which 
aspects of the use of digital resources are significant predictors of achievement in mathematics and 
reading. The results show that, as reported by 15-year-olds, about a quarter of teachers from all upper 
secondary education programmes are unwilling to use digital resources in the classroom. Students 
who attend general gymnasiums report better access to and more frequent use of digital resources, 
both inside and outside of school. In terms of the positive predictors of the use of digital resources for 
academic achievement, more effective and intentional use of digital resources for learning outside 
school appears to be more important for general gymnasium programmes. However, the use of digital 
resources in the classroom by students and teachers appears to be more important for vocational 
education programmes. The predictor with the most negative impact on mathematics and reading 
achievement in all educational programmes is the frequent use of digital resources in leisure time for 
non-educational purposes. In Slovenia, it is important to promote the effective and meaningful use of 
digital resources for educational purposes by academically disadvantaged students, as well as ongoing 
awareness-raising programmes on the pitfalls and negative effects of the excessive use of digital re-
sources in leisure time on learning achievement in all upper secondary education programmes.
Keywords: PISA, digital resources, mathematics achievement, reading achievement, upper secondary 
education programmes
UDC: 37.091.64
Scientific article
Klaudija Šterman Ivančič PhD, Research Associate, Educational Research Institute, Gerbičeva ulica 
62, SI-1000 Ljubljana, Slovenija, e-mail: klaudija.sterman@pei.si
Let./Vol. 75 (141)
Issue 4/2024
pp. 122–138
ISSN 0038 0474

123 
Introduction 
Today, the role of information and communication technologies (hereafter 
referred to as ‘digital resources’
1
) is becoming increasingly important in teaching 
and learning. The rapid development of digital resources has had a significant 
effect on the design of educational environments. The integration of digital re-
sources into the teaching and learning process has not only changed the way stu-
dents interact with educational content, but also the way teachers teach. United 
Nations Educational, Scientific and Cultural Organization (2023) has identified 
digital resources as a social necessity to ensure that education is a basic human 
right, especially in a world experiencing more frequent crises and conflicts. In the 
Slovenian educational system, there has also recently been an increasing empha-
sis on strengthening digital resources in the teaching and learning process and 
the related digital skills of teachers. As part of the European Union’s response to 
the COVID-19 pandemic in 2022, the Ministry of Education of the Republic of Slo-
venia adopted the Digital Education Action Plan 2021–2027, which identifies the 
most important action steps and areas for the development of digital education 
in Slovenia (Ministry of Education, 2023). Consequently, the large-scale Digital 
Teacher Project has been launched in Slovenian preschools, schools and univer-
sities with the aim of training a minimum of 20,000 education professionals and 
managerial employees in digital, sustainable development and financial literacy 
skills (Digitrajni učitelj, 2023).
In line with the changes digital resource use brings to the teaching and learn-
ing process, the effects of digital resources have become an important research 
subject in recent years, both internationally and nationally. Research into dif-
ferent aspects of the use of digital resources and related teachers’ and students’ 
skills has for some time been a major focus of international large-scale assessment 
studies coordinated by the International Association for the Evaluation of Edu-
1   This paper is based on the PISA study , which uses the term ‘digital resources’ instead of ‘infor-
mation and communications technology’. PISA defines ‘digital resources’ as different types of digital 
resources, such as desktop computers, laptop computers, smartphones and tablet computers, as well 
as educational computer software and other digital learning tools.
Šterman Ivančič

124 Sodobna pedagogika/Journal of Contemporary Educational Studies
cational Achievement (IEA)
2
 and the Organisation for Economic Cooperation and 
Development (OECD)
3
. The results of these studies, in which Slovenia
4
 is a par-
ticipant, encompass representative samples and different populations of primary 
and secondary students and teachers. These results provide insights into different 
aspects of the use of digital resources, such as access to digital resources in and out 
of school, students’ attitudes towards digital resources, the use of digital resources 
in learning, teachers’ willingness to use digital resources in the classroom and 
their digital skills (Fraillon and Rožman, 2023; Hooper et al., 2017; Mullis et al., 
2019; OECD, 2023a; Schulz et al., 2022).
According to the OECD (2023), in the participating countries, the PISA re-
sults’ past cycles showed that students’ use of digital resources can have a positive 
effect on reading, mathematics and science PISA scores – however, only if the 
digital resources are used in an effective and moderate way. This further implies 
that there is always a risk of negative effects of digital resource use on learning 
achievement if such resources are used to excess or for purposes unrelated to 
learning or classroom lessons. The results from secondary analyses of the PISA 
2018 study (Šterman Ivančič et al., 2020) also showed that Slovenian 15-year-olds 
who achieved a higher mean PISA score in reading literacy did more reading on-
line and used digital resources to find useful and educational information. 
Studies conducted in other countries that also focused on the relationship 
between the use of digital resources in various contexts (at school, at home, for 
school-related activities and for leisure activities) and learning achievement show 
that access to computers and the Internet improves learning performance when 
used for instruction (Bulman et al., 2016; Fiorini, 2010) and that moderate use of 
digital resources improves achievement. However, excessive use can have nega-
tive effects (Fuchs et al., 2004; Kim, 2023). Similarly, the authors (e.g. Living-
stone et al., 2007; Malamud et al., 2011; Spiezia, 2010) whose studies examined 
the effects of the use of digital resources for in-school and out-of-school activities 
on academic achievement found that digital resources had positive effects when 
used for learning but also negative effects when used excessively for leisure (e.g. 
for playing video games and using social networking sites). Furthermore, studies 
conducted in Slovenia (Šterman Ivančič et al., 2020) and elsewhere (Kim, 2023) 
showed significant differences in the use of digital resources in different contexts 
among different groups of students. For instance, boys and higher socio-economic 
status (SES) students reported more proficient use of digital resources both in and 
out of school.
The results are mixed regarding the effects that teachers’ classroom use of 
digital resources has on students’ learning achievement and on the effectiveness 
of classroom teaching, itself. Banerjee et al. (2007) note that the benefits of digital 
2   International Computer and Information Literacy Study – ICILS, Trends in International 
Mathematics and Science Study – TIMSS, Progress in International Reading Literacy Study – PIRLS 
and International Civic and Citizenship Education Study – ICCS.
3   Programme for International Student Assessment – PISA.
4   In Slovenia, all the listed international large-scale assessment studies are conducted by the 
Educational Research Institute. 
Šterman Ivančič

125 
resource use depend on how digital resources are used in an educational setting 
and that teachers face various challenges when planning to integrate them into 
their teaching. V arious studies (e.g. OECD, 2023a; Redecker , 2017; Trucano, 2005) 
illustrate that without proper planning, the use of digital resources can lead to 
students’ lack of focus and lower learning performance. To plan classroom les-
sons that include digital resources, teachers must be able to identify , evaluate and 
select the resources that best fit their learning objectives, context and pedagogic-
al approach, and to adapt or create new digital resources. Moreover, they must 
be trained in digital resource management so they can share digital resources 
with their students. In the countries that participated in the 2013 OECD TALIS 
international survey , teachers in lower secondary education reported spending an 
average of seven hours per week planning classroom lessons that included digital 
resources (OECD, 2014).
Study’s Aim and Research Questions 
Despite a growing body of research on the links between students’ and teach-
ers’ in-school and out-of-school use of digital resources and learning achievement, 
research has revealed mixed results regarding the use of digital resources in the 
teaching and learning processes and the way they contribute to learning achieve-
ment. Despite the importance of this topic and the availability of representative 
data from international large-scale assessment studies in this field in Slovenia, 
the data are scantily studied, especially in the context of the OECD’s PISA study, 
which provides insights into this field in upper secondary education programmes. 
While future generations of students and teachers are likely to have the oppor-
tunity to use the most advanced digital resources, it is not a given that everyone 
will have access to these resources or that they will use them in a responsible 
and useful way. Research demonstrates significant distinctions between groups of 
students in both the access to and use of digital resources in teaching and learn-
ing. In Slovenia the upper secondary education programme has already been con-
firmed as a significant predictor of both learning achievement and related factors 
in secondary analyses of PISA data (e.g. Šterman Ivančič et al., 2022; Šterman 
Ivančič et al., 2023b; 2023c). Correspondingly, it is important for the study pre-
sented herein to examine the disparities in the use of digital resources in and 
out of school separately by individual upper secondary education programmes. 
Considering the ongoing widespread digital skill training of Slovenian teachers 
and the increased tendency to introduce digital resources into the teaching and 
learning process, it is important for this study to answer the following nationally 
relevant research questions:
1. To what extent do 15-year-old Slovenian students from different upper sec-
ondary education programmes report the availability and use of digital re-
sources for learning and leisure activities, both inside and outside of school? 
The effects of the use of modern technologies on learning achievement in different ...

126 Sodobna pedagogika/Journal of Contemporary Educational Studies
2. To what extent do 15-year-old Slovenian students from different upper sec-
ondary education programmes report teachers’ classroom use of digital re-
sources? 
3. Which aspects of the use of digital resources inside and outside of school sig-
nificantly predict PISA mathematics and reading achievement in different 
upper secondary education programmes in Slovenia?
Methods
To answer the research questions above, secondary analyses of the OECD 
PISA 2022 data were conducted. The research method used (sample, instruments 
and data analysis) is explained in the paragraphs below. 
Sample
In Slovenia, the PISA study covers students aged 15 years and three months 
to 16 years and two months, regardless of the education programme they attend. 
The PISA sampling design is multistage and stratified, and the sample includes 
all upper secondary education programmes in Slovenia. The final sample in our 
analysis includes a representative sample consisting of 5,591 15-year-old students, 
of which 1,486 (27%) students attended a general gymnasium programme, 2,766 
(49%) students attended technical education programmes and 1,339 (24%) stu-
dents attended vocational education programmes of medium duration. Students 
attending vocational education programmes of a short duration were excluded 
from the analyses because not all data on the use of digital resources had been 
collected for them as part of the PISA study.
Instruments
Following the PISA cognitive tests in mathematics, reading and science lit-
eracy, the students completed the PISA Student Questionnaire, which is used to 
identify various accompanying achievement-related factors. Scale-based student 
reports were used for the analysis, describing the frequency of digital resource use 
at school, the availability of digital resources at school, the use of digital resources 
in the classroom, the use of digital resources for school-related activities at school, 
the use of digital resources for school-related activities outside classroom hours 
and the use of digital resources for leisure activities
5
.
To determine the frequency of digital resource use at school, the students 
answered the question, ‘This school year, how often did you use the following 
5   Due to how extensive the scales were, only a few item examples are listed for each scale. The 
complete scales are available in the internationally published questionnaire accessible via the follow-
ing link: https://www.oecd.org/en/data/datasets/pisa-2022-database.html#questionnaires 
Šterman Ivančič

127 
digital resources at school?’ They rated their frequency of use on a five-point scale 
(‘1 – Never or almost never’ to ‘5 – Several times a day’) and rated their use of the 
following digital resources: ‘Desktop or laptop’, ‘Smartphone (i.e. mobile phone 
with Internet access)’, ‘Internet access (except on smartphones)’, ‘Educational 
software, games or apps, other learning tools (e.g. Duolingo or Memrise online 
support)’ etc. 
To determine the accessibility of digital resources at school, the students an-
swered the question, ‘To what extent do you agree or disagree with the following 
statements?’ They rated the items on a four-point scale (‘1 – Strongly disagree’ 
to ‘4 – Strongly agree’). Examples of the items are as follows: ‘There are enough 
digital resources for every student at my school’, ‘There are enough digital de-
vices with access to the Internet at my school’, ‘The school’s Internet speed is 
sufficient’, ‘Teachers at my school have the necessary skills to use digital devices 
during instruction’ etc. 
To determine the use of digital resources in the classroom, the students an-
swered the question, ‘How often do you use digital resources in the following 
classroom lessons?’ They rated the use of digital resources in the following sub-
jects on a five-point scale (‘1 – Never or almost never’ to ‘5 – In every or almost 
every lesson’): ‘Slovene’, ‘Mathematics’, ‘Science (Biology, Chemistry, Physics)’ 
and ‘Computer Science, Information Technology, Informatics or similar lessons’.
To determine the use of digital resources for school-related activities at school, 
the students answered the question, ‘This school year, how often did you use digit-
al resources to conduct the following activities?’ They rated the items on a five-
point scale (‘1 – Never or almost never’ to ‘5 – Every day or almost every day’). Ex-
amples of items are as follows: ‘Create a multi-media presentation with pictures, 
sound or video’, ‘Write or edit text for a school assignment (e.g. using Microsoft® 
Word)’, ‘Find information online about real-world problems or phenomena (e.g. 
climate change, oil spills), measuring the height of a building)’, ‘Analyse data that 
you have collected yourself (e.g. using Microsoft® Excel™)’ etc.
To determine the use of digital resources for school-related activities outside 
classroom hours, the students answered the question, ‘This school year, how often 
did you use digital resources for the following activities?’. They rated the items 
on a five-point scale (‘1 – Never or almost never’ to ‘5 – Every day or almost every 
day’). Examples of items are as follows: ‘Browse the Internet for schoolwork (e.g. 
for preparing an essay or presentation)’, ‘Browse the Internet to follow up lessons 
(e.g. for finding explanations)’, ‘Communicate with my teacher’, ‘Search for infor-
mation on school-related activities or assignments’ etc.
To determine the use of digital resources for leisure activities, the students 
answered the question, ‘During a typical weekday, how much time do you spend 
doing the following leisure activities?’. They rated the items on a six-point scale 
(‘1 – No time at all’ to ‘6 – More than seven hours a day’). Examples of items are 
as follows: ‘Play video-games (using my smartphone, a gaming console, an on-
line platform or apps)’, ‘Browse social networks (e.g. Instagram®, Facebook®)’, 
‘Browse the Internet (excluding social networks) for fun (e.g. reading news, listen-
ing to podcasts and music or watching videos)’, ‘Communicate and share digital 
The effects of the use of modern technologies on learning achievement in different ...

128 Sodobna pedagogika/Journal of Contemporary Educational Studies
content on social networks or any communication platform (e.g. Facebook®, In-
stagram®, Twitter®, emails, chat)’ etc.
In the PISA 2022 sample of Slovenian students, all the scales used demon-
strated adequate internal consistency, with the coefficient alpha values ranging 
between α = 0.72 and α = 0.92 (OECD, 2024).
Data Analysis 
To answer the first and second research questions, the values of internation-
ally comparable indices
6
 and the average PISA reading and mathematics scores 
from the PISA 2022 national database for Slovenia were used. They calculated the 
average values of digital resource use indices in different upper secondary educa-
tion programmes and the percentages of students’ responses to the items about 
the accessibility and availability of digital resources at school. 
To answer the third research question, a linear regression procedure was 
used to identify the effects of digital resource use in explaining PISA 2022 math-
ematics and reading achievement. Pearson’s correlation coefficient was calculated 
using a value of 0.80 as the criterion. It attempted to detect multicollinearity be-
tween the independent (frequency of digital resource use at school, accessibility 
of digital resources at school, use of digital resources in the classroom, use of 
digital resources for school-related activities at school, use of digital resources for 
school-related activities outside of classroom hours and use of digital resources for 
leisure activities) and dependent variables (average PISA 2022 mathematics and 
reading scores). Due to the sample size, the normality of the distribution of the 
residuals of each predictor’s association with the criterion variable was assumed. 
Due to the study’s two-stage sampling, the data were analysed using IBM 
SPSS 27.0 statistical software and the IEA IDB Analyzer Version 5.0.23 tool, 
which allows the appropriate use of student weights (W_FSTUWT) and sampling 
weights to properly estimate the standard error of the parameters in the popula-
tion using the bootstrap resampling method. 
Results
The results of the comparison of digital resource use in Slovenia’s upper 
secondary education programmes with the OECD average are displayed in Table 
1. Compared to their peers from OECD countries, the 15-year-old Slovenians at-
tending all three aforementioned upper secondary education programmes (gen-
eral gymnasium, technical education programmes and vocational education pro-
grammes of medium duration) reported an average or somewhat above-average 
frequency of digital resource use and availability of digital resources at school. 
They also reported the below-average use of digital resources in the classroom for 
6   The index is a standardised value with a mean of 0 and a standard deviation of 1. 
Šterman Ivančič

129 
school-related activities at school and for school-related activities outside school. 
Compared to the OECD average, the most significant deviations in Slovenia are 
the values of the index for the use of digital resources in the classroom in all three 
upper secondary education programmes. All are well below average. Similarly, 
the value of the index for the use of digital resources for school-related activities 
outside school also deviates, especially in technical education programmes and 
vocational education programmes of medium duration. 
A comparison of the results between individual education programmes in 
Slovenia reveals that, compared to the students attending technical education 
programmes and vocational education programmes of medium duration, the stu-
dents attending general gymnasium programmes, on average, reported greater 
availability of digital resources at school and more frequent use of digital resour-
ces for school-related activities outside school. However, they also reported less 
frequent use of digital resources for school-related activities at school and signifi-
cantly less frequent use of digital resources for leisure activities. 
Gym
g
 TEP VEP
M SE M SE M SE
Frequency of DR use at school 0.12 0.02 0.04 0.02 0.05 0.03
Availability of DR at school 0.19 0.02 0.09 0.02 -0.03 0.03
Use of DR in the classroom -0.28 0.02 -0.34 0.02 -0.30 0.03
Use of DR for school-related 
activities at school 
-0.19 0.02 -0.07 0.02 -0.09 0.04
Use of DR for school-related 
activities outside classroom hours 
-0.07 0.02 -0.21 0.02 -0.37 0.03
Use of DR for leisure activities -0.27 0.02 0.08 0.02 0.10 0.04
Table 1: Use of digital resources by Slovenian students in different education programmes compared 
to the OECD mean.
Note: M – the arithmetic mean of an internationally comparable index, SE – standard error, DR – 
digital resources, Gym
g
 – general gymnasium programme, TEP – technical education programme 
and VEP – vocational education programme of medium duration. The values of the internationally 
comparable index are standardised with a mean of 0 (OECD mean) and a standard deviation of 1; for 
availability of digital resources at school, use of digital resources for school-related activities at school 
and outside classroom hours and use of digital resources for leisure activities, the Gimg index value is 
significantly higher than the SSI and SPI index values at the level p ≤ .05.
Table 2 details the percentage of the students’ responses to the items about 
the teachers’ use of digital resources on a scale measuring the accessibility of 
digital resources for learning and teaching at school.
A total of 74% of students attending vocational education programmes of 
medium duration, 78% of students in technical education programmes and 79% 
of students in general gymnasiums reported that the teachers at their school are 
willing to use digital resources in the classroom. Regarding the teachers’ com-
petence at using digital resources in the classroom, there were no differences in 
The effects of the use of modern technologies on learning achievement in different ...

130 Sodobna pedagogika/Journal of Contemporary Educational Studies
the percentages of agreement with the item among students from varying upper 
secondary school programmes. That is, approximately 70% of students from all 
three upper secondary school programmes agreed that their teachers have the 
necessary skills to use digital resources in the classroom.
Gym
g
TEP VEP 
% SE % SE % SE
Teachers at my school have the 
necessary skills to use digital 
devices during instruction. 
70.28 1.55 71.20 1.30 70.15 1.84
Teachers at my school are willing to 
use digital resources for teaching. 
79.48 1.52 78.30 1.22 73.68 1.86
Table 2: Teachers’ use of digital resources for teaching 
Note: % – percentage of students who agree or strongly agree with the item; SE – standard error;  
Gym
g
 – general gymnasium programme; TEP – technical education programme; and VEP – vocation-
al education programme of medium duration; all the differences between the groups did not prove to 
be statistically significant at the level of p ≤ .05.
The analysis of the correlation coefficients between the examined variables 
(Table 3) reveals that, in most cases, the correlation is weak to moderately strong. 
The strongest significant correlation can be observed between the use of digital 
resources for school-related activities at school and the use of digital resources for 
school-related activities outside school and for leisure activities. Of all the vari-
ables examined, the strongest significant positive correlation between the mean 
PISA mathematics and reading score exists with the accessibility of digital resour-
ces at school. The strongest significant negative correlation exists with the use of 
digital resources for leisure activities. 
1 2 3 4 5 6 7 8 M SD
1 - 0.07 0.89
2 0.01* - 0.09 0.92
3 0.00 0.05** - -0.33 0.84
4 -0.01* 0.17** 0.25** - -0.13 0.90
5 0.02* 0.24** 0.17** 0.45** - -0.19 0.80
6 -0.07** 0.03** 0.26** 0.36** 0.25** - -0.05 0.97
7 0.01* 0.10** -0.01* -0.04** 0.04** -0.19** - 485.00 89.00
8 0.01* 0.15** -0.05** -0.09** 0.08** -0.21** 0.76** - 469.00 97.00
Table 3: Values of the coefficient of correlation between the variables examined
Note: 1 – Frequency of digital resource use at school; 2 – Availability of digital resources at school; 
3 – Use of digital resources in the classroom; 4 – Use of digital resources for school-related activities 
at school; 5 – Use of digital resources for school-related activities outside classroom hours; 6 – Use of 
digital resources for leisure activities; 7 – PISA mathematics score; and 8 – PISA reading score.
As a measure of the correlation between variables, the values of the Pearson correlation coefficient were 
calculated; M – arithmetic mean; SD – standard deviation; **p < 0.01; *p < 0.05.
Šterman Ivančič

131 
Furthermore, an analysis of the predictors of PISA 2022 mathematics and 
reading achievement within upper secondary education programmes in Slovenia 
shows that the significant predictors of the use of digital resources differ accord-
ing to the education programme and achievement in mathematics and reading.
As part of the interpretation of mathematics achievement (Table 4), in the 
general gymnasium programme, the use of digital resources for school-related 
activities outside school was confirmed as a significant positive predictor. How-
ever, the accessibility of digital resources at school and the use of digital resour-
ces for leisure activities have been confirmed as significant negative predictors. 
The latter has the largest effect size in interpreting mathematics achievement 
in a general gymnasium programme. A one-unit increase in the index value for 
the use of digital resources for leisure activities is associated with a 26-point de-
crease in the PISA 2022 mathematics literacy score. As part of the interpretation 
of mathematics achievement in technical education programmes, two predictors 
have been confirmed as significant: the use of digital resources in the classroom 
as a positive predictor and, again, the use of digital resources for leisure activities 
as a negative predictor with the largest effect size. In a vocational education pro-
gramme of medium duration, the two above-mentioned predictors have also been 
confirmed as significant for interpreting mathematics achievement. However, the 
effect size of the use of digital resources for leisure activities is not as large as in 
the general gymnasium and technical education programmes. Rather similar to 
the general gymnasium programme, the use of digital resources for school-relat-
ed activities outside school has also been confirmed as a significant predictor of 
mathematics achievement, however, as a negative predictor rather than a positive 
one, which was the case for the general gymnasium programme. Overall, the ex-
planatory model explains 6% of the variability in PISA mathematics achievement 
in the general gymnasium programme, 4% in the technical education programme 
and 3% in the vocational education programme of medium duration. 
The results from the analysis of the significant effects of predictors in inter-
preting the mean PISA 2022 reading test score (Table 5) display a slightly differ-
ent picture. In general, the inclusion of predictors of diverse uses of digital resour-
ces explains more of the variability in reading achievement than in mathematics 
achievement, especially in vocational education programmes of medium duration 
(7%) and technical education programmes (6%). In the general gymnasium pro-
gramme, the included predictors explain 5% of the variability in the PISA 2022 
reading achievement. There were also disparities in the characteristics of the pre-
dictors included. In the general gymnasium programme, the use of digital resour-
ces in the classroom has been confirmed as a significant positive predictor of read-
ing achievement. Meanwhile, the use of digital resources at school and the use of 
digital resources for leisure activities have been confirmed as negative predictors. 
The latter has the largest effect, similar to the effect in mathematics. In technic-
al education programmes and vocational education programmes of medium dur-
ation, the same four predictors have been confirmed as significant for interpreting 
reading achievement. These are the accessibility of digital resources at school and 
the use of digital resources in the classroom as significant positive predictors as 
The effects of the use of modern technologies on learning achievement in different ...

132 Sodobna pedagogika/Journal of Contemporary Educational Studies
well as the use of digital resources for school-related activities outside school and 
the use of digital resources for leisure activities as significant negative predictors. 
Across all three education programmes, the use of digital resources in the class-
room had the largest positive effect on interpreting reading achievement. How-
ever, the use of digital resources for leisure activities had the largest negative ef-
fect, though only in the general gymnasium and technical education programmes. 
This factor has also been confirmed as a significant negative predictor of reading 
achievement in vocational education programmes of medium duration. Nonethe-
less, the use of digital resources for school-related activities outside school has a 
somewhat stronger negative effect on interpreting reading achievement.
PISA Mathematics Score 
B SE
B
β t
B
p
General gymnasium 
Constant 553.95 2.77 199.72 < 0.001
1 2.30 3.73 0.02 0.62 > 0.05
2 -7.74 3.23 -0.07 -2.40 < 0.05
3 0.99 2.74 0.01 0.36 > 0.05
4 -1.63 3.46 -0.02 -0.47 > 0.05
5 9.91 3.74 0.10 2.65 < 0.05
6 -24.04 3.71 -0.23 -6.48 < 0.001
Technical education programmes 
Constant 472.28 2.07 227.99 < 0.001
1 2.72 2.59 0.03 1.05 > 0.05
2 -2.98 2.76 -0.03 -1.08 > 0.05
3 7.40 2.04 0.09 3.63 < 0.001
4 -0.26 2.57 0.00 -0.10 > 0.05
5 2.57 2.29 -0.03 -1.12 > 0.05
6 -12.32 2.11 -0.17 -5.85 < 0.001
Vocational education programmes of medium duration 
Constant 413.76 2.76 150.16 < 0.001
1 4.07 2.87 0.07 1.42 > 0.05
2 3.13 3.46 0.05 0.91 > 0.05
3 8.01 2.80 0.13 2.86 < 0.01
4 0.78 2.66 0.01 0.29 > 0.05
5 -5.39 2.57 -0.09 -2.10 < 0.05
6 -5.62 2.45 -0.11 -2.30 < 0.05
Table 4: Different aspects of the use of digital resources in learning and teaching as a predictor of 
PISA mathematics scores within various education programmes
Šterman Ivančič

133 
Note: 1 – Frequency of digital resource use at school; 2 – Availability of digital resources at school; 
3 – Use of digital resources in the classroom; 4 – Use of digital resources for school-related activities at 
school; 5 – Use of digital resources for school-related activities outside classroom hours and 6 – Use of 
digital resources for leisure activities.
B = unstandardised regression coefficient; SE
B
= standard error of the coefficient; β = standardised 
regression coefficient; t
B
 = t-test; p = p-value; R
2
 = 0.06 for the general gymnasium programme; R
2
 = 
0.04 for the technical education programme and R
2
 = 0.03 for the vocational education programme of 
medium duration.
PISA Reading Score 
B SE
B
β t
B
p
General gymnasium
Constant 537,61 3.13 171,70 < 0.001
1 -7.82 3.55 -0.08 -2.21 < 0.05
2 3.09 3.55 0.03 0.87 > 0.05
3 6.42 2.97 0.07 2.16 < 0.05
4 -2.33 3.28 -0.02 -0.71 > 0.05
5 2.19 4.01 0.02 0.55 > 0.05
6 -18.96 3.51 -0.18 -5.41 < 0.001
Technical education programmes
Constant 458.31 2.49 183,87 < 0.001
1 -5.92 3.13 -0.07 -1.89 > 0.05
2 7.29 3.06 0.07 2.39 < 0.05
3 10.91 2.63 0.13 4.14 < 0.001
4 -1.12 2.54 -0.01 -0.44 > 0.05
5 -5.10 2.50 -0.06 -2.04 < 0.05
6 -12.72 2.43 -0.16 -5.24 < 0.001
Vocational education programmes of medium duration
Constant 392,46 3.77 104.23 < 0.001
1 -3.34 3.38 -0.05 -0.99 > 0.05
2 6.43 3.15 0.08 2.04 < 0.05
3 15.47 2.99 0.21 5.17 < 0.001
4 4.27 3.54 0.05 1.20 > 0.05
5 -8.20 2.95 -0.12 -2.78 < 0.01
6 -5.83 2.65 -0.10 -2.20 < 0.05
Table 5: Different aspects of the use of digital resources in learning and teaching as a predictor of 
PISA reading scores within education programmes
Note: 1 – Frequency of digital resource use at school; 2 – Accessibility of digital resources at school; 
3 – Use of digital resources in the classroom; 4 – Use of digital resources for school-related activities at 
school; 5 – Use of digital resources for school-related activities outside classroom hours and 6 – Use of 
digital resources for leisure activities.
The effects of the use of modern technologies on learning achievement in different ...

134 Sodobna pedagogika/Journal of Contemporary Educational Studies
B = unstandardised regression coefficient; SE
B
= standard error of the coefficient; β = standardised 
regression coefficient; t
B
 = t-test; p = p-value; R
2
 = 0.05 for the general gymnasium programme; R
2
 = 
0.06 for the technical education programme and R
2
 = 0.07 for the vocational education programme of 
medium duration.
Discussion
The main study results show differences in the availability and use of digital 
resources between education programmes in Slovenia. Students attending gen-
eral gymnasium programmes reported the most frequent use of digital resources 
at school, greater availability of digital resources at school and more frequent 
use of digital resources for school-related activities outside school. However, they 
also noted the less frequent use of digital resources for school-related activities at 
school and significantly less frequent use of digital resources for leisure activities. 
These are the very students who, on average, have scored highest in both math-
ematics and reading on the PISA test across all PISA cycles to date (e.g. Šterman 
Ivančič et al., 2023a). This result corroborates with studies (e.g. Bulman et al., 
2016; Fiorini, 2010; Kim, 2023; Livingstone et al., 2007) that found that access to 
and use of digital resources for learning purposes improves learning achievement 
and that more proficient use of digital resources inside and outside school is re-
ported by higher SES students
7
.
Furthermore, the results find that predictors of distinct uses of digital re-
sources explain more variability in reading achievement than in mathematics 
achievement. This is particularly true for vocational education programmes of 
medium duration. In general gymnasium programmes, the factors included ex-
plained more variability in mathematics achievement. Once again, these results 
show that the use of digital resources has different effects on different school sub-
jects and that these effects also differ among different groups of students. These 
results are consistent with meta-studies (e.g. Cheung et al., 2013; OECD, 2023a; 
Tamim et al., 2011), whose authors conclude that the use of digital resources in 
the teaching and learning process has more positive effects on learning how to 
read than on learning mathematics. Learning mathematics often requires struc-
tured and sequential problem solving, so that traditional teaching approaches 
often prove to be more beneficial, and greater positive effects are observed in stu-
dents with lower reading achievement.
This examination of the predictors of mathematics and reading achievement 
has revealed that frequent use of digital resources for leisure activities, such as 
browsing social networking sites, browsing the Internet for fun or communicating 
and sharing digital content on social networks, has significant negative effects on 
mathematics and reading achievement for both high- and low-achieving students. 
This result is consistent with the findings of other OECD studies and analyses, 
where such effects have been found in various participating countries (e.g. OECD, 
2015; Vigdor et al., 2014). In contrast, the use of digital resources in the classroom 
7  According to PISA data, the students attending general gymnasium programmes report, on 
average, a much higher socio-economic status than the students attending vocational education pro-
grammes of medium duration (e.g. Šterman Ivančič et al., 2023a).
Šterman Ivančič

135 
appears important for improving mathematics and reading achievement, especial-
ly for students with lower PISA scores. Those who, on average, score the highest 
benefit more from the use of digital resources for learning purposes outside school. 
The role of using digital resources in the classroom and promoting mathematics 
and reading achievement cannot be ignored. However, the results also show the 
importance of developing digital skills in lower achieving students so they can use 
digital resources more effectively outside school as a tool to help them learn and 
find useful content for the classroom. Similarly, Fairlie et al. (2013) examined the 
effects that digital resource training for educational purposes had on students 
from different backgrounds and with different learning achievements. They found 
that the training was highly beneficial for lower achieving students, improving 
both their homework completion rates and overall academic achievement. 
Our study revealed no significant distinctions in the use of digital resour-
ces for learning and teaching between individual education programmes. Fifteen-
year-old Slovenians from all three education programmes reported significantly 
less frequent use of digital resources in the classroom for Slovene, mathematics, 
science and computer science subjects than their average OECD peers. Further-
more, 15-year-old Slovenian students reported that approximately 30% of the 
teachers, regardless of the education programme they teach, lack the necessary 
skills to use digital resources and approximately a quarter of the teachers in these 
upper secondary education programmes are unwilling to use digital resources in 
the classroom. This result is similar to the OECD average. In 2022, approximate-
ly 74% of teachers reported using a variety of digital resources in their teaching 
process (OECD, 2023b).
Conclusion 
The study presented in this paper is based on PISA results. Because PISA 
is a cross-sectional study, conclusions cannot be drawn about the cause-and-ef-
fect relationships between the use of digital resources inside and outside school 
and PISA mathematics and reading scores. Moreover, PISA’s data are based on 
reports from 15-year-olds, while data on teachers’ use of digital resources in the 
teaching process are not available for Slovenia. The results also illustrate that the 
explanatory power of the examined mathematics and reading achievement models 
is relatively low. 
Despite these methodological reservations, the results obtained from a rep-
resentative sample of Slovenian students suggest differences in the availability 
and use of digital resources between upper secondary education programmes in 
Slovenia, with the students attending the general gymnasium programme being 
at an advantage. They not only reported better availability of digital resources at 
school but also used digital resources for learning outside school in a more effect-
ive and intentional way, which has a significantly positive impact on their higher 
learning achievement in mathematics and reading. This, however, is not the case 
for students attending vocational education programmes of medium duration. For 
The effects of the use of modern technologies on learning achievement in different ...

136 Sodobna pedagogika/Journal of Contemporary Educational Studies
these students, it is more important that their teachers or they, themselves, use 
digital resources in the classroom, which has positive effects on their mathematics 
achievement and even more so on their reading achievement. Notably, regardless 
of the education programme that students attended, both mathematics and read-
ing achievements are largely negatively affected by the frequent use of digital 
resources for leisure activities, that is for non-educational purposes. 
In conclusion, the use of digital resources in the teaching and learning pro-
cess has different effects on achievement in different subjects and within different 
groups of students. Its positive effects cannot be generalised to all areas of learn-
ing or all groups of students. It is important to increase the availability of digital 
resources and improve the digital skills of students attending technical and voca-
tional upper secondary education programmes. This would allow them to make 
meaningful and effective use of digital resources for educational purposes both 
inside and outside of school. However, it is important to promote ongoing aware-
ness-raising programmes on the pitfalls and the negative effects of the excessive 
use of digital resources during leisure time in all upper secondary education pro-
grammes in Slovenia.
References 
Banerjee, A. V ., Cole, S., Duflo, E., & Linden, L. (2007). Remedying education: Evidence 
from two randomized experiments in India. The Quarterly Journal of Economics, 
122(3), pp. 1235–1264, https://doi.org/10.1162/qjec.122.3.1235. 
Bulman, G., & Fairlie, R.W . (2016). Chapter 5 - Technology and education: Computers, 
software, and the internet. In: E. A. Hanushek, S. Machin and L. Woessmann (eds.). 
Handbook of the economics of education. Elsevier, pp. 239–280.
Cheung, A., & Slavin, R. E. (2013). The effectiveness of educational technology applications 
for enhancing reading achievement in K-12 classrooms: A meta-analysis. Educational 
Research Review, 9, 88–113.
Digitrajni učitelj (2023). Projekt Digitrajni učitelj. Accessible at: https://digitrajni.si/ (Ac-
cessed on 23.6.2024).
Fairlie, R. W ., & Robinson, J. P . (2013). Experimental evidence on the effects of home com-
puters on academic achievement among schoolchildren. American Economic Journal: 
Applied Economics, 5(3), 211–240.
Fiorini, M. (2010). The effect of home computer use on children’s cognitive and non-cogni-
tive skills. Economics of Education Review, 29(1), pp. 55–72.
Fraillon, J., & Rožman, M. (ed.). International computer and information literacy study 
2023: Assessment framework. Accessible at: https://www.iea.nl/sites/default/
files/202312/20231221%20ICILS2023_Assessment_Framework__Final_0.pdf (Ac-
cessed on 14.7.2024).
Fuchs, T ., & Woessmann, L. (2004). Computers and student learning: Bivariate and mul-
tivariate evidence on the availability and use of computers at home and at school. 
CESifo Working Paper Series 1321.
Hooper, M., Mullis, I. V . S., Martin, M. O., & Fishbein, B. (2017). Chapter 3: TIMSS 2019 
context questionnaire framework. In: I.V .S. Mullis and M. O. Martin (ed.). TIMSS 
2019 assessment frameworks. IEA: Boston College, pp. 59–78. 
Šterman Ivančič

137 
Kim, M., & Kim, H. (2023). Profiles of students’ ICT use in high-performing countries in 
PISA 2018. Computers in the Schools, 40(3), 262–281. https://doi.org/10.1080/073805
69.2023.2180338 
Livingstone, S., & Helsper, E. (2007). Gradations in digital inclusion: Children, young peo-
ple and the digital divide. New Media and Society, 9(4), 671-696. 
MVI (2023a). Akcijski načrt digitalnega izobraževanja 2021 – 2027. Accessible at: https://
www.gov.si/assets/ministrstva/MVI/SDIG/Akcijski-nacrt-za-digitalno-izobrazevanje_
ANDI-2023_web.pdf (Accessed on 23.6.2024).
Malamud, O., & Pop-Eleches, C. (2011). Home computer use and the development of hu-
man capital. The quarterly journal of economics. 126, 987–1027.
Mullis, I. V . S., Martin, M. O., & Liu, J. (2019). Chapter 2: PIRLS 2021 context question-
naire framework. In: I.V .S. Mullis and M. O. Martin (ed.). PIRLS 2021 assessment 
frameworks. IEA: Boston College, pp. 27–56.
OECD (2014). TALIS 2013 results: An international perspective on teaching and learning.
 Paris: OECD Publishing. 
OECD (2015). Students, computers and learning: Making the connection. Paris: OECD 
Publishing.
OECD (2023a). PISA 2022 ICT Framework. Accessible at: https://www.oecd-ilibrary.org/
docserver/9bd299c1en.pdf?expires=1726679561&id=id&accname=guest&check-
sum=F3A6EF7DC2D58EE5064A847B190FD6FD (Accessed on 14.7.2024).
OECD (2023b). PISA 2022 Results (Volume II): Learning During – and From – Disruption. 
Paris: OECD Publishing. 
OECD (2024). PISA 2022 Technical Report. Paris: OECD Publishing.
Redecker, C. (2017). European framework for the digital competence of educators:
 DigCompEdu. Accessible at: https://doi.org/10.2760/159770 (Accessed on 14.7.2024).
Schulz, W ., Fraillon, J., Losito, B., Agrusti, G., Ainley, J., Damiani, V ., & Friedman, T . 
(2022). ICCS Assessment Framework. Accessible at: https://www.iea.nl/sites/de-
fault/files/202206/IEA%20International%20Civic%20and%20Citizenship%20Educa-
tion%20Study%202022%20Assessment%20Framework.pdf (Accessed on 14.7.2024). 
Spiezia, V . (2011). Does computer use increase educational achievements? Student-level 
evidence from PISA. OECD Journal: Economic Studies, 1, pp. 7–17. 
Šterman Ivančič, K., & Puklek Levpušček M. (2020). Branje mladih leta 2009 in leta 2018 
ter razlike v branju glede na spol in izobraževalni program. Šolsko polje, 31(1–2), pp. 
11–33.
Šterman Ivančič, K., & Štremfel, U. (2022). Globalne kompetence in trajnostni razvoj: slov-
enski učenci in učenke v raziskavi PISA. Sodobna pedagogika, 73(2), pp. 41–57.
Šterman Ivančič, K., & Mlekuž, A. (ed.) (2023a). Program mednarodne primerjave dosežkov 
učencev in učenk – PISA 2022 Nacionalno poročilo s primeri nalog iz matematike. 
Ljubljana: Pedagoški inštitut.
Šterman Ivančič, K., & Štremfel, U. (2023b). Globalne kompetence slovenskih učencev in 
učenk: konceptualni in empirični vpogledi. Ljubljana: Pedagoški inštitut. Accessible 
at: https://www.doi.org/10.32320/978-961-270-349-3 (Accessed on 5.9.2024).
Šterman Ivančič, K., Štremfel, U., Peras, I., & Japelj Pavešič, B. (2023c). Občutek pri-
padnosti šoli, dobrobit in učna uspešnost učencev in učenk: vpogledi mednarod-
nih raziskav znanja. Ljubljana: Pedagoški inštitut. Accessible at: https://www.doi.
org/10.32320/978-961-270-355-4 (Accessed on 5.9.2024).
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P . C., & Schmid, R. F . (2011). What 
forty years of research says about the impact of technology on learning: A second-or-
der meta-analysis and validation study. Review of Educational Research, 81(1), 4–28.
The effects of the use of modern technologies on learning achievement in different ...

138 Sodobna pedagogika/Journal of Contemporary Educational Studies
Trucano, M. (2005). Knowledge maps: ICT in Education knowledge maps. Accessible at: 
https://files.eric.ed.gov/fulltext/ED496513.pdf (Accessed on 14.7.2024).
UNESCO (2023). Digital education: What you need to know. Accessible at: https://www.
unesco.org/en/digital-education/need-know (Accessed on 14.6.2024)
Vigdor, J. L., Ladd, H. F ., & Martinez, E. (2014). Scaling the digital divide: Home computer 
technology and student achievement. Economic Inquiry, 52(3), 1103–1119.
Klaudija ŠTERMAN IVANČIČ (Pedagoški inštitut, Ljubljana, Slovenija)
UČINKI RABE SODOBNIH TEHNOLOGIJ NA UČNE DOSEŽKE V RAZLIČNIH SREDN-
JEŠOLSKIH IZOBRAŽEVALNIH PROGRAMIH: RAZISKAVA PISA 2022
Povzetek: Hiter razvoj digitalne tehnologije ima pomembne učinke na oblikovanje izobraževalnega 
okolja, povezovanje digitalnih virov s procesom poučevanja in učenja pa ni spremenilo le načina inter-
akcije učencev in učenk z izobraževalnimi vsebinami, temveč tudi način poučevanja učiteljev. V študiji 
smo želeli na podatkih raziskave PISA 2022 in reprezentativnem vzorcu 5591 dijakov in dijakinj, star-
ih 15 let, preučiti, v kolikšni meri dijaki in dijakinje iz različnih srednješolskih izobraževalnih pro-
gramov v Sloveniji poročajo o razpoložljivosti in uporabi digitalnih virov za učenje inprosti čas v šoli in 
zunajnje, o uporabi digitalnih virov med profesorji pri pouku in kateri vidiki uporabe digitalnih virov 
značilno napovedujejo dosežke iz matematike in branja. Rezultati kažejo, da po poročanju 15-letnikov 
približno četrtina profesorjev v vseh srednješolskih izobraževalnih programih ni pripravljena upora-
bljati digitalnih virov pri pouku. Dijaki in dijakinje iz splošnih gimnazij poročajo o boljši dostopnosti in 
pogostejši uporabi digitalnih virov v šoli in zunajnje. Z vidika pozitivnih učinkov uporabe digitalnih vi-
rov na učne dosežke se v programih splošne gimnazije kot pomembnejša kaže bolj učinkovita in načrt-
na uporaba digitalnih virov za učenje zunaj šole, v programih srednjega poklicnega izobraževanja pa 
uporaba digitalnih virov med dijaki in dijakinjami ter profesorji pri pouku. Na dosežke iz matematike 
in branja v vseh izobraževalnih programih najbolj negativno učinkuje pogosta raba digitalnih virov v 
prostem času, ki ni v izobraževalne namene. V Sloveniji se kot pomembno kaže spodbujanje učinkovite 
in smiselne uporabe digitalnih virovv izobraževalne namene pri učno šibkejših dijakih in dijakinjah 
ter spodbujanje programov stalnega ozaveščanja o pasteh in negativnih učinkih prekomerne uporabe 
digitalnih virov v prostem času na učne dosežke v vseh srednješolskih izobraževalnih programih.
Ključne besede: raziskava PISA, digitalni viri, matematični dosežki, bralni dosežki, srednješolski 
izobraževalni programi
Elektronski naslov: klaudija.sterman@pei.si
Šterman Ivančič