The Timeless Questions About Educational Quality
Mojca Straus
Concerns about the quality of education systems have now substantially shifted from the questions about the quantity and quality of resources, such as school buildings and accessibility, to the questions about the outputs of the educational process, such as student achievement. Achievement has become one of the key indicators used in evaluating the quality of education systems. Furthermore, these questions are not constrained to the local contexts but are globalized in the sense that the outputs of educational systems, working in different societal and economical contexts, are compared. To address the comparative information needs in the process of the educational quality control, several large scale assessments have been launched in the last decades primarily to provide an information base from which the hypotheses about stability and change in education can be tested.
Some questions remain the same throughout these decades, such as how well do students in a particular country perform in comparison with students from other countries, do they reach expected levels of achievement and what should be expected of them. Further questions pertain to the methodology of the studies and validity of the usage of their results in more general contexts. The quality of an education system proves to be a complex concept that needs constant attention at all levels of the system. This thematic issue is devoted to findings emerging from the latest cycle of the Programme for International Student Assessment (PISA). Slovenian and foreign authors present views and reactions to the PISA methodology and results in the efforts for assessing the quality of the education systems in their respective countries.
In his article, Darko Strajn discusses relatively recent criticisms of PISA. The criticisms focus on the ranking of results that inscribe PISA as the foun-
7
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
dations of the neoliberal market competition entering the education field. Since the initiation of PISA, there were many discussions about the impact of the study's ranking results on the educational policies and processes around the world. In his analysis, the author explores different views on these impacts on the general understanding of the meaning and working of education as well as educational policy development.
In the next article, Urška Štremfel addresses the impact of Slovenia's below average results in reading literacy on the country's educational policy. Using policy analysis, the author provides insight into the first steps of the process of improving the Slovenia's PISA results. The author discusses the importance of having nationally defined educational priorities and goals in order to be able to actually derive a well-defined policy problem and to find the appropriate policy solution to this problem, for example by drawing lessons from the successful results of other participating countries.
One of the countries from which Slovenia might decide to learn from is Germany. In the last twelve years since the first, so-called PISA shock in 2000, Germany has successfully improved its PISA results. In their article, Christine Saelzer and Manfred Prenzel describe three major aspects of Germany's educational development; a thorough diagnosis of the problems in the country's educational system, an intense discourse between all relevant actors, and the implementation of nationwide, overarching programmes to improve teaching and learning. These elements and their impact on German students' PISA results are analyzed. Based on the PISA 2012 results, it is evident there has been a positive educational development in Germany.
Another country, Canada, has been considered very successful in PISA since its beginning in 2000. However, the recent downward trend in the country's results have initiated the call for action. The issues around Canada's PISA results and reflections of different educational actors are presented by Pierre Brochu. The author analyzes the important considerations in the efforts of finding the appropriate levers for changing the observed negative trend in Canada's student achievement.
In the United States of America, the educational policy is developed at the state level. Maria Stephens and Anindita Sen address considerations arising when three U.S. states - Connecticut, Florida, and Massachusetts - derived comparisons of states' results from the PISA data as well from data of other international studies. When different assessments sometimes indicate different or even contradicting results about the educational quality, the important question is what specific factors might explain the observed differences.
8
M. STRAUS ■ THE TIMELESS QUESTIONS ABOUT EDUCATIONAL QUALITY
Ana Kozina and Ana Miekuz studied the relationship between PISA 2012 mathematics achievement and attribution styles. In their article, they use national as well as international perspective for investigating students' attributions of causes for success and failure on the PISA 2012 mathematics achievement test in relation to actual test score. They conclude that attribution for success should be considered in educational setting for example in communicating praises for students' success in a manner promoting effort.
In the final article, Mojca Straus explores the roles of socio-economic background and mathematics-related attitudinal factors in explaining achievement in mathematics literacy of the PISA 2012 study for Slovenia in comparison with Germany, Canada and the United States. Mathematics-related self-beliefs are shown to be stronger predictors of achievement than students' drive and motivation and similarities are observed between the Slovene and German students' responses as well as between the Canadian and the United States students' responses.
The articles in this issue show that data from international assessments of student achievement represent a rich source of information on education systems in the world. However, thorough understanding of the design, methodology and implementation of the assessments is of vital importance for making valid and useful interpretations of the results. The general steps in conducting an international comparative assessment are that participating countries agree on the population of students and the curriculum domain to be assessed, and on an instrument to assess achievement in the chosen domain. The instrument is administered to a representative sample of students in each country and comparative analyses of the data are carried out. These analyses are intended to provide information about the educational quality in the form of comparisons of students' scores or sub-scores on an international achievement test. An important part of this is understanding the reasons for observed differences between and within the countries from the collection of the background data, especially in the areas where weaknesses in achievement are identified.
The story of educational quality control does not end with the publication of the international or national comparisons of participating countries' results. After the PISA 2012 results were published in late 2013, countries started with additional qualitative and/or quantitative studies designed to unravel the origins of the observed weaknesses in order to set up and carry out the appropriate remedial actions. The contributions in this issue show examples of such analyses and the interpretations of the findings. It is shown that the internationally comparative data are most often used for the functions of descriptive comparisons and trend analyses
9
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
and that it is more difficult to provide answers about the causes of any observations. When the results of the studies are used, one needs to be careful in drawing conclusions. There is an abundance of caveats that could diminish the validity of these conclusions ranging from the sampling design and response rates, coverage of domain and instrument design, data collection procedures, motivation of students and other respondents, technical procedures in data analysis and, not least, inappropriate causal inferences.
As evident from the contributions in this issue, the data from an assessment of student achievement do not, by themselves, convey messages about the quality of education or evaluation of the reforms that have been implemented. The data collected need to be interpreted with a reference to relevant comparisons, for example to the goals of education in a particular country or to the results of other countries. However, setting absolute standards in education is difficult. To try to set realistic standards for educational system comparisons with other relevant countries are essential. As shown in this issue, this is important in the largest education systems in the world and even more so in Slovenia.
The overall problem with analyses of the assessment data is how to address the imminent questions on the educational quality and effectiveness without reporting information that is easily misunderstood and/or misused. It is very difficult to determine abundant factors within or outside the education system that influence achievement. Moreover, conclusions from an assessment rarely offer clues about causal inferences. They can, however, be useful as circumstantial support for the conclusions about the determinants of achievement or as a source of inspirations for finding possible levers of improvement in further research. There are, nonetheless, important reasons for the usefulness of such studies. Not unimportantly, assessments are relatively inexpensive compared to other aspects of managing education, such as implementing curriculum changes that involve substantial professional development of teachers. Further, it is easier to mandate assessment requirements at the system level than it is to take actions that involve actual change in what happens inside the classroom. Such studies are therefore useful for getting the overall picture of the status of the things in education. And, as a consequence of media attention given to the international assessments, international studies can help education to become a priority among the areas that need policy makers' attention.
10