Slovenia on its Own Way Towards Improving PISA Results
Urska Stremfel
Introduction
PISA (Programme for International Student Assessment) becomes a prevalent assessment of the national education systems in the last decade (Hopmann et al., 2007; Pereryra et al., 2011; Meyer and Benavot, 2013). PISA results, presented in comparative achievement scales, provide an insight into how one educational system performs in comparison to other systems and also how one educational system contributes to the achievement of common goals of particular group of participating countries (e.g. European Union (EU) member states together decided a benchmark to have less than 15% of low achievers1 in PISA by 2020) (Council of the EU, 2009). Since PISA results and results of other international comparative assessment studies2 often becomes incorporated in the national educational targets, PISA also helps to identify how successfully participating countries follow their national priorities and goals.3 There is one additional insight that PISA allows. The design of PISA, which is conducted in cycles, enables the monitoring of changes in students' outcomes over time. Such changes indicate how successful education systems have been in developing the knowledge and skills of
1	PISA provides a profile of students' performance using six proficiency levels. The low-achievers are students, who do not reach the proficiency level 2, which present a baseline level of literacy at which students begin to demonstrate the competencies that will enable them to actively participate in life situations (OECD, 2010a).
2	E.g. Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS).
3	E.g. Slovenian White Paper on Education (2011, p.25) states "At the state level we need to state and map out a clear path towards the goal, that performance of Slovenian students in international comparative assessment studies are at the top, that mean at least in the upper third of the students' achievement of the developed countries".
29
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
15-year-olds. All countries seeking to improve their results can therefore draw and learn lessons from those that have succeeded in doing so since 2000, when the PISA was first conducted (OECD, 2010a, p.13).
The importance that PISA has gained in the assessment and development of national educational systems is often understood in terms of transnational policy making (Meyer and Benavot, 2013). If we understand the policy making as the solving the policy problems of / for society (Lasswell, 1951), we can also argue that it can be understood as transnational problem solving (Scharpf, 1997). That means that PISA helps participating countries to understand the weakness of their national educational systems (in international comparative perspective) and also provide the environment for finding the right solution of perceived problem. Despite some theoretical reservations towards considering comparative achievement scales as the legitimate source of policy making (e.g. Kodelja, 2005) and exploiting their results for politically motivated changes at the national level (e.g. Stremfel, 2013), PISA has become widely accepted that these comparative achievement scales (called also league tables) present an important source of the identification of national policy problems and finding policy solutions in participating states (see e.g. Grek, 2010). As such comparative achievement scales, if appropriately used, can present an important source not only for the assessment, but also for the development of national educational systems.4 Although one of the formally stated goals of PISA is to create an internationally comparative evidence base for educational policy development and implementation (Wiseman, 2013, p.304), Waldow (2009) recognized that headline news about PISA is often more about "shock"5 over the assessment results than what the assessment information contributes to discussions about long-term educational reform and improvement.
Theoretical and empirical researches (see Stremfel, 2013) show that participating countries become especially attentive to the PISA results when they perform below international (OECD, EU) average. That effect was experienced also in Slovenia. When the PISA 2009 reading literacy results were published and for the first time since Slovenia had been participating in international comparative assessment studies, it showed that Slovenian students perform below international (OECD, EU) average, the perception of the Slovenian educational system as a successful system
4	For more theoretical insight about the role the evaluation plays in the development ofpub-lic policies see Kustec Lipicer (2009).
5	Phillips and Ochs (2003) explain that education policy shock happens when there is a deviation from the norm, often involving mediocre or low performance (i.e. below expectations).
30
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
was marred at the level of experts, policy makers, practitioners and general public (Interviews by author, 2012). PISA 2012 results confirmed the underperformance of Slovenian students in reading literacy and emphasized the need for improvement of the performance of Slovenian students in that domain.
The aim of the article is through the understanding of PISA as transnational policy making, using the Slovenian PISA 2012 results, is to show how the policy problem of below average results is identified by participating member states and to illustrate how the policy solutions for the improvement of students' performance in PISA could be found. In order to illustrate the policy framework of improving PISA results, the article as a case study takes into consideration PISA reading literacy results (the domain in which Slovenia perform below OECD and EU average) and students performance at the Proficiency level 2 (the level which Slovenia together with other EU member states chose for defining a common
benchmark "to reduce the percentage of low-achieving students to 15% by
»\ 6 2020 ).
A research question the article addresses is "How to find a way towards improving Slovenia's PISA results according to the concept of transnational policy making and policy learning theory?"
The article addresses the research question in the framework of policy analysis studies. The concept of transnational policy making (in terms of governance of problems and transnational policy promotion) and theory of policy learning (in terms of lesson-drawing) are employed in order to provide an in depth insight into the process of defining and solving policy problems in the contemporary educational policies. Theoretical dispositions are further elaborated on in the case of Slovenian PISA 2012 results in reading literacy and trends in other participating EU member states from 2000 onwards. The empirical data for the case study were gathered by the analysis of the OECD and EU official reports, as well as an analysis of the respective Slovenian legislation and strategic documents. In order to provide an additional understanding of the reception of transnational policy making at the national level, the data gathered by interviews with Slovenian and EU representatives (policy makers, researchers, practitioners) from 2008 to 20127 and the results of the survey about the reception of
6	By taking into consideration the policy approaches for improving the PISA results, the article does not take into consideration the more substantive and pedagogical approaches for improving PISA results.
7	Data gathered through semi-structured interviews present an additional source of information and were used only to clarify those open issues that we were unable to identify from our analyses of official documents.
31
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
the EU and international agenda in Slovenian educational space conducted in 2012 (Stremfel, 2013) are used.
The article proceeds as follows. In the introduction, the topic and its research framework (question, methods) are explained. In the first section, the article provides insight into how policy problems are constructed in contemporary society with a special emphasis on educational policies through the lenses of the concepts of transnational policy making and new modes of EU governance. In the second section, the possibilities of finding a policy solution for the perceived policy problem are provided using the framework of policy learning theory. Here the article points out two alternative understandings of PISA policy orientation (international policy promotion in the framework of OECD recommendations and lesson-drawing from other participating member states). In the third section, the article, with the help of the case study of Slovenian PISA 2012 reading literacy results elaborates on the difficulties of finding policy solution and improving PISA reading literacy results in Slovenia. In the conclusion, the article summarizes the key findings, which could be taken into consideration by leading the way in order to improve Slovenian PISA results on the basis of lesson-drawing from other successful EU member states.
Identification of National Policy problem(s) Using PISA Results
"Policy problems are those social problems that can be resolved and are being resolved by the state by means of instruments and mechanisms at its disposal" (Fink Hafner, 2002, p. 105). In its widest sense, a policy problem is understood as a deviation between the present situation and a more desired future situation. Processing a problem is usually understood in the sense of solving it. It means that people start thinking about the means of connecting or bridging the gap between what is and what should be. Identification of a policy problem is therefore an important dimension of problem processing. Governing depends on identifying situations as problematic, acknowledging the expertise in connection with these problems and discovering governing technologies that are considered to be a suitable response (Colebatch 2006, p. 313).
From the perspective of social constructivists, the formulation of (mostly transnational) policies turns into the governance of problems. Policy-making actors are present in different spaces and at different times and they differ in terms of their experiences, values, norms and beliefs. Common cooperation is only possible if they succeed in forming a common understanding regarding the necessity of cooperation (Paster, 2005;
32
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Bernhard, 2011). The essential process in relation to this is a joint identification of the problem, which is a prerequisite for cooperation (Hoppe, 2011, p. 50). Governance, as transnational problem solving, takes place when a group of countries recognise a common policy problem and unite their efforts in making plans for its resolution, which is evident from a jointly developed policy model. Governance based on transnational networks in the field of education could not be considered as a national one, as international comparative data construct policy problems and develop policy solutions beyond and between levels (Novoa and Yariv-Mashal, 2003; Ozga et al., 2011). Together with the new conception of education (where the main emphasis is on student achievement), the development of new policy instruments (international comparative assessment studies, international comparative achievement scales, benchmarks) guarantees the capacity of governance of the OECD and the European Commission, not only by means of monitoring and assessing national education systems, but mostly by constructing specific policy problems and thus encouraging special assumptions and an understanding of policy learning. Grek (2010) argues that constructing a policy problem is necessary for establishing new modes of governance on the basis of more and more new data, standards and new policy solutions. According to the new modes of EU governance, member states, when they perform below average in PISA comparative achievement scale are faced with triple pressure:
a) Performing below international (OECD, EU) average Comparisons based on PISA should not be viewed merely as a method, but also as a policy and mode of governance (governance by comparisons). Comparisons (commonly shown in international comparative achievement scales) result in definitions of good and bad education systems, legitimise political actions and thus create a new mode of governance. They mostly encompass a rationalistic approach to policy making, wherein (assessed) participants are implicitly under pressure to arrive as close as possible at what is considered 'the best' in accordance with special criteria within a certain context of comparisons. In this regard, the leading assumption is that the most efficient (rationalist approach) and the most suitable (con-structivist approach) decisions are adopted on the basis of objective data (March and Olsen, 1989). This objective data, which PISA produces, guarantees the comparability of educational systems and enable member states to identify and eliminate the shortcomings of their educational systems on the basis of mutual comparisons. According to Senberga (2005, p.15), international comparisons exert positive pressure on national political actors, thereby resulting in policy improvements at the national level.
33
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
The meaning being below average in PISA international comparative achievement scale for the prosperity of the participating nation is well elaborated in OECD reports. OECD (2010b, p. 157) argues that "Evidence of the importance of reading literacy for the success of individuals, economies and societies has never been stronger. Past experiences suggest that there are enormous economic gains to be had by OECD countries that can improve the cognitive skills of their populations". The idea that the performance of member states in PISA is an indicator of their further economic development draws attention to PISA results across the world and exerts pressures on participating member states to perform well (that is above average) and therefore ensures their international economic competitiveness and the well-being of their nations.
b) Non-attaining of EU benchmark and common goals The underlying logic of the concept of governance is that society needs mechanisms for defining common problems, establishing collective goals in order to address and solve these problems, and developing and implementing policy instruments by means of which the goals (outputs) will be achieved (Pierre and Peters, 2000). Grek (2009) believes that within output-oriented governance, data and its management play a key role. Data enables governance through goal setting, whereby participant output is directed towards achieving goals. Upon publishing, this data serves as the instruments of encouragement and judgement of participants in terms of their output. It thus simultaneously controls the autonomy of the actors operating within the context in relation to how they will achieve their goals. This is a system of discipline based on the judgement and classification of participants in achieving (jointly defined) goals.
Grek (2009) argue that one of the most visible examples of output-oriented governance is common EU cooperation in the field of education. In order that their educational systems could importantly contribute to the development of EU smart, sustainable and inclusive growth as defined in EU 2020 strategy, EU member states agreed on common educational goals, benchmarks and indicators, which they follow and monitor together. PISA data is used for one of the benchmarks, which states that by 2020, the share of 15-year-olds with a low achievement in reading, mathematics and science should be less than 15%.
Since a benchmark has been commonly agreed, member states feel responsible to effectively contribute to its attainment. The member states' responsibility is strengthen also by the publications of the European Commission (see for example European Commission, 2013), which by analysing the progress towards attaining particular common goal at the EU lev-
34
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
el, points out the member states, which the most and the less successfully contribute to its attainment. Authors (e.g. Alexiadou, 2007) argue that these publications present a subtle pressure on member states on the basis of "naming and shaming" and encourage member states to improve their PISA results and consequently effectively contribute to commonly agreed benchmarks and goals.
c) Non-attaining of national goals
Empirical study (Stremfel, 2013) revealed that in Slovenia, the international comparative assessment studies (including PISA) are regarded as an objective indicator of the knowledge of Slovenian students and that they allow identification of policy problems when it comes to Slovenia's below-average achievements in comparison with the international (EU and OECD) average. The importance of Slovenian performance in these studies is highlighted also in the White Paper on Education (2011, p. 25), where it is stated: "At the state level we need to state and map out a clear path towards the goal, that performance of Slovenian students in international comparative assessment studies are at the top, that mean at least in the upper third of the students' achievement of the developed countries".
The international comparative assessment studies are therefore understood as an instrument of external evaluation of the national educational system. Public policy evaluation is generally defined as any assessment of the public policy effects and provides information, whether the objectives of the public policies are being achieved (Dye, 1995, p. 321). When the results of the evaluation show that public policy does not successfully follow its objectives, it calls for the abolition of public policy or its improvement.
In this section, it is presented how non-attainment of the national educational objectives (related also to the (below average) performance of students in international comparative assessment studies) is theoretically perceived as a policy problem. In the next section, the article presents two distinct theoretical ways of resolving these policy problems through the lenses of transnational policy making.
PISA Policy Impact: From International Policy Promotion to Lesson-drawing
Authors (e.g. Meyer and Benavot, 2013) argue that PISA does not allow only the identification of policy problems, but on the basis of its results also policy solutions can be provided. That is usually understood in terms of PISA strong policy orientation and its policy impact on member states (Grek, 2010). We consider that PISA policy impact in terms of problem
35
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
solving can be understood from two distinct types of policy learning: international policy promotion and lesson-drawing (see Holzinger and Knill, 2006).
a) Policy impact as international policy promotion
Cross-national policy learning is stimulated by the construction of international comparative achievement scales ranking national policies in terms of performance to previously agreed criteria (Grek, 2009). In constantly searching for new policy ideas, disseminating best practice and evaluating national policy performance, international institutions (also OECD) function as mediators of cross-national policy learning, urging national governments to adopt successful policy models (Kern et al., 2000, p. 10 in Holzinger and Knill, 2006, p. 22). Since it is believed that international institutions promote the spread of distinct policy approaches they consider particularly promising, this process is understood as international policy promotion. Countries that deviate from recommended policy models or rank low in international comparative achievement scale face pressure to legitimate their policy approaches in light of "international scrutiny" and are motivated to adopt these certain policy approaches because of legitimacy pressures of the international institutions (Holzinger and Knill, 2006, p. 22).
Carvalho (2012, p. 173) argues that having in mind the concept of a public policy instrument, one may say that PISA is driven by a specific "problematisation" of the role of education in contemporary times and by a specific model for the regulation of the educational sector. With its international comparative achievement scales publicising which countries are progressing in the right direction and which are falling further behind with respect to student achievement, PISA steers participating nations towards a particular model of curricular and structural reform (Takayama, 2012, p. 148).
Recommendations resulting from expert discourse are based on the strategy of comparison and attempt to impose similar answers for different national contexts. In different countries, the OECD recommendations have been accepted as valid among policy makers and stakeholders on the basis of the authoritative characteristic of knowledge included in these reports (Grek, 2010, p. 398). An important factor of the readiness to accept these recommendations is uncertainty with regards to how to improve their results in international comparative achievement scales (Novoa and Yariv-Mashal, 2003; Grek, 2010). Under this approach, then, PISA is seen as a way of gradually solving national problems by moving problem solving capacity from the national to the supranational level (see also Alexiadou, 2014, p. 128).
36
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Bieber and Martens (2011) explain, that OECD from the PISA results draw recommendations for policy-making by concentrating on factors that are positively correlated with student performance in PISA, though without claiming a causal relationship. These recommendations (included in international thematic reports or specific country reviews) range from rather implicit to very explicit statements. For example, OECD Economic Review for Slovenia, which refers also to PISA results, provides the recommendations considering the efficiency of Slovenian basic education. In the Review (OECD, 2011, p. 1), it is stated that "saving could be gained by enhancing spending efficiency in early childhood and basic education, which are plagued by high costs due to low pupil-teacher ratios, small class sizes and high numbers of non-teaching staff. Merging schools and extending catchment areas, while taking into account other socio-economic considerations, could bring significant efficiency gains". Therefore, the OECD Economic Review, by proposing very concrete and economic oriented measures, which does not take into consideration the particularities of the Slovenian national context (Educational Research Institute, 2011) could be seen as international policy promotion.
Although international policy promotion in the situation of uncertainty about how to improve PISA results could be an attractive idea for participating member states, even more so when PISA is understood as an objective and neutral evaluation of a national education system (Interviews, 2012). It is worth mentioning that such international policy promotion erodes the traditional idea of member states sovereignty over their national educational systems (e.g. Walkenhorst, 2008; Zgaga, 2011). On one hand, some authors are concerned about international policy promotion and see it as a portfolio of best practices imposed to national governments by global actors (e.g. OECD, EU). On the other hand, some authors (e.g. Steiner-Khamsi, 2012, p. 3) argue that travelling reforms supposedly represent best practices or international standards that have been transferred successfully from one country to another and regard policy making as a rational undertaking, and view policy learning as examples of lesson-drawing, thus one of the more desirable outcomes of evidence-based policy making.
b) Policy impact as lesson-drawing
Lesson-drawing is seen as a pragmatic tool for identifying and transferring "best practices" from one context to another with the goal of solv-
8 A policy lesson according to Rose (1993, p. 27) is "a detailed cause-and-effect description of a set of actions that government can consider in the light of experience elsewhere, including a prospective evaluation of whether what is done elsewhere could someday become effective here".
37
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
ing problems and improving educational systems in different national settings (Rose, 1991; 1993). In the case of lesson-drawing, an individual country searches for foreign examples and decides on its own to what extent and in which way it will "learn from others" when modifying, improving or making new national policy. A particular country therefore voluntary decides on its own from which country it will learn from and to what extent, as well as how, it will monitor any (new or amended) policy adjustments (including its implementation), and to whom - if anybody at all - it will report its success to (Fink-Hafner et al., 2010, p. 19).
In the lesson-drawing the decisions are based on searching for the means to pursue valued goals in a systematic and comprehensive manner, reviewing policy in the light of past experience and other available information to make adjustments where necessary (James and Lodge, 2003, p. 181). The presumption is that actors work in rational accounts. The question of "how to improve", guides specific mechanisms for improving, including sources and ways to analyse evidence (James and Lodge, 2003, p. 190). Lesson-drawing (when used for resolving identified policy problem, improving national educational policy and consequently improving PISA results) therefore requires serious scientific investigation.9
According to Rose (2002) lesson-drawing should be implemented very carefully by following ten steps: (1) Diagnosing your problem; (2) Examining where to look for a lesson; (3) Investigating how a programme works there; (4) Abstracting a cause-and-effect model for export; (5) Designing a lesson; (6) Deciding should it be imported?; (7) Identifying resource requirements and constraints; (8) Exploring the problem of context; (9) Bounding speculation through prospective evaluation; (10) Identifying foreign countries as positive or negative symbols.
After presenting two theoretical insights in PISA policy orientation and policy impact (international policy promotion and lesson-drawing), it is interesting to see how OECD itself understands PISA policy orientation. OECD (2003, p. 16) states: "Key features driving the development of PISA have been: its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons". As seen the OECD definition does not involve any specific type of policy learning and therefore (at least officially) leaves the space for employing different types of policy learning from PISA results open.
9 Philips (2013, p. 299) argues that in addition to (a) serious scientific investigation, there could be also other motives for lesson-drawing: (b) popular conceptions of the superiority of other approaches to the educational questions; (c) politically motivated endeavours to seek reform of provision by identifying a clear contrast with the situation elsewhere; (d) distortion (exaggeration), whether or not deliberate, of evidence from abroad to highlight perceived deficiencies at home.
38
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Taking into consideration the importance of preserving the sovereignty of national states over the development of their educational systems (e.g. Walkenhorst, 2008; Zgaga, 2011) and the Slovenian experience with the de-contextualized OECD recommendations (2011) to its educational system, we consider that it is important for participating member states (also Slovenia) to find their own policy solution to identified policy problem (below-average PISA results). In the next section, the article therefore provides a more detail empirical insight in the lesson-drawing as a promising strategy for improving below-average PISA results using the case study of Slovenia.
PISA Policy Problem in Slovenia and a Way Towards its Solution
When a state identifies policy problem according to its below-average performance in PISA and decides to solve it by the lesson-drawing (drawing lessons from other (successful) participating states), there are some theoretical dispositions developed which can assist and guide individual states towards that comprehensive process. In this section, the article tries to provide some empirical insight into first two of the Rose (2002) ten steps for learning from abroad, presented in the previous section.
Figure 1: Slovenian reading literacy results of PISA 2012 as a policy problem
Source: OECD (2013a); European Commission (2013).
The first step of lesson-drawing, according to Rose (2002), presents the identification of the national policy problem. Rose (ibid) argues that when political dissatisfaction is high, and especially if it is unexpected, there is often confusion about what exactly the problem is. Figure 1 shows, why Slovenian reading literacy results of PISA 2012 could be understood as a policy problem according to the three perspectives presented in the
39
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
first section of the article ((a) performing below international (OECD, EU) average; (b) non-attaining of EU benchmark and common goals; (c) non-attaining of national goal)).
Figure (a) shows that mean score of Slovenian students in PISA 2012 reading literacy was 481, while the OECD average was 496 and the EU average 489. Figure (b) shows that 21,1 % of Slovenian students could be considered as low achieving students according to PISA 2012 reading literacy results (e.g. not attaining the second (basic) level of reading literacy). That means that Slovenia does not successfully follow the EU benchmark of 15% of low achievers in PISA by 2020. Figure (b) also shows that percentage of low-achieving students in reading literacy in Slovenia (21,1 %) is higher than on the average in the EU member states (17,8 %). Figure (c) shows that taking into consideration PISA 2012 reading literacy results, the Slovenian long-term goal "to perform in the first third of the most developed countries"10 was not reached. The average main score of the first third OECD states was 520, the main score of Slovenia was 481. Results presented in figures 1 (a), (b), (c) therefore show that according to all three criteria performances of Slovenian students in PISA 2012, reading literacy could be perceived as a policy problem.
It should be noted that according to all three perspectives, the policy problem was recognized with reference to the external measures. Even the national goal was stated in terms of ranking in international comparative achievement scale and does not provide a detailed insight in more substantive national goals and priorities. According to Rose (2002, p. 6) stating that "there is no point in looking abroad for a remedy if you do not know what the problem is at home", Slovenia has not realized the first Rose step of successful lesson-drawing yet. In order to empirically explain how non defined national goals and priorities can hinder the further process of lesson-drawing according to Rose (2002), at this point of the article we move to his second step oflesson-drawing that is to the question "Where to look for a lesson"?
Rose (1993, p. ix-x) argues that lesson-drawing occurs across time and space and is both positive, leading to prescriptions about what ought to be done, and negative, in terms what not to emulate. Although there are some suggestions about the usefulness of concentrating on the failure of other member states, (see Radaelli, 2004), authors (e.g. Hemerijck and Visser, 2001) it is argued that it is more promising to look for lessons from
10 Although in the Slovenian White Paper on Education it is not exactly defined, what is considered under »the most developed countries«, we took into consideration the results of the OECD member states. OECD is often called »the club of world's most advanced countries« (OECD, 2014).
40
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Table 1: Identification of the most successful EU member states in following the EU benchmark
PISA cycle/	2000	2003	2006	2009	2012	Trend (20002012)
member state						
Austria	19.9%	20.7%	21.5%	/	19.5%	0.4%
Belgium	19.0%	17.9%	19.4%	17.7%	16.1%	2.9%
Bulgaria	40.3%	/	51.1%	41.0%	39.4%	0.9%
Croatia	/	/	21.5%	22.5%	18.7%	/
Czech Republic	17.5%	19.3%	24.8%	23.1%	16.9%	0.6%
Cyprus	/	/	/	/	32.8%	/
Denmark	17.9%	16.5%	16.0%	15.2%	14.6%	3.3%
Estonia	/	/	13.6%	13.3%	9.1%	/
Finland	7.0%	5.7%	4.8%	8.1%	11.3%	-4.3%
France	15.2%	17.5%	21.7%	19.8%	18.9%	-3.7%
Germany	22.6%	22.3%	20.0%	18.5%	14.5%	8.1%
Greece	24.4%	25.3%	27.7%	21.3%	22.6%	1.8%
Hungary	22.7%	20.5%	20.6%	17.6%	19.7%	3.0%
Ireland	11.0%	11.0%	12.1%	17.2%	9.6%	1.4%
Italy	18.9%	23.9%	26.4%	21.0%	19.5%	-0.6%
Latvia	30.1%	18.0%	21.2%	17.6%	17.0%	13.1%
Lithuania	/	/	25.7%	24.3%	21.2%	/
Luxemburg	/	22.7%	22.9%	26.0%	22.2%	/
Netherlands	/	11.5%	15.1%	14.3%	14.0%	/
Poland	23.2%	16.8%	16.2%	15.0%	10.6%	12.6%
Portugal	26.6%	21.9%	24.9%	17.6%	18.8%	7.8%
Romania	41.3%	/	53.5%	40.4%	37.3%	4.0%
Slovakia	/	24.9%	27.8%	22.2%	28.2%	/
Slovenia	/	/	16.5%	21.1%	21.1%	/
Spain	16.3%	21.1%	25.7%	19.6%	18.3%	-2.0%
Sweden	12.6%	13.3%	15.3%	17.4%	22.7%	-10.1%
United Kingdom	/	/	19.0%	18.4%	16.6%	/
Source: OECD (2013a). those who succeed. However, which educational system could be considered as successful according to the PISA results? Although we agree that there is no one way to answer this, in this article, we adopt the OECD (2010a, p. 14) understanding of successful states as not just top-scoring,
41
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
but especially those ones which are rapidly improving from the first PISA cycle in 2000 onwards. OECD (2010a, p. 13) explains:
"The design of PISA does not just allow for a comparison of the relative standing of countries in terms of their learning outcomes; it also enables each country to monitor changes in those outcomes over time. Such changes indicate how successful education systems have been in developing the knowledge and skills of 15-year-olds. All countries seeking to improve their results can draw encouragement - and learn lessons - from those that have succeeded in doing so in a relatively short period of time."
Table 1 shows the trends of the EU member states' PISA reading literacy performance since 2000 in order to identify those member states, which were the most successful in improving the results of their low-achieving students and the most successfully follow the EU benchmark "to reduce percentage of low-achievers in PISA to 15% by 2020".11
The Table 1 shows that among 28 EU member states, 18 of them have been participating in all PISA cycles (2000-2012). For these member states trends in the percentage of low-achieving students are presented. It shows that 13 of the member states succeed in reducing the percentage of their low achieving students, and in 5 of them, these percentages from 2000 to 2012 increase. The table also shows that the most successful EU member states in reducing the percentage of their low-achieving PISA students in reading literacy are Poland (12,6%), and Germany (8,1%).
OECD (2010a) claims that success of such a diverse group of countries in raising the level of their students' performance in reading indicates that improvement is possible regardless of a country's context and where it starts out from. Similarly, European Commission (2013) recognizing that the EU as a whole is lagging behind in its challenge to reduce the share of low achievers in reading, points out that this trend does, however, disguise large differences found between and within EU member states. By indicating the concrete member states and their improvement, the European Commission does not only exert the pressure on some member states on the basis of "naming and shaming" but also indicate the countries, from which the lessons could be drawn. The European Commission (2013, p. 5) states: "The reasons why some member states succeeded in significantly reducing the share of low achievers may serve as an inspiration for other countries that are struggling to overcome similar challenges or even face a deteriorating situation."
11 Although OECD identifies trends in results ofpartidpating countries on a special methodology (see OECD, 2013a) which measure trends only between the cycles when the math was a main testing domain, we have present trends from the 2000 onwards which is also the established practice of the EU.
42
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Table 2: Overview of the performance of the most successful member in following the EU benchmark by different indicators
Indicator / member state	Slovenia	Poland	G ermany
Percentage of low achievers (2000)	/	23.2%	22.6%
Percentage of low achievers (2012)	21.1%	10.6%	14.5%
Percentage of low achievers (difference 2012-2000)	/	12.6%	8.1%
Percentage of high achievers (2000)	/	5.9%	8.8%
Percentage of high achievers (2012)	5.0%	10%	8.9%
Percentage of high achievers (2012-2000)		4.1%	0.1%
Gap between 90 th and 10th percentiles (2000)	/	260 points	284 points
Gap between 90 th and 10th percentiles (2°i2)	236 points	222 points	237 points
Change in gap between 90th and 10th percentiles (20i2-2000)	/	38 points	47 points
Proportion of total variation explained by between-school variance (2000)	/	62%	59%
Proportion of total variation explained by between-school variance (2009)	41.8%	65.4%	67.2%
Change in proportion of total variation explained by between-school variance (2009-2000)	/	3.4%	8.2%
Relationship between reading performance and the PISA index of economic, social and cultural status (ESCS) (2000)	/	40 points	52 points
Relationship between reading performance and the PISA index of economic, social and cultural status (ESCS) (2009)	/	39 points	44 points
Change in relationship between reading performance and the PISA index of economic, social and cultural status (ESCS) (2009-2000)		1 point	8 points
Difference in performance between native students and students with immigrant background (2000)	/	/	84 points
Difference in performance between native students and students with immigrant background (2009)	/	/	56 points
Change in difference in performance between native students and students with immigrant background (2009-2000)	/	/	- 28 points
Source: OECD (2010a; 2013a; 2013b).
Since some authors (see Stremfel et al., 2014) argue that one indicator cannot provide enough insight in the functioning of the individual educational system, Table 2 shows how in EU member states, which
43
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
states succeed the most in following the EU benchmark (reduce a number of low-achievers in reading literacy), the which trends in other indicators have changed.
Table 2 shows that Poland and Germany, which succeed the most in following EU benchmark (reducing the percentage of low-achievers), were not as successful in other selected indicators. Estimating which of them would be the most appropriate to learn from in order to improve Slovenian PISA results and successfully follow the EU benchmark is therefore a comprehensive task. The review of trends in different indicators shown in Table 2, first of all requires that a learning country (Slovenia) define concrete goal about which set of indicators it would like to improve upon. One single benchmark (defined at the EU level) is too broad and cannot provide that focus and learning the state should find itself. Even OECD (2010a, p.4) recognized that "PISA results suggest that the countries that improved the most, or that are among the top performers, are those that establish clear, ambitious policy goals (...)."
Conclusions
If a new mode of governance in the EU is viewed as governing, steering and supervising actors (Kooiman, 2003, p. 3), for them to participate in collective policy problem solving and thus achieve the pursued goals jointly (Pierre and Peters, 2000), the highlighted lack of clarity of educational goals both at the supranational and the national level: (a) opens up room for political manipulation of international organisations (Borras and Radaelli, 2011) or (b) present a huge obstacle on the way of improving the results on the basis of lesson-drawing. The wide-ness and openness of goals (and consequently their lacking clarity) allows the development of legitimate, reasonable and good policies and the (imaginary) common good in the context of social learning (Borras and Conzelmann, 2007) and therefore pursuing a specific not necessary evidence-based educational model.
With apparent PISA neutrality EU and OECD steers the member states towards achieving specific educational goals. The EU benchmark (reducing a number oflow achievers to 15% by 2020) facilitates assessments and comparisons of member states' achievements (output-oriented governance and governance by comparison) in pursuing the common EU goals. PISA comparative achievement scale thus exerts dual pressure on the EU member states. The primary pressure to perform well is related to securing the international competitiveness of the state. The secondary pressure to perform well is related to avoiding the blaming and shaming by the European Commission and by other member states for not attaining com-
44
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
mon agreed goals (Alexiadou, 2007; Ioannidou, 2007). Once a member state perceives a policy problem (related to lack of economic competitiveness) following its ranking on PISA achievement scale, the best models for solving the problems in question (governance of problems) have commonly already been developed at the OECD level. In the article, this is shown using the example of OECD Economic Review for Slovenia (2011). In the case that member states follow these recommendations, the presented dynamics facilitates the deepening of the OECD cooperation in the field of education towards what is preferred by the OECD (international policy promotion), while the member states have over the past few years - in the circumstances of the economic crisis - been following the OECD more so than before, aiming to maintain their competitiveness within the knowledge-based economy (also see Tsarouhas, 2009).
However, it is also necessary to be aware of the fact that actors have different sources for a critical appraisal of the knowledge provided by international comparative assessment studies and an effective use of that knowledge for development of their national educational systems. In such a context, deep and careful reflection about the nature of knowledge and its mobilisation within public policy is essential. This raises a question of whether the use of (international) comparisons as a mode of governance has not resulted in excessive legitimacy of knowledge they produce and whether it is time for actions towards a diversity of knowledge types, communicated by means of knowledge-based governance tools (Delvaux and Mangez, 2010).
The main implications of understanding PISA as transnational problem solving would therefore be that the expert knowledge, which the PISA and other international comparative assessment studies provide, should be used at the national level in accordance with neopositivist and critically rational means of "speaking truth to power" and not in accordance with the interpretative and neopragmatic means of "making sense together" (Hoppe 2011, p. 55).12 In this author's opinion, the role of national experts is to assess what data (from PISA and international comparative assessment studies) and proposals for solving the identified policy problems are to be taken as legitimate and definite in implementing the changes and improvements in the national system (Wiseman 2010, p. 9). That was already recognized, when the OECD Economic Review for Slovenia
12 Experts and the expert knowledge would thus be used in an instrumental sense ofmaking the right decisions and not for the advocacy of political decisions and the ideology of (supranational and national) political actors (Stone, 2000; Jones, 2009; Nassehi and Demsz-ky, 2011). After all, in Slovenia some in-depth critical deliberations regarding the (non-)use of expert knowledge in education policy making have already been undertaken as well (Gaber, 2007; Kodelja, 2007) and might be worth reconsidering.
45
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
was published and national educational experts warned about some of its misleading conclusions.
The appropriate use of expert data, which can be acquired from PISA and other international comparative assessment studies, can facilitate the preservation of distinct national characteristics and the quality of the education system and make thoughtless acceptance of international policy promotion of certain educational models much more unlikely (Grek, 2008). In order to preserve the sovereignty of national state over its educational system, the article therefore suggests that instead of an uncritical reception of the international promotion of certain educational model (OECD, 2010c; Hanuschek and Woessmann, 2011), the more promising alternative for improving PISA results is lesson-drawing.
Ifwe understand the policy problem as the deviation between the actual current situation and the desired future situation, more emphasis in Slovenia should be put on the concretization of the desired future situation in terms of concretization of the national educational goals. The argument is that the EU benchmark (to reduce number oflow-achievers to 15% by 2020) and Slovenian strategic goal (to perform in the first third of the developed countries in international comparative assessment studies) are too broad to identify concrete policy problem, to provide a solid base for national educational reform and to target specific policy measures for improving PISA results and pursuing these goals. In addition, Table 2 shows that monitoring one single indicator (reducing number of low-achievers to 15% by 2020) is not sufficient for in depth understanding of long-term performance of successful member states, which is necessary in order to draw lessons from them and to find their own way of improving PISA results. One single benchmark / indicator therefore cannot provide an in depth insight into which policy measures EU member states should focus on, which measures have succeeded the most in reducing the number of low-achievers in their educational system (Germany, Poland), which have been employed and how these policy measures have impacted other indicators, as well as which are important for ensuring equity and quality in their educational system. Since the lesson-drawing is a timely and expensive process (Rose, 2002), it is even more important that states do not make a mistake already in the first two steps of learning from abroad (identifying of policy problem and finding a state, from which they will learn from). If we took into consideration the theoretical dispositions (of international policy promotion and lesson-drawing) presented in the article, it seems that the main policy lesson for Slovenia from PISA results is that clear goals should be stated and then followed to more overreaching goals - not only to perform in the first third of the most developed
46
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
participating countries in international comparative achievement scales, but more importantly to be aware (on the all levels of the educational systems), which educational goals we are following in order to contribute to the welfare of the nation. PISA results should therefore be seen as the external mirror for finding and monitoring member states own ways of improving educational results and not a goal in itself.
References
Alexiadou, N. (2007) The Europeanisation of education policy - changing governance and »new modes of coordination«. Research in Comparative and International Education. 2 (2), pp. 102-116. Alexiadou, N. (2014) Policy learning and Europeanisation in education: The governance of a field and the transfer of knowledge. In: Nordin, A. & Sundeberg, D. (eds.) Transnational Policy Flows in European Education: the making and governing of knowledge in the education policy field. Oxford: Symposium Books. Berhard, S. (2011) Beyond Constructivism: The Political Sociology of an EU Policy Field. International Political Sociology. 5 (4), pp. 426-445. Bieber, T., and Martens, K. (2011) The OECD PISA Study as a Soft Power in Education? Lessons from Switzerland and the US. European Journal of Education. 46 (1), pp. 101-116. Borras, S., and Conzelmann, T. (2007) Democracy, Legitimacy and Soft Modes of Governance in the EU: The Empirical Turn. Journal of European Integration. 29 (5), pp. 531-548. Borras, S., and Radaelli, C. (2011) The Politics of Governance Architectures: Creation, Change and Effects of the EU Lisbon Strategy. Journal of European Public Policy. 18 (4), pp. 463-484. Carvalho, L. M. (2012) The Fabrications and Travels of a Knowledge -Policy Instrument. European Educational Research Journal. 11 (2), pp.172-188.
Colebatch, K. H. (2006) What work makes policy? Policy Sciences. 39 (4), pp. 309-321.
Council of the EU (2009). Council conclusions of 12 May 2009 on a strategic framework for European cooperation in education and training ('ET 2020'). Brussels: Council of the European Union. Delvaux, B., and Mangez, E. (2010) Knowledge and policy: a new deal.
Leuven: University of Leuven: KNOWandPOL Research Report. Dye, T. R. (1995) Understanding public policy. Englewood Cliffs, New Jersey: Prentice Hall.
Educational Research Institute (2011) Kritična tiskovna konferenca ob poročilu OECD o slovenskem šolstvu. / Critical press conference about
47
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
the OECD Economic Review for Slovenia. Ljubljana, 16th March 2011.
European Commission (2013) PISA 2012: EUperformance and first inferences regarding education and training policies in Europe. Brussels: European Commission.
Fink-Hafner, D. (2002) Analiza politik - akterji, modeli in načrtovanje politike skupnosti. In Jelovac, D. (ed.). Jadranje po nemirnih vodah menedžmenta nevladnih organizacij. / Sailing troubled waters of management of non-government organizations. Ljubljana: Radio Študent: Študentska organizacija Univerze v Kopru: Visoka šola za management. pp. 105-123.
Fink-Hafner, D., Lajh, D., and Deželan, T. (2010) The Open Method of Coordination in the Global Context of Policy Cooperation. In: Fink-Hafner, D. (ed.) The Open Method of Coordination. Ljubljana: Faculty of Social Sciences.
Gaber, S. (2007) Spoprijem za hegemonijo ali vaje iz praktične teorije? Sodobna pedagogika [Contemporary Pedagogy]. 58 (2), pp. 62-81.
Grek, S. (2008) From symbols to numbers: the shifting technologies of education governance in Europe. European Educational Research Journal. 7 (2), pp. 208-218.
Grek, S. (2009) The role of the OECD as an agent of Europeanization: problematisation and change in education governance in Europe. Paper presented at Edinburgh's Europa Institute Seminar: Practicing EU Government: Problematisation, 6. May, Edinburgh, Great Britain.
Grek, S. (2010) International Organisations and the Shared Construction of Policy »Problems«: problematisation and change in education governance in Europe. European Educational Research Journal. 9 (3), pp. 396-406.
Hanuschek, E., A., and L. Woessman, L. (2011) The Cost of Low Educational Achievement in the European Union. EENEE Policy Brief.
Hemerijck, A., and Visser, J. (2001) Learning and Mimicking: How European Welfare States Reform. Amsterdam: University of Amsterdam, Working Paper.
Holzinger, K., and Knill, C. (2006) Causes and conditions of cross-national policy convergence. In: Knill, C. (ed.). Cross-national Policy Convergence: Concepts, Causes and Empirical Findings. London: Routledge.
Hopmann, T. S., Brinek, G., and Retz, M. (2007) PISA According to PISA. Does PISA Keep What it Promises. New Brunswick in London: Transaction Publishers.
48
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Hoppe, R. (2011) The Governance of Problems: Puzzling, Powering and Participation. Bristol: The Policy Press.
Ioannidou, A. (2007) A Comparative Analysis of New Governance Instruments in the Transnational Educational Space: a shift to knowledge-based instruments? European Educational Research Journal. 6 (4), pp. 336-347.
Interviews by author (2012) Interviews conducted with public officials, experts and practitioners in the field of education policy. In: Štremfel, U. (2013). Nova oblika vladavine v Evropski uniji na področju izobraževalnih politik. [New modes of governance in the EU in the field of education policies.], pp. 400. Ljubljana: Faculty of social sciences.
James, O., and Lodge, M. (2003) The Limitations of 'Policy Transfer' and 'Lesson Drawing' for Public Policy Research. Political Studies Review. 1 (2), pp. 179-193.
Jones, H. (2009) Policy-making as discourse: a review of recent knowledge-to-policy literature. A Joint Information and Knowledge Management and Overseas Development Institute Emergent Working Paper No. 5.
Kodelja, Z. (2005) Komparativne edukacijske raziskave in šolska politika / Comparative educational research and school policy. Šolsko polje. 16 (3-4), pp. 211-226.
Kodelja, Z. (2007) Reforme, stroka in šolske politike. Sodobna pedagogika [Contemporary Pedagogy]. 58 (2), pp. 34-48.
Kooiman, J. (2003) Governing as Governance. London: Sage.
Kustec Lipicer, S. (2009) Vrednotenje javnih politik. Ljubljana: Fakulteta za družbene vede.
Lasswell, H. D. (1951) The Policy Orientation. In: Lerner, D., and Lasswell, H. D. (eds.) The Policy Sciences. Recent Development in Scope and Method. Stanford: Stanford University Press.
March, G. J., and Olsen, P. J. (1989) Rediscovering Institutions. The Organizational Basis of Politics. New York: The Free Press.
Meyer, H. D. and Benavot, A. (2013) PISA, Power, and Policy: the emergence of global educational governance. Oxford: Symposium Books.
Nassehi, A. von der, Hagen-Demszky, A., and Mayr, K. (2007) The Structures of Knowledge and of Knowledge Production. KNOWandPOL Report, Literature review, Part 8.
Novoa, A., and Yaric-Marshal, T. (2003) Comparative Research in Education: A mode of governance or a historical journey? Comparative Education. 39 (4), pp. 423-443.
49
ŠOLSKO POLJE, LETNIK XXV, ŠTEVILKA 5-6
OECD (2003) Literacy Skills for the World of Tomorrow - Further results from PISA 2000. Paris: OECD.
OECD (2010a) PISA 2009. Results, Learning Trends: Changes in Student Performance Since 2000. Paris: OECD.
OECD (2010b) PISA 2009 Results: What Students Know and Can Do. Studentperformance in reading, mathematics and science. Paris: OECD.
OECD (2010c) The High Cost of Low Educational Performance. The longrun economic impact of improving PISA outcomes. Paris: OECD.
OECD (2011) OECD Economic Surveys Slovenia. Paris: OECD.
OECD (2013a) PISA 2012 Results: What Students Know and Can Do. Student performance in mathematics, reading and science. Paris: OECD.
OECD (2013b) PISA 2012 Results: Excellence through Equity. Giving every student the chance to succeed. Paris: OECD.
OECD (2014) OECD members and partners. Available from: http:// www.oecd.org/about/membersandpartners/. (Accessed: 24th May 2014).
Ozga, J., Dahler-Larsen, P., Sergerholm, C., and Simola, H. (eds.) (2011) Fabricating Quality in Education: Data and Governance in Europe. London: Routledge.
Paster, T. (2005) The New Modes of EU Governance: Combining Rationalism and Constructivism in Explaining Voluntarist Policy Coordination in the EU. Österreichische Zeitschriftfür Politikwissenschaft. 34 (2), pp. 147-161.
Pereyra, M. A., Kottoff, H. G., and Cowen, R. (eds.) (2011) PISA under examination. Changing Knowledge, Changing Tests, and Changing Schools. Rotterdam: Sense Publishers.
Phillips, D., and Ochs, K. (2003) Processes of policy borrowing in education: Some analytical and explanatory devices. Comparative Education. 39 (4), pp. 451-461.
Pierre, J., and Peters, G. B. (2000) Governance, Politics and the State. New York: St. Martin's Press.
Radaelli, C. (2004) Who learns what? Policy learning and the open method of coordination. Paper presented at Economic and Social Research Council Workshop on the Lisbon Strategy, University of Birmingham, 26. November, Birmingham, Great Britain.
Rose, R. (1991) What is Lesson Drawing? Journal of Public Policy. 11 (1),
pp. 3-30.
Rose, R. (1993) Lesson-Drawing in Public Policy: A Guide to Learning across Time and Space. Chatham: Chatham House Publishers.
Rose, R. (2002) Ten Steps in Learning Lessons from Abroad. EUI Working Paper. Florence: European University Institute.
50
u. STREMFEL ■ SLOVENIA ON ITS OWN WAY TOWARDS IMPROVING PISA RESULTS
Scharpf, W. F. (1997) Introduction: The Problem-Solving Capacity of Multi-Level Governance. Journal of European Public Policy. 4 (4), pp. 520-538.
Steiner-Khamsi, G. (2012) Understanding Policy Borrowing and Lending. Building Comparative Policy Studies. In: Steiner-Khamsi, G., and Waldow, F. (eds.) Policy Borrowing and Lending in Education. London and New York: Routledge.
Stone, D. (2000) Learning Lessons, Policy Transfer and the International Diffusion of Policy Ideas. Warwick: University of Warwick, Centre for the Study of Globalisation and Regionalisation Working Paper, No. 69/01.
Šenbrega, K. (2005) The Open Method of Coordination and Mutual Policy Learning, Opportunities and Constraints. Lund: Lund University, Department of Political Science, University Essay from Lund University.
Štremfel, U. (2013) Nova oblika vladavine v Evropski uniji na področju izobraževalnih politik. / New modes of governance in the EU in the field of education policies. Ljubljana: Faculty of social sciences.
Štremfel, U., Straus, M. and Lajh, D. (2014) Sprejemanje rezultatov PISA 2009 v Sloveniji skozi prizmo javnopolitičnega učenja. Šolsko polje. 25 (1-2), pp. 79-104.
Takayama, K. (2012) Bringing a Political "Bite" to Educational Transfer Studies: Cultural Politics of PISA and the OECD in Japanese Education Reform. In: Steiner-Khamsi, G., and Waldow, F. (eds) Policy Borrowing and Lending in Education. London: Routledge.
Tsarouhas, D. (2009) The Open Method of Coordination and integration theory: are there lessons to be learned? Paper presented at 11th European Union Studies Association Biennial Conference, 23.-25. April, Los Angeles, USA.
Waldow, F. (2009) What PISA Did and Did Not Do: Germany after the " PISA-shock". EuropeanEducationalResearchJournal. 8 (3), pp. 476-483.
Walkenhorst, H. (2008) Explaining change in EU education policy. Journal of European Public Policy. 15 (4), pp. 567-587.
White Paper on Education (2011) Ljubljana: Ministry of Education, Science and Sport.
Wiseman, A. W. (2013) Policy Responses to PISA in Comparative Perspective. In: Dieter Meyer, H., and Benavot, A. (eds.) PISA, Power and Policy. The emergence of global educational governance. Oxford: Symposium books.
Zgaga, P. (2011) Education for "a better world": is it still possible? Education Inquiry. 2 (2), pp. 331-343.
51