08 / 09 Z N A N S T V E N A P O R O Č I L A P E D A G O Š K E G A I N Š T I T U T A Maša Vidmar Peter Tymms P R E I Z K U S T E M E L J N I H K O M P E T E N T N O S T I O T R O K O B V S T O P U V Š O L O ( P I P S - B A ) izhodišča, prevod, priredba in aplikacija preizkusa v Sloveniji P E R F O R M A N C E I N D I C A T O R S I N P R I M A R Y S C H O O L S : O N - E N T R Y B A S E L I N E A S S E S S M E N T A N D F O L L O W - U P M A T E R I A L ( P I P S - B A ) introducing pips into Slovenian context – background, translation, adaptation, and application of the assessment Aktivnosti v okviru projekta Perspektive evalvacije in razvoja sistema vzgoje in izobraževanja omogoča sofinanciranje Evropskega socialnega sklada Evropske unije in Ministrstva za šolstvo in šport. Pedagoški inštitut, Gerbičeva 62, p.p.: 4241, 1000 Ljubljana, T + 386 (0)1 - 420 - 12 - 40, F + 386 (0)1 - 420 - 12 - 66, pedagoski.institut@pei.si, http:// www.pei.si, Id. št. za DDV: SI68705093, Matična številka: 5051614000, Podračun pri UJP: 01100-6030346085 08 / 09 Z N A N S T V E N A P O R O Č I L A P E D A G O Š K E G A I N Š T I T U T A Maša Vidmar Peter Tymms P R E I Z K U S T E M E L J N I H K O M P E T E N T N O S T I O T R O K O B V S T O P U V Š O L O ( P I P S - B A ) izhodišča, prevod, priredba in aplikacija preizkusa v Sloveniji P E R F O R M A N C E I N D I C A T O R S I N P R I M A R Y S C H O O L S : O N - E N T R Y B A S E L I N E A S S E S S M E N T A N D F O L L O W - U P M A T E R I A L ( P I P S - B A ) introducing pips into Slovenian context – background, translation, adaptation, and application of the assessment Kontakt: Maša Vidmar, mlada raziskovalka, Pedagoški inštitut, Ljubljana email: masa.vidmar@pei.si To poročilo je del projekta “Perspektive evalvacije in razvoja sistema vzgoje in izobraževanja v Republiki Sloveniji” (2009), financiranega s strani MŠŠ in ESS; projekt koordinira dr. Janez Kolenc. Znanstveno poročilo 08/09 Avtorja: Maša Vidmar in Peter Tymms Naslov: Preizkus temeljnih kompetentnosti otrok ob vstopu v šolo (PIPS-BA) / izhodišča, prevod, priredba in aplikacija preizkusa v Sloveniji Performance Indicators in Primary schools: On-entry Baseline Assessment and Follow-up material (PIPS-BA) / Introducing PIPS into Slovenian context – background, translation, adaptation, and application of the assessment Izdajatelj: Pedagoški inštitut, Ljubljana (zanj Mojca Štraus) Oblikovanje: Emina Djukić in Jaka Kramberger Za strokovno in jezikovno raven poročil odgovarjajo avtorji CIP - Kataložni zapis o publikaciji Narodna in univerzitetna knjižnica, Ljubljana 37.091.26 VIDMAR, Maša Preizkus temeljnih kompetentnosti otrok ob vstopu v šolo (PIPS-BA) [Elektronski vir] : izhodišča, prevod, priredba in aplikacija preizkusa v Sloveniji = Performance indicators in primary schools : on-entry baseline assessment and follow-up material (PIPS-BA) : introducing PIPS into Slovenian context - background, translation, adaptation,and application of the assessment / Maša Vidmar in Peter Tymms. - El. knjiga. - Ljubljana : Pedagoški inštitut, 2009. - (Znanstveno poročilo / Pedagoški inštitut ; 09, 08) Način dostopa (URL): http://www.pei.si/UserFilesUpload/file/zalozba /ZnanstvenaPorocila/08_09_preizkus_temeljnih_kompetentnosti_otrok_ob_vstopu_v_solo.pdf ISBN 978-961-6086-96-7 1. Tymms, Peter 245596928 To delo je objavljeno pod licenco Creative Commons. Avtor/ji besedila dovoli/jo reproduciranje, distribuiranje, prikazovanje in izvajanje ter predelavo pod naslednjimi pogoji: priznanje avtorstva, nekomercialno ter deljenje predelanega dela pod enakimi pogoji. Polno besedilo licence je na voljo na URL naslovu: http://creativecommons.org/licenses/by-nc-sa/2.5/si/legalcode Dovoljenja za morebitno uporabljena avtorska slikovna gradiva so podana sproti v besedilu. Kazalo 6 Razširjen povzetek 9 Abstract 10 Introduction 12 PIPS On-entry Baseline Assessment 15 Translation and adaptation for the use in Slovenia 18 Study 1: Pilot version of Slovene PIPS-BA 18 Method 18 Results 21 Discussion 21 Conclusions 22 Study 2: Final version of Slovene PIPS-BA 22 Method 22 Participants 22 Instruments 24 Procedure 4 25 Results 25 Literacy and numeracy skills of Slovenian first-graders 26 Longitudinal predictions of attainment in Slovene, Math and Environmental Education 28 Discussion 29 Predictive validity 31 Conclusions and limitations 32 Final remarks 33 References 37 List of tables 5 Razširjen povzetek V besedilu predstavljamo nekatere razloge oziroma prednosti spremljanja učencev od vstopa v šolski sistem dalje. Predstavljamo enega izmed preizkusov, ki omogočajo merjenje otrokovih kompetentnosti ob vstopu v šolo, in sicer britanski Performance Indicators in Primary Schools: On-entry Baseline Assessment (PIPS-BA). V besedilu se osredotočamo na potek prevoda in priredbe preizkusa za slovensko šolsko okolje in predstavljamo rezultate dveh raziskav na slovenskih prvošolcih. V Sloveniji sistem spremljanja učencev skozi osnovno šolo zagotavljajo Nacio- nalni preizkusi znanja (NPZ), s katerimi se učenci prvič srečajo v šestem razredu OŠ, nato pa še v devetem. Učenci do šestega razreda tako niso vključeni v nobeno obliko sistematičnega spremljanja njihovega napredka na nacionalni ravni, poleg tega tudi nimamo podatka o njihovih temeljnih znanjih/kompetentnostih ob vstopu v šolski sistem. V primeru, da bi imeli preizkus, ki bi to omogočal, bi učitelji lažje prilagajali pouk in način poučevanja močnim in šibkim področjem posameznih učencev, kar bi olajšalo notranjo diferenciacijo pouka. Lahko bi relativno zgodaj začeli proces identi-fikacije učencev, ki nosijo večje tveganje za učno neuspešnost, ali kažejo nadarjenosti (in ustrezno ukrepali). Predvsem pa bi ob vse večji avtonomiji šol preizkus predstavljal povratno informacijo šoli in učiteljem o napredku njihovih učencev glede na napredke učencev s podobnimi izhodiščnimi dosežki na drugih šolah. Navedeno nas je vodilo k temu, da smo se na Pedagoškem inštitutu lotili pre- voda in priredbe pripomočka, ki bi vse to omogočal. Odločili smo se za Preizkus temeljnih kompetentnosti otrok ob vstopu v šolo (PIPS-BA). V izvirni različici gre za britanski preizkus PIPS-BA, ki so ga razvili na CEM centru na Univerzi v Durhamu leta 1994 in je sestavljen iz treh knjižic: PIPS On-entry Baseline Assessment, Follow-up in Extension (v slovenski različici sta prvi dve združeni v eni knjižici). Preizkus je zas-novan na podlagi rezultatov raziskav in se osredotoča na tiste značilnosti, za katere so raziskave pokazale, da so pomembne na začetku šolanja. Namen preizkusa je nuditi učiteljem obsežne informacije o posameznem učencu takoj po vstopu v šolo. Poleg tega predstavlja informativno in zanesljivo mero, na podlagi katere lahko spremljamo napredek otrok v prvem letu šolanja (preizkus se namreč lahko ponovno izvede ob koncu prvega oz. na začetku drugega razreda). Takšna informacija predstavlja pov-6 ratno informacijo učitelju o njegovem delu. Na CEM centru so poleg knjižne različice razvili tudi elektronsko (računalniško). Preizkus je bil preveden v osem jezikov in ga uporabljajo v številnih državah. PIPS-BA preizkus se trenutno uporablja v več kot 4000 šolah vsako leto v Angliji, Avstraliji in na Škotskem. Slovenska različica preizkusa PIPS-BA meri jezikovne (t.i. zgodnjo pismenost) in matematične (t.i. računski pojmi in spretnosti) kompetentnosti ter obsega štiri sk-lope: besedni zaklad, zavedanje fonetičnih pravil (ponavljanje in rimanje), zgodnje branje ter zgodnja matematika. Zgodnje branje obsega naslednje naloge: pisanje, bral-ni ''pojmi'' (npr. pokaži črko, besedo, začetek stavka ipd.), poznavanje črk, prepoznavanje besed ter branje. Zgodnja matematika obsega naslednje naloge: štetje, seštevanje – odštevanje, prepoznavanje števil ter zahtevnejše matematične naloge (npr. računanje s simbolnim zapisom, besedilne naloge). Poleg tega preizkus vsebuje tudi vprašalnik stališč do različnih dejavnosti v šoli. V besedilu predstavljamo rezultate dveh raziskav, v katerih smo uporabili pre- izkus PIPS-BA na slovenskih prvošolcih. Na podlagi prve (pilotne) raziskave ( N = 135, povprečna starost 6 let 1 mesec) smo preizkus izpopolnili, v drugi ( N = 326, povprečna starost 6 let 2 meseca) pa smo ugotavljali temeljne kompetentnosti slovenskih prvošolcev ter povezanosti rezultatov preizkusa z učno uspešnostjo. Rezultati kažejo, da je 56% otrok ob vstopu v šolo pravilno napisalo svoje ime, 31% otrok pa je pravilno napisalo svoje ime in priimek. 22% otrok je ob vstopu v šolo pravilno poimenovalo vse črke slovenske abecede, 4 % pa nobene; v povprečju so otroci pravilno poimenovali 15 črk. Pri nalogi branja je večina otrok (75%) dosegla nič točk, skoraj 10% otrok je ob vstopu v šolo bralo (dosegli so vsaj 60 od 69 točk). Rezultati kažejo tudi, da zna večina otrok ob vstopu v šolo seštevati in odštevati (ob pomoči slikovnega gradiva). 50% otrok zna ob vstopu v šolo odštevati s prehodom čez desetico (npr. 16-8), 20% otrok zna seštevati s prehodom čez desetico (npr. 8+5). Poleg tega je 60% otrok pravilno poimenovalo vse enomestne številke, 8% otrok pa nobene. 40% otrok je pravilno poimenovalo tudi števila do 20. 29% otrok je pravilno samostojno rešilo računsko nalogo s formalnim zapisom (npr. Ali lahko sešteješ: 6+4?). Napovedno veljavnost preizkusa PIPS-BA smo preverjali tako, da smo z linear- no regresijo analizirali, kako dobro dosežek na posameznem sklopu PIPS-BA (besedni zaklad, zavedanje fonetičnih pravil, zgodnje branje in zgodnja matematika) napoveduje učni dosežek ob koncu prvega razreda na enem izmed treh predmetov (slovenščina, matematika in spoznavanje okolja). Pri tem smo upoštevali tudi otrokov dosežek na neverbalnem testu inteligentnosti in vključenost v vrtec. Rezultati so pokazali, da PIPS-BA dosežki pomembno napovedujejo učno uspešnost ob koncu šolskega leta, in sicer najbolje pri slovenščini (36 % pojasnjene variance), pa tudi pri matematiki (23 % pojasnjene variance), nekoliko slabše pa pri spoznavanju okolja (12 % pojasnjene variance). 7 Kot je razvidno iz navedenih rezultatov, so razlike med otroki v temeljnih kompetentnostih ob vstopu v šolo precejšnje, prav zato bi uporaba preizkusa učiteljem lahko olajšala načrtovanje notranje diferenciacije pouka ter spremljanje napredka učencev. Seveda pa je potrebno zagotoviti smiseln in učinkovit sistem, v katerega bi bil preizkus umeščen: (1) usposabljanje učiteljev (ali vzgojiteljev ali šolske svetovalne službe) za uporabo in interpretacijo preizkusa (2) podporni mehanizmi pri interpret-aciji rezultatov ter nadaljnjem delu z učenci (npr. v obliki rednih izobraževanj, konfer-enc, delavnic, možnost svetovanja na šoli ali preko elektronske pošte) ter (3) evalvacija uporabe preizkusa. 8 Abstract The present paper discusses general rationale behind baseline assessments and presents in detail British assessment Performance Indicators in Primary Schools: On-entry Baseline Assessment (PIPS-BA). In Slovenia, National assessments do not include on-entry assessment at the beginning of compulsory schooling, nor is there a similar assessment for elective use. These arguments and the growing autonomy of schools in Slovenia have leaded to the decision of translating and adapting the assessment PIPS-BA. The PIPS-BA provides classroom teachers with informative and reliable information about their pupils to help with lesson planning. It also provides a starting point for the monitoring of the pupils’ progress. At the end of the year the assessment can be repeated, allowing the teachers to look at children’s progress over the year and to link that to their professional knowledge about the children. It was originally developed in 1994 by the CEM Centre in United Kingdom. Two studies have been conducted employing Slovene version of the assessment. Study 1 (135 first-graders; M = 6 years 1 month; SD = 3 months) was a preliminary (pilot) study; it presents some translation and adaptation issues, rising from the different schooling systems of the countries and also from different languages. It presents the solutions we have used. In study 2, 325 first-graders ( M = 6y 2mth; SD = 3mth) were assessed at the beginning of school year, using Slovenian version of PIPS-BA. At the end of the school year Academic Achievement Standards for Slovene, Math and Environmental Education were completed by pupils’ teachers. Literacy and numeracy skills of Slovenian first-graders are presented. PIPS-BA explains about one third of the variance in Slovene, one fourth of the variance in Math, and somewhat less in Environmental Education academic achievement. Some final thoughts on implementing the assessment are presented. 9 Introduction It is now widely accepted that assessment is an essential feature of sound edu- cational practice. Virtually all educational establishments have policies and practices in place to assess their pupils and students (Wilkinson, Watt, Napuk & Normand, 1998). The new emphasis on the importance of systematic assessment throughout primary schooling has led schools and policy makers to consider some form of systematic assessment for school entrants (Blatchford & Cline, 1992). The baseline assessment (BA) on-entry to school is defined as a measure of children’s knowledge, understand-ing, abilities and skills at the beginning of formal education (Wilkinson et al., 1998). Blatchford and Cline (1992) list five main reasons for assessment on school entry: 1. A basis for measuring future progress/attainment. There is growing appreciation of value-added results which take baseline score and social factors into account. BA enables comparison of the progress for pupils with similar baseline scores and social background. This means fairer comparison between classes, schools and areas. In a way it enables evaluation of the effectiveness of schools and introduces the concept of accountability (what will done with the BA score). 2. Getting a picture of the new intake as a whole, based on groups of children. Staff in school may be interested in descriptive information on children, giving some details on children’s skills and experience, which could be compared to information from previous years. Such information might help with general approaches to teaching during the first year at school. In addition, such information can help planning the allocation of resources. 3. Getting a profile of the new entrant. This information helps teachers with classroom organization and lesson planning to meet children’s individual learning needs. Blatchford in Cline (1992) refer to a study that found children’s knowledge of reading, writing and math on-entry to school varies widely and is related to educational attainments one, three and seven years later. BA would therefore enable teachers to respond to the needs of individual children and would make it less likely that schooling would simply reinforce knowledge differences already evident on school entry. 10 4. Identifying children who may have difficulties in school. These children might benefit from special help. These predictions are generally not stable for the accurate identification of the future learning difficulties, but do identify present problems and also start the process of identification for the future. 5. Professional development of teachers. The authors list this reason separately, in the conclusions. They say that systematic assessment together with progress information should provide a valuable basis for teachers to evaluate their own work. Wilkinson et al. (1998) state that there is considerable debate in the litera- ture as to whether any one baseline assessment scheme can adequately address each of the purposes. Over the last 20 years a variety of approaches have been developed, including rating scales based on general day-to-day classroom observation, guided observation schedules, approaches involving test-like structured tasks or questioning, and mixture of all these. Wilkinson et al. (1998) reviewed several on-entry schemes, including Performance Indicators in Primary Schools: On-entry Baseline Assessment (PIPS-BA). All reviewed schemes (a) serve similar purpose, (b) assess children’s competence in literacy and numeracy/mathematics, (c) are conducted in the first term of primary schooling and (d) enable value-added calculation. However, PIPS-BA has some additional desirable characteristics (some may overlap): › a ‘’test-form’’ (not a checklist, scale, narrative description), › a booklet or computer form (not based on observation or interviews): easy to use format (needs minimal training) and objective administration, › adaptive assessment, › time-efficient (20 minutes and not day-to-day assessment), › source of information is child’s performance on structured tasks (not teacher-or parent-reports), › curriculum–free, since the pupils have not been following a curriculum up to the school entry, › widely employed (over 4000 schools each year in different countries), and › internationally used (including non-English speaking countries), which allows international comparisons (see Tymms, Merrell & Jones, 2004). In continuation, PIPS-BA is presented, as well as translation, adaptation and application of the assessment into Slovenian context. 11 PIPS On-entry Baseline Assessment The PIPS-BA (including follow-up tasks) was originally developed in 1994 (Tymms et al., 2004), taking into account empirical rather than theoretical position (Tymms, 2001). It is firmly based in research and focuses on key aspects that are important in the early stages of schooling for later academic success (review in Tymms, 1999; 2001). Tymms, C. Merrell, Henderson, Albone and Jones (2007) refer to studies that show letter identification, phonological awareness in various forms, concepts about print and vocabulary as being good predictors of later reading. The body of research on predictors of mathematics is modest and on science, even smaller. PIPS-BA uses a combination of objective assessment and teacher rating to pro- vide teachers with valuable and reliable information about their pupil’s basic academic skills when they first start school (CEM Centre, 2005). This helps teachers with classroom organization and lesson planning to meet children’s individual learning needs. Assessment also provides a firm basis against which the value-added can be measured (taking into account the starting points of individual children). At the end of the year the assessment can be repeated, allowing the teachers to look at children’s progress over the year and to link that to their professional knowledge about the children (Tymms, 2001). The feedback also enables teaching staff to evaluate their performance in comparison to other schools. It is intended to promote school improvement by providing professionals high quality information. PIPS -BA is an assessment of early literacy and early math, which include four broad areas of assessment: vocabulary, phonics, early reading and early math. Questions are asked in a series of sub-units (Table 1, Figure 1). Assessment takes about 20 minutes per child and is usually administered by child’s teachers within first few weeks upon entry to school. The assessment is available in either text or computer-delivered (CD) format. In both cases the assessment is completed by an adult working with each child on a one-to-one basis. The text version comprises a colorful booklet that the adult and child work through together. The adult records the child’s responses on a pupil record sheet. With the CD version, the computer program presents the child with questions (orally through recorded sound files and visually on the screen). An adult uses the mouse to indicate the child’s responses, 12 which are then recorded on the computer. Test-retest correlations within and between the formats are .92 to .98 (Tymms, 2001). Table 1 Broad areas, sub-units and tasks in the PIPS-BA (adapted from Tymms et al., 2007) Sub-unit/Broad Vocabularya area Vocabulary Vocabulary – the child is asked to identify objects embedded within a picture. Phonological awarenessa Phonics The items in these sections are measures of phonological awareness, which is abbreviated to phonics. Repeating Words – the child hears a word and is asked to repeat it. Rhyming Words – the child selects a word to rhyme with a target word from a choice of three options. Early readinga Writing Writing – the child is asked to write his/her own name and the quality of writing is scored against examples. IAR Ideas about reading – based on the ideas developed by Marie Clay for the Concepts about Print assessment. Letters Letter identification – a fixed order of mixed upper and lower case letters. Reading Word recognition and reading. This starts with word recognition and moves on (advanced) to simple sentences that the child is asked to read aloud. The words within these sentences are high frequency and common to most reading schemes. Towards the end less common words and more complex syntax are used. Early numeracy/mathb Counting Counting and Numerosity – the child is asked to count objects. These are then hidden and the child is asked how many objects (s)he saw. Sums (informal) Sums – addition and subtraction problems presented without symbols. Numbers Digit identification – single, two-digits and three-digits. Math (advanced) More difficult math problems including sums presented with formal notation or in a text form (e.g. What is 3 more than 1?). Note. English text version of the assessment comprises of three separate booklets: Baseline, Follow-up and Extension (CEM Centre, 2005a, b, c). The Slovene PIPS-BA incorporates sub-units / tasks from the Baseline and Follow-up booklets. The tasks from the ‘’combined’’ assessment are presented in table 1 (advanced Reading and Math are from the follow-up booklet). a Assesses early literacy. b Assesses early math. 13 Figure 1. Examples of tasks in Reading, Sums and Math sub-units. Reading (advanced) Sums Math (advanced) The assessment forms a single coherent scale, explaining 73% of the variance. However, the second and the third factor were also identified (both accounting for 14% of variance) (Tymms et al., 2007). The assessment is presumably fractal (Andrich 2007, see Tymms et al., 2007) and for this reason the analyses look at the overall test and the sub-units. The PIPS-BA also has a section that measures child's attitudes towards different school-related activities. The CD version also includes personal and social-emotional development section that is assessed through teachers' ratings of key features. In continuation, we will only address sub-units presented in table 1. PIPS-BA has been translated into eight languages and is used in around 4000 schools each year in England, Scotland and Australia (Tymms et al., 2007). It is also used on a more modest scale in New Zealand and the Netherlands. In addition, it has been translated for the use in Germany, Lesotho, Thailand, France, Hong Kong and South Africa. Thus, large, longitudinal datasets are available for international comparisons research. Analysis of data from different population groups confirms that the assessment items retained their characteristics and relative difficulties (Tymms et al., 2004). 14 Translation and adaptation for the use in Slovenia In Slovenia, National assessments, covering areas of National Curriculum take place in the 6th and 9th grade of the nine-year compulsory schooling. Throughout compulsory schooling some evaluation studies are also conducted. In addition, Slovenia takes part in several international studies: TIMSS (Trends in International Mathematics and Science Study), PIRLS (Progress in International Reading Literacy Study), PISA (Programme for International Student Assessment) and others. However, neither of these includes on-entry assessment at the beginning of compulsory schooling nor is there a similar assessment for elective use in the country. In addition, we considered a growing autonomy of Slovenian schools in our decision to translate and adapt PIPS-BA. We opted for an adaptation of the already existing assessment rather than developing a new one, because we expected the two would produce similar results, but the former would be less time and cost consuming. PIPS-BA was chosen for its psychometric (for review see this paper, Method section in study 2) and other desirable characteristics (presented above). The main purpose regarding the adaptation of the assessment for the use in the Slovene educational and cultural context was to provide teachers rich information on pupils’ competencies at an early stage in order to help them with lesson planning, classroom organization, and also to start the process of identification of gifted or children with special educational needs. We believe that other purposes of the BA (listed above; Blatchford & Cline, 1992) are as important, but would become important at later stages of implementing the assessment (e.g. monitoring the children’s academic progress to provide feedback to teachers; international comparisons of on-entry chil-chil- dren’s skills/knowledge/competence). Basic guidelines were followed in the process of adaptation of PIPS-BA: 1. Preliminary translation of introductory chapters and all the tasks. 2. Consultation with CEM Centre. 15 3. Verification of the translation by Slovene experts (professor of Slovenian language, second English translator, professor of developmental psychology). 4. Consultation with CEM Centre. 5. Pilot study (135 students from 26 Slovene schools). 6. Improvements of the Slovene version. 7. Back translation from Slovene to English. 8. Verification by the CEM Centre. 9. Main study (326 students from 39 Slovene schools). In the translation process ‘’simple’’ translation was possible for all the Early numeracy/math tasks, Vocabulary and some of the Early Reading. Minor graphic changes were made (e.g. the words in the pictures were translated; writing of numbers 7 and 9 was adapted; see Figure 1). Figure 2. Two examples of graphic modifications: Slovene (left) and English (right) version. Translating the Phonics (Repeating and Rhyming Words) and Word Recog- nition task presented a challenge. After consultation with the CEM Centre Slovene translation and adaptation aimed at: › in the Repeating Words task, looking for words with similar phonetic structure and with the same number of syllables (e.g. mantle – mandelj (almond)). It was also considered whether a word has a meaning in the respective language (not all of the words were meaningful); › in the Rhyming Word task, considering (a) children’s acquaintance with the words (they should become less familiar towards the end of the task); (b) the type of distracters resembling the English original, e.g. all the pictures present fruits or have the ‘’o’’ sound in the middle; › in the Word Recognition task, accounting for (a) children’s acquaintance with the words (they should become less familiar towards the end of the task); (b) length of the Slovene word in comparison to the English words; (c) the same first or middle letters in the words (e.g. all the words begin with S). 16 The original stimuli were retained as much as possible; however, some pictures had to be replaced. Another problem that we encountered was the fact that letter acquisition is different between countries. English children acquire upper and lower case simultane-ously, whereas Slovene children acquire most upper case letters first before acquiring the lower case ones. This led to changes in scoring for the task Writing (Vidmar & Zupančič, 2006a). Accordingly, the letters in Letter identification (Figure 3) and some other tasks were changed to upper case only. We also dropped the letters from English alphabet that are not used in Slovene (X, Y, W) and replaced them with Slovene specific letters (Č, Š, Ž). Figure 3 . Excerpt from Letter identification task: Comparison of Slovene (left) and English (right) version. 17 Study 1: Pilot version of Slovene PIPS-BA The purpose of this study was to apply the pilot version of Slovene version of PIPS-BA and based on results to make necessary modifications of the assessment. Method A sample of 135 first-graders (44% girls) participated in the pilot study (which was part of a larger longitudinal project). The students’ parents were fully informed on the purpose and methods of the study. Only the students whose parents gave a written consent were included in the sample. Children were six years old ( M = 6 years 1 month; SD = 3 months) and had just entered one of the 26 participating state-funded schools in different regions of the country (1 to 20 students per school participated). PIPS-BA was individually administered during first seven weeks of school. Results Data were analyzed using Winstep (Linacre, 2007) which produced Rasch measurements (difficulty parameters) for each question. It also produced correlations between the item and total score. Both are presented in table 1. We also analyzed distribution of correct and incorrect answers for each item to determine whether the item was too easy or too difficult (using SPSS). Table 2 Rasch measures and item-total score correlations for PIPS-BA items Rasch Corr. Rasch Corr. Rasch Corr. with with with Item measure total Item measure total Item measure total score score score WRIT_ -0.49 0.43 RHYME 1 2.02 0.11 IAM 1 / / SLO RHYME 2 1.30 0.24 IAM 2 -3.30 0.11 VOCAB 1 -4.01 0.05 RHYME 3 0.34 0.24 IAM 3 / / 18 VOCAB 2 / / RHYME 4 0.76 0.25 IAM 4 / / VOCAB 3 -4.01 0.07 RHYME 5 -0.02 0.19 IAM 5 -3.30 0.12 VOCAB 4 / / RHYME 6 0.06 0.33 IAM 6 -3.30 -0.15 VOCAB 5 -4.01 0.05 RHYME 7 0.59 0.25 IAM 7 -3.30 0.07 VOCAB 6 0.15 0.20 RHYME 8 -0.72 0.30 VOCAB 7 1.37 0.02 RHYME 9 1.02 0.14 COUNT 1 VOCAB 8 -1.96 -0.06 COUNT 2 -3.30 0.24 VOCAB 9 -1.33 0.02 LETTERS 1 0.72 0.70 COUNT 3 -1.20 0.06 VOCAB 10 -2.57 0.21 LETTERS 2 -1.55 0.38 COUNT 4 -3.17 0.15 VOCAB 11 -2.33 -0.04 LETTERS 3 0.38 0.63 VOCAB 12 -0.42 0.16 LETTERS 4 0.06 0.62 SUMS 1 -1.96 0.21 VOCAB 13 -2.13 0.04 LETTERS 5 0.50 0.69 SUMS 2 -1.23 0.18 VOCAB 14 / / LETTERS 6 1.04 0.55 SUMS 3 -0.82 0.16 VOCAB 15 -1.05 0.31 LETTERS 7 -0.16 0.61 SUMS 4 -0.97 0.31 VOCAB 16 -1.33 0.26 LETTERS 8 -0.22 0.62 SUMS 5 -1.14 0.30 VOCAB 17 -0.20 0.23 LETTERS 9 0.97 0.69 SUMS 6 -0.13 0.14 VOCAB 18 -1.81 0.16 LETTERS 10 -0.11 0.60 SUMS 7 -0.24 0.19 VOCAB 19 4.72 0.06 LETTERS 11 0.92 0.68 SUMS 8 1.35 0.30 VOCAB 20 0.55 0.15 LETTERS 12 0.18 0.66 VOCAB 21 2.18 0.25 LETTERS 13 -0.21 0.61 NUMBERS 1 -1.33 0.42 VOCAB 22 0.99 0.39 LETTERS 14 0.29 0.72 NUMBERS 2 -2.13 0.32 LETTERS 15 0.68 0.69 NUMBERS 3 -1.67 0.28 IAR 1 -2.57 0.10 LETTERS 16 0.68 0.74 NUMBERS 4 -1.55 0.37 IAR 2 -4.01 0.02 LETTERS 17 0.47 0.63 NUMBERS 5 -1.43 0.38 IAR 3 -2.88 0.05 LETTERS 18 0.10 0.65 NUMBERS 6 -0.92 0.42 IAR 4 1.49 0.33 LETTERS 19 0.81 0.62 NUMBERS 7 3.26 0.24 IAR 5 0.15 0.35 LETTERS 20 1.13 0.60 NUMBERS 8 -0.03 0.48 IAR 6 0.16 0.08 LETTERS 21 0.69 0.69 NUMBERS 9 -0.39 0.52 IAR 7 2.37 0.13 LETTERS 22 1.23 0.58 NUMBERS 10 -1.59 0.33 IAR 8 1.33 0.32 LETTERS 23 1.12 0.66 NUMBERS 11 -0.49 0.13 IAR 9 2.12 0.18 LETTERS 24 1.42 0.58 NUMBERS 12 0.34 0.33 IAR 10 -0.40 0.18 LETTERS 25 0.66 0.59 NUMBERS 13 0.34 0.37 LETTERS 26 0.88 0.65 NUMBERS 14 1.43 0.23 REPEAT 1 / / LETTERS 27 3.41 0.51 NUMBERS 15 1.97 0.24 REPEAT 2 -2.13 0.05 LETTERS 28 0.68 0.67 NUMBERS 16 1.57 0.29 REPEAT 3 1.60 0.32 NUMBERS 17 2.49 0.10 19 REPEAT 4 0.47 0.04 WORDS 1 2.87 0.59 NUMBERS 18 1.80 0.22 REPEAT 5 -1.33 0.18 WORDS 2 2.55 0.61 NUMBERS 19 1.03 0.24 REPEAT 6 -1.14 0.17 WORDS 3 2.14 0.72 NUMBERS 20 -1.00 -0.01 REPEAT 7 -0.75 0.28 WORDS 4 2.18 0.71 NUMBERS 21 4.07 0.18 REPEAT 8 1.06 0.23 WORDS 5 -0.38 0.46 NUMBERS 22 3.89 0.28 WORDS 6 -0.99 0.17 WORDS 7 0.74 0.19 WORDS 8 -0.89 0.08 WORDS 9 2.67 0.43 WORDS 10 1.92 0.56 WORDS 11 1.13 0.47 WORDS 12 2.57 0.78 WORDS 13 1.67 0.53 WORDS 14 0.72 0.25 Note. / is shown for items for which all students gave correct answers (Rasch measure cannot be computed). IAR = Ideas about Reading. IAM = Ideas about Math. Table 1 shows Rasch values in the present study varied from -4.01 to 4.72 (theoretically they are not limited; Hambleton, Swaminathan & Rogers, 1991). It also shows that almost half of Vocabulary items do not correlate with total score. Additional descriptive analyses revealed that in: › Vocabulary: for 13 out of 22 objects over 90 % students gave correct answers › Ideas about reading: over 95 % students gave correct answers on 3 out of 10 items › Ideas about Math: over 98 % students gave correct answers on each item › Counting: over 89 % students gave correct answers on each item › Sums: over 75 % correct answers on each but one item › First Numbers: over 91 % students gave correct answers on each item › Second Numbers: over 74 % students gave correct answers on each item Majority of items in other tasks (Repeating Words, Rhyming Word, Writ- ing, Letter identification, Word recognition) did not have such high percentage of correct answers. 20 Discussion Results of a pilot study revealed an interesting issue. In several tasks ceiling effect was found – most of the Vocabulary, Early Math and some of the Early Reading items were correctly answered by the majority of children. Phonics and most of the Early Reading showed satisfactory levels of difficulty. This can be explained by the fact that school entrants in Slovenia are one to two years older in comparison to the children in UK (Tymms et al., 2004). The findings led us to adding more difficult tasks – advanced Reading and Math items (in English version these tasks comprise follow-up assessment). Note, that table 1 presents complete (not pilot) version of the assessment. For translation and adaptation of the added tasks steps 1 through 4 were followed (see above). In addition, we removed some of the easy items and replaced them with more difficult ones, e.g.: › in the Vocabulary task, less familiar objects were added (pictures were changed), › in the Ideas about Reading task, less familiar concepts were added (e.g. capital letter, circumflex), › Ideas about Math task was dropped, › in the Sums task, addition and subtraction problems over ten were added, › in the Numbers tasks, some one-digit numbers were removed and replaced by two-digits. We also revised the rank-order of the letters in the Letter identification task. We decided to adopt the order of learning the letters in Slovene schools (the pilot letters were ranked by their frequency in Slovene language). Conclusions The assessment was revised in accordance with issues revealed in the pilot study (some easy items were replaced or dropped, more difficult tasks were added). The resulting version of PIPS-BA was back translated by an independent translator and was verified by the CEM Centre. 21 Study 2: Final version of Slovene PIPS-BA The purpose of study 2 was to present the literacy and numeracy skills of Slovenian first-graders and to explore how PIPS-BA scores predict academic achievement at the end of the first grade, taking into account enrolment into preschool and children’s non-verbal cognitive ability. Method Participants A follow-up sample of 326 first-graders (49% girls), their mothers and teachers participated. At the beginning of the school year (Wave 1, W1), the children were six years old ( M = 6 years 3 months; SD = 3 months) and had just entered one of the 39 participating state-funded schools in different regions of the country. Mothers provided data on their education ( M = 12.7 years; SD = 3.1 years) in W1. The children’s teachers ( N=87) participated in the W2, i.e. at the end of the school year. Teachers had 16 years of education (university degree in school education). Each teacher assessed between 1 and 14 children. One third of children was not enrolled into preschool, 36% were enrolled into preschool for 3 years (entered at age 3), and 31% of them were attending preschool for 5 years (entered at age 1). Children’s gender was not significantly related to his/her age of entry to preschool ( p(χ2) = .29), but children with less educated mothers were enrolled into preschool at an earlier age than were children whose mothers completed more years of schooling ( F(2, 258) = 12.4, p< .001). Instruments Performance Indicators in Primary Schools: On-entry Baseline Assessment (PIPS - BA, CEM Centre, 2005a, b; Slovene version Vidmar & Zupančič, 2006b). The assessment measures four broad areas described above (see table 1). The text version was employed. It is an A4 booklet, in which instructions for administrator are printed on the left hand pages and pictures to be used in the assessment are found on the right. An adult works through the booklet together with a child, marking child’s answers on 22 a pupil record sheet. The assessment is administered individually within the first seven weeks of school and takes about 20 minutes per child. Sub-units reliabilities (test-retest) varied from .34 for Ideas about Reading to .99 for Reading (CEM Centre, 2001; Tymms at al., 2007). Reliabilities of the four broad areas were estimated to .84, .65, .75 and .78 for Vocabulary, Phonics, Early Reading and Early Math, respectively. Overall test-retest reliability was .98 (Tymms et al., 2004). Internal reliabilities of the Slovene version were .74, .62, .95 and .92 for Vocabulary, Phonics, Early Reading and Early Math, respectively. The original English version of PIPS-BA strongly predicted academic achieve- ment: all of the tasks significantly predicted outcomes 3 years later (Tymms, 1999); corrected for unreliability (attenuation) the multiple correlation (all tasks used as predictors) was .71 and .66 for reading and math, respectively (Tymms, 1999). These correlations were somewhat lower for the outcomes seven years later (Tymms et al., 2007). PIPS On-entry total score correlated with reading and math three year later (.70 and .65, respectively, Tymms, 2001). Total score correlations with mathematics, reading, science and the composite measure 7 years later varied from .55 to.63 (Tymms et al., 2007). The authors concluded that PIPS-BA predicted nearly 50% of the variance in the outcome measures of pupils leaving primary school. Attainment of Performance Standards in Slovene, Math and Environmental Edu- cation (Zupančič, 2006) is a set of items/standards that are entirely curriculum based. The instrument consists of minimal and advanced standards proposed in the National Curriculum; Slovene consists of 62 items (χ = .99), Math of 25 (χ = .98) and Environmental Education of 44 items (χ = .99). For each item, the teacher rated whether the child underachieves, achieves or overachieves the standard on a 6-point scale (0 – does not achieve, 5 – overachieves greatly). A separate score was obtained for each course by calculating a mean of teacher’s answers on the respective items. Colored Progressive Matrices (CPM; Raven, Raven, & Court, 1999) includes three sets of 12 tasks, designed to assess general non-verbal cognitive ability in children under the age of 11 years. The tasks consist of a matrix pattern with a section missing, and six alternative responses, one of which completes the missing section of the matrix. Children were asked to select the missing section. Within each set, the difficulty of tasks gradually increased. The final score was obtained by summing the correct answers. The split-half reliabilities in early childhood ranged from 0.65 to 0.90. Retest reliability coefficients ranged from 0.81 to 0.95, from 0.68 to 0.92, and from 0.95 to 0.71 over 10 days, one-month, and one-year time period, respectively. In preschool children, the correlations between the CPM and the Primary Mental Abilities test were moderate and the validity of the CPM increased in older children (for a review, see Raven et al., 1999). In the present study CPM scores ranged from 10 23 to 34 ( M = 23.07, SD = 4.84). Compared to the Slovene normative data for children aged 6;9 years score of 23 in just below the 50th percentile (which has a score of 24) (Raven et al., 1999). Procedure A year prior to the on-set of the study, the 50 state-funded schools were sys- tematically selected from a listing provided by the Ministry of Education and Sports. Eleven schools either refused to participate or none of the future first-graders fit the preschool enrolment criterion. School coordinators (appointed by the headmaster) mediated researchers’ contact with parents near a beginning of the academic year. Parents were given a short questionnaire concerning the child’s enrolment into preschool. Written consents forms were offered to parents whose child: (1) was not enrolled into preschool, or (2) attended preschool for 3 years, or (3) attended preschool for 5 years. Approximately 50% of parents signed the consent. In W1 (within first seven weeks of the school year), trained research assistants made an appointment at school and individually administered PIPS-BA with the participating children (about 20 minutes per child). The procedure took place in a separate and quiet room within the child’s school. Through school coordinators, mothers were given questionnaires (not related to the purpose of this paper), in which they gave information on the years and level of education. A few months later, the trained research assistants administered CPM to each participating child individually (it took from 10 to 20 minutes per child) in the same place as PIPS-BA. In W2 (at the end of the school year), the teachers assessed the target children’s academic achievement using the Attainment of Performance Standards in Slovene, Math and Environmental Education. The completed material was returned to the school coordinator who sent it to the researchers. 24 Results Literacy and numeracy skills of Slovenian first-graders Descriptive statistics and some additional frequency data are presented. Table 3 Descriptive statistic for tasks in PIPS-BA Broad area Sub-unit Task Na M (Max) % SD Vocabulary Pictures 321 9,74 (22) 44% 5,00 Phonics Phonics Repeating Words 324 5,97 (8) 75% 1,43 Rhyming Words 326 4,76 (9) 53% 2,85 Early Reading Writing 326 4,10 (5) 82% 0,90 Ideas about Reading 324 4,11 (10) 41% 1,98 Letters Name letter 326 0,89 (1) 89% 0,32 Upper case 302 15,22 (25) 61% 9,29 Lower case 305 1,10 (3) 37% 1,17 Reading Word Recognition 326 3,72 (14) 27% 5,41 Reading 324 9,30 (69) 13% 20,75 Early Math Counting 326 3,65 (4) 91% 0,90 Sums (informal) 322 4,96 (8) 62% 2,05 Numbers 1st and 2nd numbers 323 6,85 (10) 69% 3,51 Two digit numbers 323 1,87 (9) 21% 3,19 Three digit numbers 324 0,20 (3) 1% 0,63 Advanced Math 320 3,13 (17) 18% 3,04 Note. Minimum score for all tasks is 0. This is adaptive assessment with increasing difficulty within a task. For the questions that were too difficult (child was not asked) zero score is assumed. a N varies due to missing data. Table 3 shows that on average children scored highest in Counting and Name Letter. They scored lowest on Reading and Three Digit numbers, both also having a high variance. Vocabulary consists of 22 objects (items) to point. On average children cor- rectly pointed almost half of the objects. Phonics comprises of two tasks. Rhyming task showed that on average chil- dren got 5 rhymes correctly, however 9% of children scored zero. Repeating Words task proved relatively easy, with children correctly repeating 6 words. 25 Early Reading consists of seven tasks. The Writing task revealed that upon entry to school 56% of children could write their first name and 31% of children could write their full name. Ideas about Reading task showed that all children have developed some concepts about reading, with only one child scoring zero. Majority of the children could point to a word (53%) or a letter (78%). Few percent (<10%) of children could point to a capital letter, coma or circumflex. Letters sub-unit showed that 11% of children could not name the first letter of their name when it was shown to them, 4% of the children could not identify a single letter of the alphabet and, 22% of children knew all the letters. On average, children identified 15 letters on-entry to school. Almost half (40%) of the children could not identify any of the three lower case letters, one fifth identified all three correctly. Word Recognition task showed that two thirds of children did not recognize any of the words and one tenth recognized all of the words correctly. SD (table 3) shows that there was quite a variation for this task. Reading also had a large SD, likely due to a wide range of possible values (0 to 65) and the fact that majority of the children (75%) scored zero. Less than 10% of children could read on-entry to school (they scored over 60 points). Early Math captured six tasks. Counting was one of the easiest tasks, with 85% of children scoring maximum. The Sums revealed that most of children can do informal sums (with visual presentation). However, only 50% of children can do simple subtraction problems with a minuend over 10 and the difference under 10 (e.g. 16-8) and only 20% can do simple addition problems with a sum over 10 (e.g. 8+5). Review of the literature (Geary, 2004; Geary, Hoard, Byrd-Craven, Nugent, & Numtee, 2007; Geary, Hoard, Byrd-Craven & DeSoto, 2004) shows that English does not have a term for this milestone in development of the math skills; therefore we use a more descriptive phrase. In Numbers task, 40% of children recognized all one-digit numbers and teens, 11% of children recognized all two-digits (over 20) correctly and 65% scored zero. For three digits 4% scored all correctly and 88% scored zero. Advanced math problems proved difficult with an average score of 3 points (out of 17), and none of the children scoring maximum. However, 29% of children scored a point on a formal sum question (e.g. Can you do 6+4?). Longitudinal predictions of attainment in Slovene, Math and Environmental Education The predictive validity of PIPS-BA (Vocabulary, Phonics, Early Reading, and Early Math) for later attainment in Slovene, Math and Environmental Education was investigated. Linear regression analysis was performed for each of the courses separately. Academic achievement in the specific subject was included as a criterion. PIPS scores on four broad areas, non-verbal intelligence score and enrolment into preschool were included as predictors. 26 The PIPS scores were calculated with Winstep (Linacre, 2007) which produced Rasch measurements (difficulty parameters) for each child and each question. These values were the basis for calculating child's broad area PIPS scores, which we employed in the regression analysis. Rasch values are theoretically not limited (Hambleton, Swaminathan & Rogers, 1991), but in the present study they varied from -8.11 to 6.46. For attainment scores cases with 10% or more missing data were removed from the analysis. Missing value analysis was performed separately for each course; 11, 12 and 9 cases were removed for Slovene, Math and Environmental Education, respec- tively. For cases with less than 10% missing values no estimations and replacements were made; mean scores for Slovene ( M = 1.9; SD = 0.7), Math ( M = 2.0; SD = 0.7) and Environmental Education ( M = 2.1; SD = 0.6) were calculated with available data. There was no missing data for the CPM scores. There was also no missing data for enrolment into preschool. Table 4 PIPS-BA and CPM scores as predictors of school attainment at the end of the first grade SLOVENE (N=315 a) MATH (N=314 a) ENVIRON.EDUC. (N=317 a) predictor ΔR2 (χ) predictor ΔR2 (χ) predictor ΔR2 (χ) PIPS Reading .26 (.23) *** PIPS Math .22 (.37) *** PIPS Math .10 (.24) *** PIPS Vocabulary .06 (.23) *** PIPS Reading .01 (.16) * PIPS Vocabulary .02 (.17) ** PIPS Math .03 (.18) *** CPM .01 (.13) * total R2 .36 .23 .12 Note. Adjusted R2 are presented. β = a standardized coefficient of the estimated regression model. Only statistically significant predictors are presented. Total R2 = the overall predictive power of the significant predictor variables. a N varies due to missing data. * p < .05 , ** p < .01, *** p < .001. Table 4 displays a summary of the regression analysis. It shows that PIPS-BA scores predicted first-graders’ academic performance at the end of the school year. All χ values were positive, indicating that high PIPS scores predicted high attainment scores, and vice versa low PIPS scores predicted low attainment scores. PIPS Early Math score predicted attainment in all of the three courses, however it is most predictive of academic achievement in Math. Attainment in Slovene at the end of the school year was strongly predicted by PIPS Early Reading, Vocabulary and Early Math, but also by non-verbal intelligence ( R 2 27 of .26 or higher is described as a large effect size; Cohen, 1988). Attainment in Math was moderately predicted by PIPS Early Math and Reading; almost one quarter of the variance was explained (R 2 between .13 and .25 represents a medium effect size). PIPS scores were somewhat less predictive of attainment in Environmental Education with one tenth of the variance explained by PIPS Early Math and Vocabulary. Enrolment into preschool did not appear as significant predictor of academic achievement at the end of first grade. The contribution of CPM score was statistically significant only for the prediction of attainment in Slovene, and even that was very low contribution of 1%. Discussion In the study 2, Slovene version of PIPS-BA was administered to school en- trants in the beginning of the school year. Their literacy and numeracy skills are presented as well as predictive value of PIPS-BA scores for academic achievement at the end of the first grade. Studies single out the skills of letter and number identification as being important for later academic achievement (Tymms, 1999). However, Tymms et al. (2007) found that the predictive power of these two skills changed with age. In their study the best indicator of later achievement in mathematics, reading, and science was the ability to identify numbers and do informal sums, whereas some studies list identify-ing letters. Therefore, we focus here on these two skills. Letter identification task revealed that 4% of the children could not identify a single letter of the alphabet; 20% of children correctly identified 3 letters or less, whereas almost the same percentage of pupils identified all the letters correctly. This means that almost half of the population of Slovene entrants was found on the op-posite sides of the distribution. In addition, a majority of children failed to recognize words and read. However, almost 10% of school entrants can read (they recognized all of the words and made very few mistakes in reading). As already noted by Tymms et al. (2001), this proves that pupils’ starting points vary enormously. Number identification task showed that 40% of children recognized all one digit numbers and teens, whereas almost 8% of children failed to recognize any single digit (no teens included). In England, the percentage of children who failed to recognize any single digit is 17%; however, pupils in England are one to two years younger than the school entrants in Slovenia (Tymms, 2001). The same author reported that 0,3% of pupils identified all the digits correctly (singe, double, triple), whereas in our sample 3% of pupils were able to do that. The majority of Slovene children could do informal sums, but they still made mistakes. Half of the children could do simple subtraction problems with a minuend over 10 and the difference under 10 (e.g. 16-28 8), but only 20% could do simple addition problems with a sum over 10 (e.g. 8+5). Subtraction at this point was easier, because children had a visual presentation of the number, from which they had to subtract; on the contrary, when they did addition the second summand was not visually presented – children had to imagine the objects they were adding. We presume that a large variation in starting points presents a great challenge to lesson planning. Having data on individual pupils available at an early stage can help compensating for the deficits (not necessarily teaching children letters or numbers) and further developing the skills. Obviously, this is not incorporated in the assessment itself. Therefore, for the assessment to be effective, it has to be meaningfully included in the monitoring and counseling system. There has been some debate on reasons for including additional predictors in baseline assessments, if number and letter identification alone provide good predictions. Tymms (1999) reported that most tasks from the PIPS-BA could form the basis of a good predictor, provided that more items were included and that the child did not become distracted or bored. However, combining these different indicators into one 20-minute assessment with four broad areas made the assessment interesting for a child. In addition such assessment was also more complete, because different measures/aspects of the same concept were included. Moreover, general BA score carries little implication for lesson planning and adjusting to pupils’ educational needs. On the other hand, detailed information on child’s score on either each task or each broad area facilitates the process of lesson planning. Tymms et al. (2007) believe PIPS-BA is a measure of general baseline skills, but holding more specific information on different aspects of the same concept. However, if future data for Slovenia reveal that some parts of the assessment consistently lack predictive power, omission should be considered. Predictive validity PIPS-BA was administered in the beginning of the academic year. At the end of the same school year, teachers evaluated pupils’ attainment in three courses: Slovene, Math and Environmental Education. PIPS-BA demonstrated differential predictive power for the attainment in Slo- vene (strong prediction), Math (moderate prediction) and Environmental Education (weak prediction) (Cohen, 1988). Early Math was the most consistent predictor, explaining significant portion of variance in all three courses. On the contrary, Phonics was the only PIPS score that demonstrated lack of predictive power for the school attainments at the end of the academic year. Similarly, enrolment into preschool was not predictive of the criterion. Non-verbal intelligence explained negligible (but statistically significant) 1% of variance in Slovene attainment. 29 The findings are quite consistent with those reported by Tymms et al. (2007). In their study the regression analysis included PIPS broad areas, home background, sex, and age as predictors and different outcome measures seven year later as criterion (e.g. Reading, Math and Science achievement). Predictors explained 35% to 38% of the variance in achievement scores. Tymms (1999) performed another set of multiple regression analysis employing PIPS tasks rather than the four broad areas scores as predictors. Predictors explained 44% to 50% of the variance in criteria. Tymms et al. (2007) also reported findings of another study in which different assessments (Infant Index, pupil background measures, early literacy skill and letter-knowledge), administered at 5, predicted 30% of variance in measures of reading, math and science at age 11. These results are similar to the results in the present study. However, a relatively weak prediction for the attainment in Environmental Education was found in the present study. The finding that the most important predictor for all of the outcomes was Early Math is also similar to both studies. In general, predictive power of the Slovene assessment was relatively lower than of the English version. The differences may have occurred due to the measures of academic achievement employed in our study, i.e. curriculum based attainment standards. These standards were found in the National Curriculum, but were loosely defined and this left teachers a chance for different interpretations. Thus, the employed measures of academic achievement were not school grades or results on academic achievement tests as they are not used in Slovenia until grade 4. Actually any assessments in terms of academic achievement or efficiency are strongly avoided. The teachers keep descriptive school records and provide parents written reports (essay-like) on how their child was doing in school across the academic year. The strengths of a child are emphasized, while eventual weaknesses are carefully ‘’wrapped’’. These tendencies may also have had an effect on teachers’ ratings of children’s Attainment Standards in our study – the means were high with relatively low standard deviations (Zupančič, 2006) and the latter may have underestimated the relationships under consideration. It is expected that when more appropriate measures of academic achievement are available, (e.g. National Assessment, Grade Point Average) the predictive relations will change. The former measure is not available until grade 6 and the later until grade 4. A lack of predictive power of PIPS Phonic score (comprising Rhyming Words and Repeating Words tasks) was found in the present study. This could be explained by the easiness and low variance of the Repeating Words task (see table 3). In addition, Phonics demonstrated the lowest internal reliability (χ=.62). The Repeating Words task decreased the reliability of the Phonics section (χ=.50). We should consider changing this task. Our findings differ from those reported by Savage et al. (2004; 2007) that phonological awareness was a significant unique predictor of all the outcomes. However, Tymms et al. (2007) also found that correlations between the PIPS Phonics and outcome measures were generally lowest, never getting as high as .4. 30 Tymms (1999) demonstrated that the equations for predicting reading and math attainment were similar (based on PIPS tasks, not broad areas). Our findings show that there are some differences across prediction equations for the three courses. Out of the three courses the attainment in Slovene was predicted best. In addition, three PIPS scores explained significant proportion of variance, whereas two PIPS scores only were predictive for attainment in Math and Environmental Education. The significant predictions are consistent with the content area of each course: Early Reading and Vocabulary were strong predictors of attainment in Slovene, and Early Math was moderate predictor of attainment in Math. The results support the validity of an adapted version of PIPS in Slovenia. Conclusions and limitations In conclusion, it appears that the process of translation and adaptation process of PIPS-BA for the use in Slovenia was done thoughtfully and carefully. Slovene version of PIPS-BA showed good reliabilities and considerable predictive validity against academic achievement at the end of the school year, explaining between one tenth and one third of the variance. It is expected that these predictions will improve with employment of more objective attainment measures in the future. We find it crucial that all the alterations prior and after the pilot study were discussed and approved by the CEM Centre. However, the adaptations were considerable and made the assessment more useful for Slovenia, but less appropriate for international comparisons. Nevertheless, they are still possible for the items that were unaltered, while accounting for children's age as well (example Tymms & Jones, 2006; Tymms et al., 2004). In the future, available data will be used to further improve the assessment (e.g. Repeating Words task). Future studies should also focus on the international comparison of the on-entry skills. Our main aim for the future is to explore predictive validity of the PIPS-BA for academic achievements 2, 6 and 9 years later. 31 Final remarks This paper discusses general rationale behind baseline assessments and presents in detail British assessment PIPS-BA. The process of translation and adaptation into Slovenian educational and cultural context is described. The results of the PIPS-BA on two samples of Slovenian first-graders are presented. Results showed large variation in pupils’ starting points and significant links to academic achievement, both of which only demonstrate the necessity of using baseline assessment in practice. Such assessment would help identify the children who require different kinds of teaching approaches or different allocation of resources. Therefore, we strongly believe that this assessment could bring various improvements into the educational system. However, the sole existence and use of the assessment cannot provide such improvements. Set-ting up a meaningful and effective training, monitoring and counseling system is crucial. It is our belief that incorporating BA in some sort of support system is necessary to fulfill the purpose of the assessment. In addition, there are some limitations to the use of BA. First, BA taking just 20 minutes has a limit to what can be assessed. Such BA can only act as an initial screen and may identify children with general learning abilities/difficulties. Further, BA cannot specifically predict academic future of individuals. It should, however, help express this in a probability language (Tymms, 1999). 32 References Blatchford, P., & Cline, T. (1992). Baseline assessment for school entrants. Research Papers in Education, 7(3), 247-269. CEM Centre (2001). Performance Indicators in Primary schools: technical report 2001. CD-ROM Version. Durham: University of Durham, UK. CEM Centre (2005a). Performance Indicators in Primary schools: On-entry Baseline Assessment. [Booklet format.] Durham: University of Durham, UK. CEM Centre (2005b). Performance Indicators in Primary schools: On-entry Baseline Assessment: Follow-up. [Booklet format.] Durham: University of Durham, UK. CEM Centre (2005c). Performance Indicators in Primary schools: PIPS On-entry Baseline Follow-up: Extension material. [Paper format.] Durham: University of Durham, UK. CEM Centre (2007). Reception assessment. Downloaded October 11, 2007 from www. pipsproject.org Cohen, J. (1988). Statistical power analysis for the behavioural sciences. New York: Lawrence Erlbaum Associates. Geary, D.C. (2004). Mathematics and learning disabilities. Journal of Learning Disabilities, 37(1), 4-15. 33 Geary, D.C., Hoard, M.K., Byrd-Craven, J. & DeSoto, M.C. (2004). Strategy choices in simple and complex addition: Contributions of working memory and counting knowledge for children with mathematical disability. Journal of Experimental Child Psychology, 88, 121-151. Geary, D.C., Hoard, M.K., Byrd-Craven, J., Nugent, L. & Numtee, C. (2007). Cognitive mechanisms underlying achievement deficits in children with mathematical learning disability. Child Development, 78(4), 1343-1359. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park: Sage. Jones, P., & Tymms, P. (2006). England and Hong Kong: Is it possible to compare the starting points of children in Cantonese and English schools? Symposium: On-entry baseline: Translation, adaptation and progress Measures. [Paper presented at 5th Conference of the International Test Commission, 6th – 7th July, 2006, Brussels, Belgium]. Linacre, J. (2007). Winsteps Rasch measurement. Software version 3.64.0. Raven, J., Raven, J. C., & Court, J. H. (1999). Priročnik za Ravenove progresivne matrice in besedne lestvice. 2. zvezek. Barvne progresivne matrice. [Manual for Raven’s progressive matrices and vocabulary scales. Section 2. Colored progressive matrices] Ljubljana: Center za psihodiagnostična sredstva. 34 Savage, R., & Carless, S. (2004). Predicting curriculum and test performance at age 7 years from pupil background, baseline skil s and phonological awareness at age 5. British Journal of Educational Psychology, 74, 155-171. Savage,R., Carless, S. & Ferraro, V. (2007). Predicting curriculum and test performance at age 11 years from pupil background, baseline skills and phonological awareness at age 5 years. Journal of Child Psychology and Psychiatry 48(7), 732-739. Tymms, P. (1999). Baseline assessment, value-added and the prediction of reading. Journal of Research in Reading, 22(1), 27-36. Tymms, P. (2001). The development of a computer-adaptive assessment in the early years. Educational and Child Psychology, 18(3), 20-30. Tymms, P., Merrell, C. Henderson, B., Albone, S., & Jones, P. (2007). Links between children’s starting points and finishing points in primary school. [Paper presented at the EARLI Conference Budapest, August 28th – September 1st, 2007, Budapest, Hungary] Tymms, P., Merrell, C., & Jones, P. (2004). Using baseline assessment data to make international comparisons. British Educational Research Journal, 30(5), 673-689. Vidmar, M., & Zupančič, M. (2006a). Translation and adaptation issues when introducing PIPS into Slovenian context. Symposium: On-entry Baseline: Translation, adaptation and progress measures. [Paper presented at 5th Conference of the International Test Commission, 6th – 7th July, 2006, Brussels, Belgium]. 35 Vidmar, M., & Zupančič, M. (2006b). PIPS: Preizkus temeljnih kompetentnosti otrok ob vstopu v šolo – začetni preizkus in spremljanje. [Neobjavljeno gradivo.] [PIPS: On-Entry Baseline Assessment and Follow-up. Unpublished.] Ljubljana: Pedagoški inštitut. Zupančič, M. (2006) Predloge za oceno doseženih standardov znanja v 1. razredu. [Neobjavljeno gradivo.] [Attainment of Performance Standards in Slovene, Math and Environmental Education. Unpublished.] Ljubljana: Filozofska fakulteta, Oddelek za psihologijo. Wilkinson, J. E., Watt, J., Napuk, A., & Normand, B. (1998). Baseline assessment literature review and pre-school record keeping in Scotland. Interchange, 55, 4-19. LIST OF TABLES 36 List of tables Table 1. Broad areas, sub-units and tasks in the PIPS-BA (adapted from Tymms et al., 2007) Table 2. Rasch measures and item-total score correlations for PIPS-BA items Table 3. Descriptive statistic for tasks in PIPS-BA Table 4. PIPS-BA and CPM scores as predictors of school attainment at the end of the first grade List of figures Figure 1. Examples of tasks in Reading, Sums and Math sub-units Figure 2. Some graphic changes: Comparison of Slovene (left) and English (right) version Figure 3. Letter identification task: Comparison of Slovene (left) and English (right) version 37 Z N A N S T V E N A P O R O Č I L A P E D A G O Š K E G A I N Š T I T U T A ( E L E K T R O N S K A Z B I R K A ) V toku leta 2009 je pričel Pedagoški inštitut na svoji spletni strani (http://193.2.222.157/ Default.aspx) objavljati znanstvena poročila v novi elektronski zbirki Znanstvena poročila Pedagoškega inštituta. Uredniški odbor zbirke v letu 2009 sestavljajo Janez Kolenc, Anton Kramberger, Darko Štrajn Zbirka služi naslednjim ciljem: 1. promociji in diseminaciji raziskovalnih dosežkov članov PI, tudi študentov in gostujočih kolegov, v obliki končnih raziskovalnih poročil za tretje stranke ali v obliki drugih delno zaokroženih znanstvenih del, z navedbo že opravljenih kolegialnih presoj, 2. objavi prispevkov k širšim akademskim razpravm znotraj in izven PI, s pogojem, da so so/avtorji prispevkov notranji ali zunanji raziskovalci PI, sodelujoči raziskovalci PI ali doktorski študenti v okviru PI. S C I E N T I F I C R E P O R T S O F T H E E D U C A T I O N A L R E S E A R C H I N S T I T U T E ( E - S E R I E S ) In the course of 2009, a new series Znanstvena poročila Pedagoškega inštituta (i.e. Scientific Reports of the Educational Resarch Institute, Ljubljana) has been initiated on the Institute‘s website ((http://193.2.222.157/Default.aspx). In 2009 the editorial committee consisted of Janez Kolenc, Anton Kramberger, Darko Štrajn The Series serves the following goals: 1. The promotion and dissemination of research activities and achievements by PI fac-ulty, students and visiting fellows in the form of final research reports for third parties or in any other forms for a not-fully-completed scientific work, with a fair mention-ing of all the already done occasional collegial peer-reviews (i.e meetings, conferences, symposia etc.), 2. Contribution to academic debates within and outside the PI, insofar as PI researchers and/or external collaborating researchers and/or PhD students take part in such debates. 38 39 Document Outline masa.vidmar@pei.si www.pipsproject.org