1 Setting minimum standards in the frame of outcomes’ indicators : methods, interest and use. Thierry Rocher Ministry of Education – DEPP France

2 PUBLIC SECONDARY SCHOOLContext LOLF (fundamental law of finances) : a new public budget framework MISSIONS LABOR HEALTH EDUCATION … PROGRAMMES PUBLIC PRIMARY SCHOOL … PUBLIC SECONDARY SCHOOL LEAD ALL THE PUPILS TO THE MASTERY OF BASIC COMPETENCIES AT THE END OF PRIMARY SCHOOL … OBJECTIVES Indicators 2007 2010 prevision observed target %

3 Indicators Proportions of pupils who acquired the basic competencies (in reference to standards of ‘socle commun’) : In french and maths At the end of primary school (grade 5) and at the end of lower secondary school (grade 9) Indicators calculated by : Type of school : ‘priority education 1’, ‘priority education 2’, ‘public except PE’, ‘private’ 30 educational regions (« académies »)  i.e. 120 « strata » These indicators must be comparable from one year to another In order to asses the completion of objectives Ad hoc programme of students’ assessments

4 Constraints Timeline Cost Trends Implications :Before summer for the vote of the public budget at the Parliament Cost No supplementary funds Trends Linking procedures Implications : MCQ for fast and cheap marking (i.e. implies no writing) Samples (but huge if regional indicators) Pre-experimentation of a large pool of items

5 Process Definition of basic competencies, framework Items development Experimentation 2006 Setting the cut-off points Test 2007 = year 0 …

6 Framework Existing standards (2006) : « socle commun de connaissances et de compétences ». Not yet measured Basic competencies in reference to the ‘socle’ and defined by responsible for pedagogical policy : At the end of primary school, at the end of lower secondary school, in french and maths exemple : maths grade 5

7 Test development Working groups (teachers, inspectors, ministry’s representatives) Mathematics (1 sequence): Grade 5 : 221 items, BI design, 13 blocks of 17 items, 13 booklets of 68 items Grade 9 : 169 items, BI design, 13 blocks of 13 items, 13 booklets of 52 items French (2 sequences) : Grade 5 : 2 « parallel » versions (resp. 67 and 75 items), 2 booklets to control the order effect Grade 9 : 2 « corpus » (resp. 77 items and 95 items), 4 booklets with 1 corpus and a half

8 Experimentation Experimentation of test and proceduresSampling in each of the 30 « académies » : In total : grade 5 pupils in about 500 primary schools and grade 9 pupils in about 500 lower secondary schools Engineering : Standardized procedures (sampling, administration, controls, etc.) Important role of local educational and statistical offices Survey well perceived at a school level High participation rates (around 90%) Final samples : about 7000 pupils by grade and subject

9 Analyses and cleaning Preliminary analysis : CleaningFatigue and training effects Behaviour of guessing responses Discrimination Dimensionality Adequacy with standards (“socle commun”) Cleaning Maths grade 5 : 50 on 221 Maths grade 9 : 24 on 169 French grade 5 : 6 on 74 (comprehension) French grade 9 : 29 on 172

10 Setting the cut-off pointsHow to set the cut-off point ? Arbitrary from the data (‘relative’) For example, rate of 15% below Problem : relative, no link with the content Arbitrary from theoretical judgement (‘absolute’) For example, a priori cut-off point Problem : possible gaps with the observed performance Confrontation between experts’ judgements and students’ performance

11 Setting the cut-off pointsThe « miracle method » does not exist Two methods were considered: First : from a global point of view (Hofstee) Second : from a virtual ‘bordeline’ group of students (Angoff) Application : One day meeting with each group Individual work in the morning, confrontation and discussion in the afternoon

12 Setting the cut-off points – 1st method% pupils who have not acquire the basic competencies data curve % maximum - % non mastery judgement zone % minimum - minimum cut-off maximum cut-off observed score cut-off

13 Setting the cut-off points – 1st methodQuestion 1 : « On the basis of your professional experience, according to you, what are the minimum and the maximum percentages of the pupils who have not acquired the basic competencies [at grade 5/9 in french/maths] ?» Question 2 : « What is the score – number of correct answers – below which one may consider that pupils have not acquired the basic competencies ? More precisely, give a minimum « acceptable » score (lenience) and a maximum « acceptable » score (harshness). » Items that a pupil who acquired the basic competencies… …must absolutely correctly answer …should be able to correctly answer …do not have necessarily to correctly answer Item 1 Item 2 … Item N   = minimal N- = maximal   Un score « acceptable » leur a été présenté de la manière suivante : «  un score qui correspond à votre propre perception et qui vous semble légitime du point de vue de ce que l'institution scolaire peut attendre d'un élève en fin d’école. » En fait, les experts ont travaillé à partir des items du test, en suivant cette consigne : « La borne inférieure relève d'un niveau d'exigence minimal. En pratique, il s'agit de déterminer les questions fondamentales qui doivent absolument être réussies. A contrario, la borne supérieure renvoie à l'idée que les élèves devraient réussir toutes les questions posées, si on s'en tenait à une vision rigide des compétences de base. En pratique, pour déterminer cette borne supérieure, il s'agit simplement d'éliminer certaines questions qui ne semblent pas relever stricto sensu des compétences de base. »

14 Setting the cut-off points – 2nd methodFinally, two groups of pupils : Pupils who have acquired basic competencies Pupils who have not acquired basic competencies Reality is not as simple ‘Borderline’ group of pupils Aim : to find the item level of difficulty which is the best adapted to these ‘borderline’ pupils

15 Setting the cut-off points – 2nd methodRanking of the items by their level of difficulties Easy items 1 success 2 . Cursor = item from which the ‘borderline’ pupils have difficulties (i.e. 50% success) failure X Difficult items

16 1st method : maths grade 5

17 1st method : maths grade 9

18 1st method : french grade 5

19 1st method : french grade 9 (1)

20 1st method : french grade 9 (2)

21 2nd method : maths grade 5

22 2nd method : maths grade 9

23 2nd method : french Grade 5 Grade 9 % non mastery % correct 27 63 3359 31 60 Grade 9 (1) % non mastery % correct 19 66 16 68 30 60 (2) % non mastery % correct 21 63 64 32 57 11 72 8 74

24 Convergence After discussion within the working groups, cut-off points were fixed : Maths grade 5 and 9 : around 10% French : grade 5 – 14%, grade 9 – 21% Cut-off items identification Presentation of the results to the responsible of the pedagogical policy (inspectorates) for validation Students’s performance Item difficulty Students who have not acquired the basic competencies Difficult items Easy items Students who have acquired the basic competencies Cut-off point

25 Test 2007 Item selection Test 2007Optimized at the cut-off points with the content constraint Test 2007 National sample (not regional) : about 8000 pupils in grade 5 and 8000 in grade 9 Anchoring on the 2006 data

26 Conclusion Setting the cut-points :Not a pure formal analytical process … … but necessary to legitimate the numbers Process also useful for discussion with responsibles of pedagological policies Compromise between expectations and observations Methodological issues in the measurement of students’ competencies Next : What use of these indicators ?