1 Helen Donaghue Lyndon TaylorEvaluating a pre-sessional reading test using stimulated recall interviews Helen Donaghue Lyndon Taylor

2 3 example test questions how this informedthe test recall interviews 3 example test questions how this informed test revision teaching .

3 test design long pre-sessional course reading test test plan

4 recall interviews How did students approach the test?What reading skills/strategies did they use? Did test items elicit the cognitive processes they were designed to elicit? Selected questions, covering all LOs.

5 data collection 2 cohorts: May & December 201649 test-takers in total, 23 interviews 2 researchers 20-30 minutes, as soon as possible after the test audio recorded, transcribed

6 analysis Statistics Transcriptions analysed question by question looking at cognitive processes and reading/test strategies Cross reference stats and interviews

7 results reading for main ideas

8 3 test items Q40, 24, 10

9 Q40 item facility: 0. 41 point biserial: 0Q40 item facility: 0.41 point biserial: 0.52 Choose the correct letter, A, B, C or D. 40. The main message of the text is: A. New technology may cause loneliness. B. We should pay more attention to our health. C. Social contact is important for us. D. Women are more sociable than men. Can read carefully to establish purpose or main idea of the text by identifying or recognising main points that are central to the meaning We'll look first of all at the results for question 40. You'll see the item is looking for the main message of one of the texts. At the bottom is the learning outcome this item is designed to test. The importance of social contact (option C, the correct answer) is an overarching theme for the text; the distractors reflect points which are mentioned in the text, but only in isolated sections. The stats at the top are very good. 41% of students answered the question correctly, which shows that it was slightly difficult, but not unacceptably so. (Around 50% is the ideal). This point biserial correlation score shows that the item is discriminating well between higher ability and lower ability test takers. Anything above about 0.3 is considered to show good discrimination is very good. So if the question is discriminating well between the higher ability and the lower ability students, what can the more able students do (or do well) that the less able student's can't do (or can't do well)? Maybe the interview transcripts can tell us.

10 Shafiq: 88% - correct answer given: C40. The main message of the text is: A. New technology may cause loneliness. B. We should pay more attention to our health. C. Social contact is important for us. D. Women are more sociable than men. Question 40 is the main idea, the whole text. One paragraph talks about A. 'Women are more social than men', only D paragraph talks about that. 'Social contact is important for us': this is the main general idea that is used in every paragraph. The first test taker we're going to look at is Shafiq, who scored 88% overall, certainly a higher ability test taker. We're not using real names of course. He chose the correct option, C, and there's a small extract from the interview here. Shafiq showed that he was able to read for ideas that are central to the meaning of the text. He was able to locate information on each of the distractors in the text, but realised that these ideas could only be found in isolated sections and did not reflect the central theme. But he was able to establish that the importance of social contact ran through the whole text.

11 Sultan: 28% - incorrect answer given: B40. The main message of the text is: A. New technology may cause loneliness. B. We should pay more attention to our health. C. Social contact is important for us. D. Women are more sociable than men. I think we should do attention to our health. And they said like cigarettes, a lot of fatty food, they cause the problem. I chose from mind because I read it before and I chose B. This test taker, Sultan, chose incorrect option B. You can read what he said at the bottom. He remembered some information about health and unhealthy lifestyles that he'd read in the text, and used this to make a quick match with one of the headings. There was no attempt to understand the text more globally. Poor reading skills and strategies led him to the wrong answer, which is good in terms of the test item but shows that the student perhaps needs instruction on how to read for main ideas.

12 Q24

13 Q24: Correct heading: People are more aware of physical than social problemsEvidence about the benefits of face-to-face social contact makes it difficult to understand why little attention has been paid to the importance of building social networks. One reason is that the media focuses obsessively on food, money, exercise and drugs. We recognise that cigarettes, salt, fatty food and being overweight can shorten our lives, while antibiotics, physical activity and a balanced diet can prolong them. This knowledge has changed the way most of us eat, work and spend our leisure time. But despite studies that confirm the benefits of social relationships, the number of people who say they feel isolated has almost trebled since the late 1980s, according to population surveys in Europe, the US and Australia.

14 Q24 item facility: 0.35 point biserial: 0.51Can read carefully to establish the main idea of paragraphs by understanding discourse markers, cohesion and organisation, and distinguishing main idea from supporting details.

15 correct answer Shafiq: 88% - Option chosen: People are more aware of physical than social problems * What do you think the questions are asking you to do? What is the main general idea of the paragraph... So when I start doing it I read the paragraph, I have a general idea in mind, then I read all these headings.

16 correct answer Safa: 75% - Option chosen: People are more aware of physical than social problems First I read all the text carefully. I put a heading in my mind so when I moved to the question I already choose the nearest meaning in my mind. *So you had information in your mind for each paragraph and also the Arabic writing in the margin that --? Yes, that's summarising the paragraph. *So when you did the paragraph matching exercise did you need to go back to the text? Sometimes, for example, to ensure I've understood the question carefully.

17

18 Incorrect answer Musa: 40% - Option chosen: The effects of social relationships on health Evidence about the benefits of face-to-face social contact make it difficult to understand why little attention has been paid to the importance of building social networks. One reason is that the media focuses obsessively on food, money, exercise and drugs. We recognise that cigarettes, salt, fatty food and being overweight can shorten our lives, while antibiotics, physical activity and a balanced diet can prolong them. This knowledge has changed the way most of us eat, work and spend our leisure time. But despite studies that confirm the benefits of social relationships, the number of people who say they feel isolated has almost trebled since the late 1980s, according to population surveys in Europe, the US and Australia. I think about the first sentences and I move to the end. 'Confirmed benefit of social relationships, the number of people' and I found the answer. I don't know if it's correct or not.

19 Q 10

20 Qs 10: Can read carefully to establish main idea of paragraphs by understanding discourse markers, cohesion and organisation, and distinguishing main idea from supporting details.

21 Question 10 Which of the following three main ideas are included in this text. Choose three options from the following list. You can write the answers in any order. A. the contagious nature of laughter (66% of students) B. the jokes people laugh at (13%) C. the reasons people laugh (92%) D. the negative aspects of laughter (32%) E. the laughter in shopping malls (29%) F. the development of laughter in children (63%) This is a multiple matching question which aims to have students distinguish main ideas from supporting examples or less important information. The distractors are mentioned in the text but are not main ideas. The stats show us that option C is too easy and is not discriminating between good and bad test takers - it needs to be changed. One puzzle is why so many students chose option A. Let's look at why some of the students chose this option.

22 distractor: The contagious nature of laughterKareem (38%) *I see, okay, so you were choosing between A and F. Because maybe this I didn't understand the meaning of this. *So you chose it. Yes. Bai (55%) Because I don’t know this word the meaning, I think maybe teacher want to examine and I just choose it. These weaker students chose the distractor A because they didn’t understand ‘contagious’. Interestingly, Bai points this out explicitly - because she doesn't understand it she thinks it is being tested.

23 distractor: The contagious nature of laughterJing (80%) Actually I found this whole paragraph … contagious is … so I … this question. *Yeah, so you’ve got laughter is contagious at the end and you can’t control it at the beginning. Yeah. *Do you know the meaning of contagious? Actually I don’t know. *Okay, so why did you choose it if you didn’t know the meaning? I guess this is maybe something you can’t … the whole meaning- *The whole thing, I see. …and the nature means it’s about you can’t change it, it’s from inside, yeah.

24 Questions Which of the following three main ideas are included in this text. Choose three options from the following list. You can write the answers in any order. A. the contagious nature of laughter (66% of students) B. the jokes people laugh at (13%) C. the reasons people laugh (92%) D. the negative aspects of laughter (32%) E. the laughter in shopping malls (29%) F. the development of laughter in children (63%) Distractor E seems to work better. 29% chose it.

25 distractor: the laughter in shopping mallsMin (63%) *And 'Laughter in shopping malls' why did you not choose that? Also because it was just a short idea here. Mai (55%) But laughter in shopping malls, I think for me, I don't know, it's not very important, I don't know, so that's why I didn't choose it. *Why do you think it's not important? Because it's an example, it's an example.

26 distractor: the laughter in shopping mallsKareem (38%) *Shopping Malls. Okay, do you think that's a main idea? But they're mentioned. *They're mentioned but the question is asking you about main ideas. Maybe a misunderstanding.

27 teaching We'll end by telling you what changes we made to our teaching based on students answers to the questions that required ss to read for main ideas.

28 teaching practice: timeLess time on scanning/reading for specific information, more on reading for main ideas Spending more time with the text as a whole before giving out any questions

29 teaching practice: awarenessmatching words doesn’t work not every paragraph has a first topic sentence so the main idea of a paragraph is not always discernible from the first sentence

30 teaching practice: activitiesstudents read paragraphs and decide on their own heading before giving them headings to choose from students highlight texts/build diagrams to show main and supporting ideas pairs of strong/weak students do mini recall interviews together

31 stimulated recall interviews

32 reference Khalifa & Weir. (2009) Examining Reading. Cambridge: Cambridge University Press