VATE NEWSLETTER NO 5, 2002 (INSERT)

Standardised Testing: What space for professional judgement? A VATE Issues Paper

Brenton Doecke, Gail Reynolds, Arlene Roberts

1. Introduction

This paper arises out of discussions in Curriculum Committee in response to an increasing use by secondary schools of standardised literacy tests in the junior school. The VATE office has been inundated with calls from concerned English teachers asking for guidance about how best to respond to this trend. Clearly, teachers have no option but to implement school policy, but they are far from prepared to mindlessly accept the validity of standardised testing. They are also concerned that such policies threaten to undermine the role of their professional judgements when assessing the literacy needs of their students – a paradoxical outcome, given the government’s aim to enhance the status of the teaching profession through the Victorian Institute of Teaching .

The following discussion does not offer a critique of standardised testing, but merely gives an account of Gail Reynolds’ experience in administering the DART and AIM tests at her school. Gail has been teaching English since the late seventies, working in a variety of settings, including primary, secondary and CAE. She is now English Coordinator at a suburban secondary college.

Gail reflects on how the test results compared with her own judgements of the literacy abilities of individual students in her class. We begin with a general account of the way English staff at her school reacted to the tests. We then offer four case studies that draw on Gail’s knowledge of individual students in her Year 7 English class.

2. The school context

Gail’s secondary college decided to administer both the DART and AIM tests to Year 7 students, and this was done at the start of this year. The DART test was given by English teachers to their own students. The AIM test was administered by homeroom teachers and other subject teachers at a set time. However, although English teachers felt more ownership of the DART test than the AIM test, this did not dispel their misgivings about standardised testing. The DART test took a week to administer and almost as long to mark, an extra burden for English teachers who already have a very demanding workload.

Both teachers and students felt some resentment about the time the DART test took to administer and mark. Students kept asking: ‘When are we going to get back to real English? Do we have to do another test?’ By and large, teachers felt that the DART test was simply too time consuming, and that it took them away from their real work as English teachers. Even though some test results confirmed their observations of the literacy abilities of individual students, this was offset by nagging concerns about the value of the test. As English Coordinator at the school, Gail also felt concern about the impact of the testing on beginning teachers – they appeared to put more faith in the test results than in their own judgements.

The four case studies below raise a variety of questions about the validity of standardised testing and the implications that it has for the English teaching profession.

3. Four case studies: an English Coordinator reflects (by Gail Reynolds)

Case study 1: Mark

Mark is a Year 7 student who, if I were to rank my class, is in the top of the middle: very close to the top. Mark was the first student to finish the test. He dashed up to me and said ‘Mrs Reynolds, Mrs Reynolds I’ve finished’, whereupon I urged him to, ‘Go back and check over your answers…you know that’s really quick.’ But he was back two minutes later, saying, ‘I’ve finished, I’ve finished.’

Mark’s score was absolutely disastrous. He had skim read, made snap judgements and hadn’t worked through exactly what was being asked. My judgement of Mark’s language ability and skill level differed from the DART result. I talked to Mark about the test. He spent a long time telling me earnestly that, ‘My Dad says it’s really important that you finish a paper and you have to finish it very fast’. He then went on to volunteer what was wrong with his written paper and most of his analysis was spot on!

From my perspective, the test result did not accurately show Mark’s language ability. The test categorised him as a student who needed support – a judgement that is not borne out by his work in the English classroom.

Case study 2:Kate

Kate’s parents held firm views about her ability as an English student. They came in on parent teacher night and said ‘Mrs Reynolds. Oh, you’ve got Kate and look it’s a shame …she can’t do English you know. We’ve had her tested and she can’t do any of it. And you know she’s going to need a lot of support.’ I was sitting there with my mouth open because as far as I was concerned, she was in the top section of my class. When we play grammar games, she’s fast, she’s quick, she know the answers. She writes beautifully, although she makes the odd spelling mistake. As a hobby, she’s writing a novel which is very much teen pulp fiction, but nonetheless a significant accomplishment. When she shows it to me, she asks, ‘Now if I did this, would it work and be an improvement?’ Her other hobby is reading. She’s always coming up to me to have a talk about the book she’s reading at the moment.

Kate performed badly in the DART test, apparently supporting her parents’ view that she needs support in English. However, in my view it is wrong to categorise Kate as someone who is struggling with English. Such a judgement causes me to mutter darkly to myself about self-fulfilling prophecies.

Case study 3: Don and Mike

This case study might be called: ‘You can’t measure caviar if you’re weighing fish’. Don and Mike finished second and third when we did the DART test. Their tests were neatly done and completed in a fraction of the time allowed. Don and Mike found nothing to challenge them. The upper limit of the test didn’t give them any scope to show what they could do. As an English Faculty, we had actually decided to provide some of the students additional material on the day of the test, because we thought that some of our brighter kids might get bored with the DART exercise. They did not experience very much trouble with the additional tasks, but were challenged by some of the exercises and they indicated that they had enjoyed doing the extra work. However, the results from the additional material were not a part of the DART test and could not be taken into account.

From my perspective, Don and Mike were disadvantaged because the test didn’t allow them to show their real abilities as English students. The test was not a reliable guide for identifying high achievers.

Case study 4: Tom

Tom really struggles in English. He is in one of the literacy support groups that we run at our school. He is a lovely boy. My working knowledge of Tom is that he doesn’t know how to put his sentences together, he doesn’t get basic ideas like paragraphing and things like that; they don’t resonate with him; he doesn’t understand them. Yet his DART test results put him smack bang in the middle range. When I went to talk to him, he was very cheery and he told me, ‘Oh these tests are easy. I’ve done lots and lots of this kind of test. I’ve done them every year I’ve been at school and sometimes I got to do them twice a year.’ And I think he had just learnt to produce the required answers.

My hunch is that Tom has very poor skills in English but he has been trained up to perform in standardised tests like DART.

4. Issues for the Profession

Some of the English teachers who rang VATE over the past few weeks have expressed concern about the cultural bias of standardised testing. The ‘free’ writing task in the AIM test required students to write on a holiday destination in Australia, prompting some students to say to their teacher: ‘Miss, we have never been on a holiday in Australia’ and ‘Miss, my family never go on holidays’. Such anecdotes abound, suggesting that it is timely for someone to write a critique of psychometric claims about instruments like AIM and DART, and the ideology underpinning such tests.

Some psychometricians (‘insiders’ of a certain discourse community with a specialist ‘knowledge’) would probably react to the teachers’ concerns by saying that they simply do not understand the intricacies of measurement. Gail’s worries might be dismissed as ‘teething’ problems: when teachers get used to administering such tests, they’ll learn how to integrate them more smoothly within existing school programs. Yet teachers do understand that language and literacy comprise an extraordinary range of skills which are embedded in a complex network of human relationships and contexts - this is an important dimension of their ‘knowledge’ as English teachers. They are especially mindful of the ways that a focus on testing can narrow curriculum and pedagogy, undermining their efforts to provide students with rich and variegated opportunities to extend their language and to connect with life beyond school. The anecdotes we have been hearing reflect a genuine concern that tests like DART and AIM represent a ‘normalising’ technology (Popkewitz, 1998) that renders school literacy practices increasingly meaningless (mere examples of ‘school’ literacy), privileging those students whose circumstances and upbringing predispose them to engage in such tasks. The increasing use of standardised testing is a strange trend, given the government’s espousal of middle years reform that supposedly addresses youth disenchantment from schooling.

And there is a further issue. Gail’s case studies raise important questions about the place of her professional judgement within the testing regime that appears to be forming around us. Her thumbnail sketches of individual students typify the professional knowledge and skills of any accomplished English teacher. All accomplished English teachers are obliged to constantly exercise subtle judgements about the language and literacy needs of their students, as Gail’s anecdotes show. It is not as though English teachers ever cease to monitor the literacy of their students. Why are accomplished teachers’ judgements apparently being discounted? Why is there a tendency to treat only standardised test results as a true indicator of performance, both of students’ literacy levels and the ‘value adding’ that teachers have been able to achieve? Why is there a growing tendency for both Commonwealth and State governments to allocate funding on the basis of standardised tests which privilege this form of accountability over teacher judgements? How will beginning teachers learn to exercise diverse forms of assessment (forms of assessment suited to a middle years environment) if schools rely simply on standardised testing to show improved student outcomes?

A useful point of reference here is provided by the Standards for Teachers of English Language and Literacy (STELLA) that have recently been published online (see www.STELLA.org.au). Here is how STELLA describes the work of accomplished teachers of English with respect to their responsibilities when assessing and reviewing student learning and planning for future learning:

Accomplished English/Literacy teachers understand the central role of assessment in advancing student learning, improving the effectiveness of teaching practice and contributing to planning for future learning. They recognise that the school community as a whole benefits from constructive and coordinated assessment and reporting practices. They develop regular assessment and reporting programs to provide students, care-givers and school authorities with timely and accurate feedback on student achievement and progress in language and literacy development. They ensure that their feedback to students recognises achievement while clearly indicating directions for improvement. Accomplished English/Literacy teachers make judicious use of a wide range of formal and informal assessments. They ensure that assessment tasks and items are relevant, valid, fair and transparent and relate as closely as possible to real and diverse conditions of use and practice. They meet the requirements of mandated testing programs without compromising their teaching goals or the learning needs of their students They constantly use assessment information to monitor and re-evaluate their short and long-term teaching and learning goals.

This statement reflects the ways in which English teachers have historically been willing to work with other stakeholders in education to achieve worthwhile outcomes for their students. English teachers have always been prepared to use standardised testing in order to assess students’ performances, but they are also conscious of the limitations of such testing, and the need to employ other forms of assessment.

So where do we go from here? We have been discussing a trend, not mandated reform. The signs are worrying, but we should still be confident about committing ourselves to avoiding the ‘testing craze’ (Swope and Miner, 2000) that has compromised English curriculum and pedagogy in places like the United States. Admittedly, recent developments in Victoria have not been encouraging: we need only think of the circumscribed notions of literacy that underpin Early Years or the TER mentality that characterises VCE. Why hasn’t anyone noticed the absence of references to Shirley Brice Heath or Allan Luke in any of the justifications for Early Years offered by Hill and Crevola? Why have crude forms of assessment been allowed to skew the VCE, reducing students to the level of competing to get the right ‘score’ for tertiary entrance? With both reforms, questions about the culturally embedded nature of literacy practices and the need for schools to establish links with the cultural practices in which students engage outside school simply go begging.

However, English teachers in Australia have a rich tradition of curriculum development and pedagogy behind them. This is reflected in STELLA, as well as in publications likeEnglish in Australia andIdiom. It is also reflected in the ongoing practice of English teachers in Victoria. Proponents of standardised testing like AIM and DART should be prepared to argue the need to rely on such measures of performance. They should be prepared to explain how such tests have more weight than the knowledge and experience of English teachers. They should be prepared to explain why the insights that English teachers have always shown when monitoring the literacy abilities of their students should have less validity than their own instruments for sorting students into manageable groups. They should articulate their own vision of language and literacy and explain how their assessment measures will actually enable students to step into the future.

We can then weigh their vision up against our own. That, indeed, is a ‘measurement’ that would be worth doing.

References

Popkewitz, T.S. (1998) Struggling for the Soul: The Politics of Schooling and the Construction of the Teacher. New York: Teachers College Press. Swope, K. and Miner, B. (eds) (2000) Failing Our Kids: Why the testing craze won't fix our schools. Milwaukee, Wisconsin: Rethinking Schools (www.rethinkingschools.org)

___________________

ABOUT THE AUTHORS

Brenton Doecke is a Senior Lecturer, Faculty of Education, Monash University Gail Reynolds is an English Coordinator in a Victorian secondary school Arlene Roberts is Education Officer at VATE