By Paul Temple
When education students are taught about the difference between norm-referenced and criterion-referenced assessment, the example often given of criterion referencing is the driving test. The skills you need to demonstrate in order to pass the practical test are closely defined, and an examiner can readily tell whether or not you have mastered them. So you have to do a hill start without the car running backwards, reverse around a corner without hitting the kerb or ending up in the middle of the road, and so on. The driving test could then, in principle, have a 100% or a 0% pass rate. (A non-education example of a norm-referenced examination is the football league: to stay in the Premier League, a team doesn’t have to be objectively brilliant, just fractionally better than that year’s weakest three teams.) But the driving test is also a threshold assessment: the examiner expects the candidate to be able to negotiate the town centre one-way system competently, but not to show that they can take part in a Bond movie car-chase. You have to cross the threshold of competent driving: you don’t have to show that you can go beyond it.
This seems a clear enough distinction: so why do so many academics apparently have difficulty with it? Journal editors frequently have to deal with reviews of articles under consideration which in effect say, “This is not the paper I myself would have written on the topic.” Well, yes, we probably knew that already. A reviewer’s comment recently on a paper of which I was an author said, in summary, “This paper reports on the views of university staffs, but what about the students?”. To which we responded to the editor, “But we said our research was about staff views; reporting on student views would have meant a different research project.” The reviewer in this case was not seeing a pre-determined threshold to be crossed (“Does this paper adequately report on staff views?”), but was asking something like, “Does this paper cover to my satisfaction the whole of this aspect of higher education?”, or at least, “those areas of it which particularly interest me”. In other words, is it up to Bond movie standards?
The examination of doctoral theses can raise this problem in acute form. Most university regulations setting out the criteria for the award of doctorates say things like, “A distinct contribution to the knowledge of the subject”, “A critical assessment of the relevant literature”, and so forth. In other words, the regulations are creating thresholds: the use of the indefinite article is important – “a critical assessment”, not “the (definitive) critical assessment”. So the questions the examiners should ask themselves are, “Is this, or is this not, a distinct contribution to knowledge?”; “Does this give a critical assessment of the literature, or not?”. The relevant questions are not, “Does this thesis provide the last word on the subject?”, or “Does this thesis cover every detail of the literature?” – and certainly not, “Is this the best thesis that I could imagine on the topic?”.
The problem arises, I think, because academics mostly are trained to dissect material placed before them and to study under a conceptual microscope the strengths and weaknesses that emerge. The world being an imperfect place, a competent academic can thus always find a weakness in whatever it is they are examining. (I plead guilty here: my default approach to book reviewing is to ask, “How has the author managed to get this so wrong?”.) But this is an absolutist approach – correct in some situations, but not in most academic assessments, where thresholds have (for better or for worse) been set and what is required is a judgement on whether or not they’ve been – just – crossed. Not the Bond movie thing.
SRHE member Paul Temple, Centre for Higher Education Studies, UCL Institute of Education, University College London.
