by Dennis Sherwood
Forty years on…
I don’t remember much about my experiences at work some forty-odd years ago, but one event I recall vividly is the discussion provoked by a case study at a training event. The case was simple, just a few lines:
Sam was working late one evening, and happened to walk past Pat’s office. The door was closed, but Sam could hear Pat being very abusive to Alex. Some ten minutes later, Sam saw Alex sobbing.
What might Sam do?
What should Sam do?
Quite a few in the group said “nothing”, on the grounds that whatever was going on was none of Sam’s business. Maybe Pat had good grounds to be angry with Alex and if the local culture was, let’s say, harsh, what’s the problem? Nor was there any evidence that Alex’s sobbing was connected with Pat – perhaps something else had happened in the intervening time.
Others thought that the least could Sam do was to ask if Alex was OK, and offer some comfort – a suggestion countered by the “it’s a tough world” brigade.
The central theme of the conversation was then all about culture. Suppose the culture was supportive and caring. Pat’s behaviour would be out of order, even if Pat was angry, and even if Alex had done something Pat had regarded as wrong.
So what might – and indeed should – Sam do?
Should Sam should confront Pat? Or inform Pat’s boss?
What if Sam is Pat’s boss? In that case then, yes, Sam should confront Pat: failure to do so would condone bad behaviour, which in this culture, would be a ‘bad thing’.
But if Sam is not Pat’s boss, things are much more tricky. If Sam is subordinate to Pat, confrontation is hardly possible. And informing Pat’s boss could be interpreted as snitching or trouble-making. Another possibility is that Sam and Pat are peers, giving Sam ‘the right’ to confront Pat – but only if peer-to-peer honesty and mutual pressure is ‘allowed’. Which it might not be, for many, even benign, cultures are in reality networks of mutual ‘non-aggression treaties’, in which ‘peers’ are monarchs in their own realms – so Sam might deliberately choose to turn a blind eye to whatever Pat might be doing, for fear of setting a precedent that would allow Pat, or indeed Ali or Chris, to poke their noses into Sam’s own domain.
And if Sam is in a different part of the organisation – or indeed from another organisation altogether – then maybe Sam’s safest action is back where we started. To do nothing. To walk on by.
Sam is a witness to Pat’s bad behaviour. Does the choice to ‘walk on by’ make Sam complicit too, albeit at arm’s length?
I’ve always thought that this case study, and its implications, are powerful – which is probably why I’ve remembered it over so long a time.
The truth about GCSE, AS and A level grades in England
I mention it here because it is relevant to the main theme of this blog – a theme that, if you read it, makes you a witness too. Not, of course, to ‘Pat’s’ bad behaviour, but to another circumstance which, in my opinion, is a great injustice doing harm to many people – an injustice that ‘Pat’ has got away with for many years now, not only because ‘Pat’s peers’ have turned a blind eye – and a deaf ear too – but also because all others who have known about it have chosen to ‘walk on by’.
The injustice of which I speak is the fact that about one GCSE, AS and A level grade in every four, as awarded in England, is wrong, and has been wrong for years. Not only that: in addition, the rules for appeals do not allow these wrong grades to be discovered and corrected. So the wrong grades last for ever, as does the damage they do.
To make that real, in August 2025, some 6.5 million grades were awarded, of which around 1.6 million were wrong, with no appeal. That’s an average of about one wrong grade ‘awarded’ to every candidate in the land.
Perhaps you already knew all that. But if you didn’t, you do now. As a consequence, like Sam in that case study, you are a witness to wrong-doing.
It’s important, of course, that you trust the evidence. The prime source is Ofqual’s November 2018 report, Marking Consistency Metrics – An update, which presents the results of an extensive research project in which very large numbers of GCSE, AS and A level scripts were in essence marked twice – once by an ‘assistant’ examiner (as happens in ‘ordinary’ marking each year), and again by a subject senior examiner, whose academic judgement is the ultimate authority, and whose mark, and hence grade, is deemed ‘definitive’, the arbiter of ‘right’.
Each script therefore had two marks and two grades, enabling those grades to be compared. If they were the same, then the ‘assistant’ examiner’s grade – the grade that is on the candidate’s certificate – corresponds to the senior examiner’s ‘definitive’ grade, and is therefore ‘right’; if the two grades are different, then the assistant examiner’s grade is necessarily ‘non-definitive’, or, in plain English, wrong.
You might have thought that the number of ‘non-definitive’/wrong grades would be small and randomly distributed across subjects. In fact, the key results are shown on page 21 of Ofqual’s report as Figure 12, reproduced here:
Figure 1: Reproduction of Ofqual’s evidence concerning the reliability of school exam grades
To interpret this chart, I refer to this extract from the report’s Executive Summary:
The probability of receiving the ‘definitive’ qualification grade varies by qualification and subject, from 0.96 (a mathematics qualification) to 0.52 (an English language and literature qualification).
This states that 96% of Maths grades (all varieties, at all levels), as awarded, are ‘definitive’/right, as are 52% of those for Combined English Language and Literature (a subject available only at A level). Accordingly, by implication, 4% of Maths grades, and 48% of English Language and Literature grades, are ‘non-definitive’/wrong. Maths grades, as awarded, can therefore be regarded as 96% reliable; English Language and Literature grades as 52% reliable.
Scrutiny of the chart will show that the heavy black line in the upper blue box for Maths maps onto about 0.96 on the horizontal axis; the equivalent line for English Language and Literature maps onto 0.56. The measures of the reliability of the grades for each of the other subjects are designated similarly. Ofqual’s report does not give any further numbers, but Table 1 shows my estimates from Ofqual’s Figure 12:
| Probability of | ||
| ‘Definitive’ grade | ‘Non-definitive’ grade | |
| Maths (all varieties) | 96% | 4% |
| Chemistry | 92% | 8% |
| Physics | 88% | 12% |
| Biology | 85% | 15% |
| Psychology | 78% | 22% |
| Economics | 74% | 26% |
| Religious Studies | 66% | 34% |
| Business Studies | 66% | 34% |
| Geography | 65% | 35% |
| Sociology | 63% | 37% |
| English Language | 61% | 39% |
| English Literature | 58% | 42% |
| History | 56% | 44% |
| Combined English Language and Literature (A level only) | 52% | 48% |
Table 1: My estimates of the reliability of school exam grades, as inferred from measurements of Ofqual’s Figure 12.
Ofqual’s report does not present any corresponding information for each of GCSE, AS or A level separately, nor any analysis by exam board. Also absent is a measure of the all-subject overall average. Given, however, the maximum value of 96%, and the minimum of 52%, the average is likely to be somewhere in the middle, say, in the seventies; in fact, if each subject is weighted by its cohort, the resulting average over the 14 subjects shown is about 74%. Furthermore, if other subjects – such as French, Spanish, Computing, Art… – are taken into consideration, the overall average is most unlikely to be greater than 82% or less than 66%, suggesting that an overall average reliability of 75% for all subjects is a reasonable estimate.
That’s the evidence that, across all subjects and levels, about 75% of grades, as awarded, are ‘definitive’/right and 25% – one in four – are ‘non-definitive’/wrong – evidence that has been in the public domain since 2018. But evidence that has been much disputed by those with vested interests.
Ofqual’s results are readily explained. We all know that different examiners can, legitimately, give the same answer (slightly) different marks. As a result, the script’s total mark might lie on different sides of a grade boundary, depending on who did the marking. Only one grade, however, is ‘definitive’.
Importantly, there are no errors in the marking studied by Ofqual – in fact, Ofqual’s report mentions ‘marking error’ just once, and then in a rather different context. All the grading discrepancies measured in Ofqual’s research are therefore attributable solely to legitimate differences in academic opinion. And since the range of legitimate marks is far narrower in subjects such as Maths and Physics, as compared to English Literature and History, then the probability that an ‘assistant’ examiner’s legitimate mark might result in a ‘non-definitive’ grade will be much higher for, say, History as compared to Physics. Hence the sequence of subjects in Ofqual’s Figure 12.
As regards appeals, in 2016, Ofqual – in full knowledge of the results of this research (see paragraph 28 of this Ofqual Board Paper, dated 18 November 2015) – changed the rules, requiring that a grade can be changed only if a ‘review of marking’ discovers a ‘marking error’. To quote an Ofqual ‘news item’ of 26 May 2016:
Exam boards must tell examiners who review results that they should not change marks unless there is a clear marking error. …It is not fair to allow some students to have a second bite of the cherry by giving them a higher mark on review, when the first mark was perfectly appropriate. This undermines the hard work and professionalism of markers, most of whom are teachers themselves. These changes will mean a level-playing field for all students and help to improve public confidence in the marking system.
This assumes that the legitimate marks given by different examiners are all equally “appropriate”, and identical in every way.
This assumption. however, is false: if one of those marks corresponds to the ‘definitive’ grade, and another to a ‘non-definitive’ grade, they are not identical at all. Furthermore, as already mentioned, there is hardly any mention of marking errors in Ofqual’s November 2018 report. All the grade discrepancies they identified can therefore only be attributable to legitimate differences in academic opinion, and so cannot be discovered and corrected by the rules that have been in place since 2016.
Over to you…
So, back to that case study.
Having read this far, like Sam, you have knowledge of wrong-doing – not Pat tearing a strip off Alex, but Ofqual awarding some 1.5 million wrong grades every year. All with no right of appeal.
What are you going to do?
You’re probably thinking something like, “Nothing”, “It’s not my job”, “It’s not my problem”, “I’m in no position to do anything, even if I wanted to”.
All of which I understand. No, it’s certainly not your job. And it’s not your problem directly, in that it’s not you being awarded the wrong grade. But it might be your problem indirectly – if you are involved with admissions, and if grades play a material role, you may be accepting a student who is not fully qualified (in that the grade on the certificate might be too high), or – perhaps worse – rejecting a student who is (in that the grade on the certificate is too low). Just to make that last point real, about one candidate in every six with a certificate showing AAA for A level Physics, Chemistry and Biology in fact truly merited at least one B. If such a candidate took a place at Med School, for example, not only is that candidate under-qualified, but a place has also been denied to a candidate with a certificate showing AAB but who merited AAA.
And although you, as an individual, are indeed not is a position to do anything about it, you, collectively, surely are.
HE is, by far, the largest and most important user of A levels. And relying on a ‘product’ that is only about 75% reliable. HE, collectively, could put significant pressure on Ofqual to fix this, if only by printing “OFQUAL WARNING: THE GRADES ON THIS CERTIFICATE ARE ONLY RELIABLE, AT BEST, TO ONE GRADE EITHER WAY” on every certificate – not my statement, but one made by Ofqual’s then Chief Regulator, Dame Glenys Stacey, in evidence to the 2 September 2020 hearing of the Education Select Committee, and in essence equivalent to the fact that about one grade in four is wrong. That would ensure that everyone is aware of the fact that any decision, based on a grade as shown on a certificate, is intrinsically unsafe.
But this – or some other solution – can happen only if your institution, along with others, were to act accordingly. And that can happen only if you, and your colleagues, band together to influence your department, your faculty, your institution.
Yes, that is a bother. Yes, you do have other urgent things to do.
If you do nothing, nothing will happen.
But if you take action, you can make a difference.
Don’t just walk on by.
Dennis Sherwood is a management consultant with a particular interest in organisational cultures, creativity and systems thinking. Over the last several years, Dennis has also been an active campaigner for the delivery of reliable GCSE, AS and A level grades. If you enjoyed this, you might also like https://srheblog.com/tag/sherwood/.
