SRHE Blog

The Society for Research into Higher Education


4 Comments

Rethinking metrics, rethinking narratives: why widening access at elite universities requires more than procedural fairness

by Kate Ayres

For many years, the fair access agenda in UK HE has emphasised more transparent and consistent admissions processes that are underpinned by clearer criteria and targeted support. As a qualified accountant and training in Lean Six Sigma, I’ve always been drawn to efficiency, clarity, and measurable improvement – principles that shaped much of my work in HE. However, as I moved into more senior roles and worked more closely with institutional decision-makers, I started to ask a different kind of question: why do some reforms, even when implemented well, seem to make little real difference?

That question sits at the centre of my doctoral research. Despite significant reforms the social composition of Durham University’s student body has felt largely unchanged. From within the institution, it was evident that fairer offer-making was not translating into meaningful shifts in the home-student entrant profile. This revealed an uncomfortable truth: so far, no amount of investment or policy reform can, by itself, reshape the social forces that determine who sees a Durham degree as desirable.

To understand why, we need to stop looking only at what universities do, and start looking at how students behave, and how the wider customer base, or audience, signals who belongs where.

Why aren’t internal reforms enough?

The limited shift in Durham’s home-student body prompts a key question: are our current metrics assuming universities can control demand, when in fact they can only affect the choices of applicants already in their pool?

My research used fourteen years of UCAS admissions data for Durham University to analyse how applicant characteristics, predicted attainment, school type, and socio-economic background intersect with admissions decisions and outcomes. Using multivariate logistic regression and Difference-in-Differences (DiD) analysis, I examined the impact of Durham’s 2019 move from decentralised to centralised admissions.

Results

Since the centralisation of admissions in 2019:

  • Contextual students are now 72% more likely to receive an offer, reflecting a major shift in offer-making behaviour.
  • Contextual applicants to selecting departments remain 40% less likely to get offers than those applying to recruiting ones.
  • No improvement is seen in firm-acceptance rates, suggesting culture or fit still shape applicant choices.
  • Insurance-acceptance has risen 21%, showing Durham is increasingly seen as a backup option for these students.
  • Contextual students are now 2% less likely to enrol after receiving an offer, raising concerns about deeper barriers to entry.

Trend Analysis

The findings were initially encouraging with Contextual applicants became more likely to receive an offer after centralisation. However, the increased offer rate had very limited effect on who actually enrolled. Contextual applicants were increasingly likely to accept alternative universities before Durham. Meanwhile, the proportion of entrants from higher parental SES groups increased, and independent-school students (already overrepresented) continued to make up around one-third of Durham’s home undergraduate intake in 2023.

Who is in control of demand?

While Durham has a history of taking affirmative action for contextual students, these findings illustrate that the OfS-set POLAR4 ratios will never be achievable for somewhere like Durham because these measures assume that universities themselves control demand. Drawing on Organisational Ecology, I argue that this assumption is flawed.

To understand why improved offer-making did not shift entrant composition, we need to look beyond institutional behaviour and examine the ecosystem dynamics that shape demand. Just as ecosystems rely on diversity, so does HE. No institution can appeal to every audience, nor should it. Organisations operate within ecosystems shaped by social, economic, and political forces, and crucially by their audiences, who ultimately determine demand. Therefore, it is the audience that defines an organisation’s niche. In HE, applicants gravitate toward universities that align with their social tastes, expectations, and sense of belonging. Therefore, the most powerful forces shaping demand are the social networks and information transmissions within and these influence applicants long before they apply: what they hear at school, family expectations, and what peers believe “people like us” do—and where “people like us” go.

Currently, wider systemic shifts are reinforcing and entrenching Durham’s niche, especially among white independent-school applicants:

  1.  As Oxbridge intensifies its widening participation initiatives, applicants who traditionally succeed (predominantly white students from independent schools) are increasingly less likely to secure offers.
  2. These applicants seek the closest alternative to the Oxbridge experience, with Durham emerging as a preferred option.
  3. Durham is increasingly accepted as a firm choice because of its perceived “fit” with these applicants’ identity and expectations (as seen in this research).
  4. These applicants typically achieve their predicted grades, making entry more likely.
  5. Their growing presence reinforces existing social narratives about Durham’s student profile.
  6. Consequently, the entrant composition remains socially narrow, and these dynamics may intensify.
  7. The narrative of Durham as a socially exclusive institution persists.
  8. Applicants from non-traditional backgrounds thus perceive a lack of belonging.
  9. As a result, these applicants are less likely to select Durham as their firm choice.

While these dynamics may prompt questions about whether Durham could or should shift away from its position as an “almost-Oxbridge” institution, the evidence suggests that only limited movement is structurally possible. Organisational Ecology predicts that Durham’s niche will remain relatively stable over time and there are many benefits of sticking with a niche approach. The university may be able to broaden its appeal slightly at the margins, drawing in more students from POLAR4 Q3 and Q4 backgrounds, but POLAR4 Q1 and Q2 students are likely to remain outliers. The real question is therefore not whether Durham can radically transform its appeal, but whether it can create the conditions in which those who do apply feel they can belong and thrive. This is where the OfS should take action because, rather than holding universities accountable for applicant pools (which they do not control), it should focus on the areas where institutional agency is strongest. Improving the lived experience of contextual students, strengthening narrative and cultural inclusion, and raising offer-to-acceptance conversion rates are all within Durham’s sphere of influence. Current patterns, particularly the relatively low acceptance and entry rates among contextual applicants, suggest that cultural barriers remain. Regulators should therefore attend less to the composition of the total entrant pool and more to how effectively institutions support, retain, and attract those who already see themselves as potential members of the community.

Taken together, the wider systemic effects detailed above reinforce, rather than shift, Durham’s niche. Only a proportion of applicants will ever feel an affinity with the institution, which is entirely natural in a diverse HE ecosystem where students gravitate toward environments that resonate with their identities and expectations.

These systemic forces lie largely outside Durham’s control, and changing the feedback loop requires more than procedural reform. It demands narrative change within the social networks where ideas of belonging are first formed, and a commitment to ensuring that the lived experiences of contextual students at Durham are positive and affirming. Building stronger partnerships with schools can help shift these early perceptions, while amplifying the stories and experiences of students from diverse backgrounds can offer powerful, alternative points of identification. Applicants make decisions based not just on information, but on a deep, intuitive sense of whether a place feels like it’s for “people like us”. This cannot be achieved through admissions policy, strategy, or marketing alone. Institutions can also look to examples such as the University of Bristol, which has reshaped its entrant pool through doing exactly this. Their efforts have influenced not only who feels able to apply, but who can genuinely imagine themselves thriving within the institution, resulting in a gradual shift in their niche.

Proposal for new metrics

If we evaluate universities on metrics that assume they control demand, we will misread both the problem and the solution. In the short term, universities cannot determine who chooses to apply, but they can influence who feels confident enough to accept an offer, which may, as seen with Bristol, create gradual shifts in the entrant pool over time. Universities can and should work to broaden their niches, yet Organisational Ecology reminds us that institutions rarely move far from their point of peak appeal, meaning Durham’s niche is likely to remain relatively stable and only widen at the margins. Expecting rapid transformation would be like assuming a population adapted to the Arctic could swiftly relocate to the Caribbean. That’s not saying it’s not possible, but it is not fast. Any substantial change in who feels an affinity with Durham will likewise unfold slowly, as cultural experiences and social narratives evolve. In the meantime, improving the lived experience of contextual students, and seeing this reflected in rising conversion rates, is the most realistic and meaningful early sign of movement within the niche. This stability also means that proportion-based performance measures will continue to make the University appear as though it is underperforming, even when it is behaving exactly as expected within its ecological position. Durham has added complexities in that it will always occupy a relatively small share of the HE market because the physical constraints of Durham City limit expansion. This adds presents further broadening of the niche simply because they can’t change by admitting more students.

Therefore, metrics focused solely on broad institutional demand will never fully capture the dynamics of access or institutional “progress”. However, rising conversions – from offer to firm acceptance or offer to entry – among contextual students would signal a growing sense of fit, belonging, or affinity. And even if these students never form a majority, improving conversion is a meaningful and realistic way to measure widening participation progress, because it focuses on what an institution can actually influence, the student experience.

To take these social forces seriously, and to acknowledge that a healthy HE system depends on a diversity of institutions meeting the diverse needs of students, we need metrics that reflect audience attraction and demand dynamics. Current proportion-based measures, fail to capture these realities. Instead, I propose:

  • Because Russell Group institutions occupy a similar position in the Blau Space (they attract applicants with comparable social, cultural, and educational characteristics), organisational ecology theory suggests they compete in neighbouring overlapping niches. This means that isolated widening participation initiatives at a single institution may simply redistribute socially advantaged applicants across the group rather than increase diversity overall. Coordinated widening participation strategies across the Russell Group would therefore reduce competitive displacement and support genuine, sector-wide broadening of access.
  • Introduce regulatory metrics that reward successful conversion, for example offer-to-firm-acceptance rates for underrepresented groups, rather than focusing solely on offers or entrant proportions. This would bring cultural belonging into WP evaluation by capturing the fact that where these students accept an offer and enter, there is likely be a greater sense of affinity, a place where they feel they can “fit”, belong, and succeed.
  • Measure and report the impact of cross-institution outreach among universities with similar audience profiles, recognising that widening participation is driven by sector-level dynamics rather than isolated institutional efforts.
  • Track behavioural demand patterns (such as firm-choice decisions) across groups of institutions to reveal how social signalling influences applicant preferences.

The future of access lies in changing what we measure—and what we tell ourselves

Universities often feel they are held solely accountable for widening access, yet my research demonstrates that applicant perceptions, social networks, and systemic hierarchies play an equally powerful role. The most important conclusion of this research is that access outcomes are co-produced. Universities are not solely responsible for entrant composition; applicants are active agents whose perceptions and choices shape institutional realities. To make meaningful change, we need approaches that reflect this distributed responsibility. To make real progress, we must rethink both the metrics we prioritise and the narratives we reproduce.

Fair admissions processes matter – but without addressing the social dynamics shaping applicant behaviour, procedural fairness alone will never deliver equitable outcomes. By shifting the sector’s focus to behavioural metrics and narrative change, we can begin to challenge the feedback loops that sustain exclusivity and move toward a system where access is genuinely a collaborative effort.

Durham University may never appeal to more than a small share of the applicant pool, but perhaps the real measure of success is ensuring that those who do not fit the perceived mould feel confident enough to accept and enter. Ecosystems flourish through diversity, and so does HE; no single institution can – or should – meet every need. Our responsibility is to keep access fair, to reshape the narratives that limit choice, and to support those who want to join us to feel that they truly belong. In focusing on this conversion (from offer to entrant) we move toward a more honest and sustainable understanding of what widening participation success looks like. We cannot control the applicant pool, but we can influence the student experience, the narratives that spread through their networks, and their confidence in imagining themselves belonging here.

Dr Kate Ayres is a Chartered Management Accountant (CIMA) with a DBA from Durham University, where her research explored market niches and widening participation in UK HE through organisational ecology using quantitative methods. She has worked across finance, academic, and project management roles in UK Higher Education, including positions at Durham University and the University of Oxford. Kate currently serves as an Academic Mentor on the Senior Leaders Apprenticeship at Durham University Business School. Her work brings together analytical insight, organisational experience, and a commitment to improving HE culture. She also co-manages and sings with the Durham University Staff Chamber Choir, which she founded.

Image of Rob Cuthbert


6 Comments

Mr Sherwood v The Office of Qualifications and Examinations Regulation[1]


[1] The ITV programme ‘Mr Bates v The Post Office’ was shown on British TV during the first week of January 2024 and has generated in the UK a media firestorm and a swift government response. Those, probably mostly outside the UK, who are unfamiliar with the story might like to read this explainer from Private Eyebefore reading this editorial. Or just Google it.

by Rob Cuthbert

Mr Sherwood, you’re the only one who’s been reporting these problems …

We have complete confidence that our system is robust.

This is a story of injustice on a massive scale, over a long period. The story of someone affronted by the unfairness who refused to give up, even though the authorities lined up to oppose him and try to make him go away. A story which has not yet attracted the attention it seems to deserve, given the way it affects the lives of tens of thousands of people who put their faith in a flawed system.

Every year a new group of tens of thousands of people are subject to the same repeated injustice. Most of them have no idea that they might have been unfairly treated. If they try to use official procedures for complaint and recompense most of them will fail. The authorities’ repeated mantra is that the system is ‘the best and fairest way’.

It could be, but it isn’t. And one person’s attempts to make things better have been met with denial, opposition, obfuscation, and the use of official processes to discourage media attention, by a public agency which is “independent of government”.

The Office of Qualifications and Examinations Regulation (Ofqual) is charged with regulating and maintaining standards and confidence in GCSEs, A levels, AS levels, and vocational and technical qualifications. Ten years ago Ofqual were aware of some potential problems in grading. To determine the extent of the problem, they took entire cohorts of GCSE, AS and A Level scripts and re-marked them, comparing the marks given by an ordinary examiner to comparable re-marks given by a senior examiner. Eventually this led to two careful and scholarly reports: Marking Consistency Metrics in 2016 and Marking Consistency Metrics – An Update  in 2018.

The reports showed varying reliability in the grades awarded by examiners, compared with the ‘true’ or ‘definitive’ grade awarded by a senior examiner. Dennis Sherwood, an independent analyst and consultant, interpreted Ofqual’s measurements of grade reliability as a consequence of what he termed ‘fuzziness’. Fuzziness is the range around a senior examiner’s ‘definitive’ mark that contains the ‘legitimate’ marks given by an ordinary examiner. The 2018 report found that grades for, say, English and History are much less reliable than those for Maths and Physics. In Sherwood’s terms, the ‘fuzziness’ of the marks associated with English and History is greater than for Maths and Physics.

Problems arise when a marking range straddles a grade boundary. For example, if a script is legitimately marked in a range from 38-42, but a grade boundary is set at 40, then more than one grade could result from that one script, depending on who marks it and how. Ofqual have admitted that this is the case:

“…more than one grade could well be a legitimate reflection of a student’s performance and they would both be a sound estimate of that student’s ability at that point in time based on the available evidence from the assessment they have undertaken.” (Ofqual, 2019).

The 2016 report says: “… the wider the grade boundary locations, the greater the probability of candidates receiving the definitive grade.” GCSEs have nine grades plus unclassified, and A-levels have six plus unclassified, meaning grade widths are inevitably narrower than, for example, university degree classifications with just four plus fail. With comparatively narrow grade widths more candidates will be close to a boundary. In other words, and however good the marking is, grading for many candidates will not always give a ’true’ or ‘definitive’ grade.

This situation is admitted by Ofqual and has been known for more than five years, since the 2018 Report. Dr Michelle Meadows, formerly Ofqual’s Executive Director for Strategy, Risk and Research, said in evidence to the House of Lords Education for 11-16 year olds Committee (2023) on 30 March 2023:

It’s really important that people don’t put too much weight on any individual grade. … I know, unfortunately, that a lot of weight is placed on particular GCSEs for progression, maths and English being the obvious ones. In maths that is less problematic because the assessment in maths is generally highly reliable. In English that is problematic. This is not a failure of our GCSE system. This is the reality of assessment. It is the same around the world. There is no easy fix, I am afraid. It is how we use the grades that needs to change rather than creating a system of lengthy assessments.” (emphasis added).

Dame Glenys Stacey, Ofqual’s Chief Regulator until 2016, was reappointed as Acting Chief Regulator after the departure of Sally Collier in the aftermath of the 2020 results, and she said in 2020 (House of Commons Education Committee, 2020a: Q1059):

“It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think Michelle mentioned, that we have faith in them, but they are reliable to one grade either way.”  (emphasis added)

According to Ofqual’s own research, we have a national system of grading that is only 95% reliable – and then only if you accept that grades are reliable within plus or minus a grade. The problem is that most people use grades more precisely than that. If you don’t get a grade 4 or above in GCSE English or Mathematics, you may be allowed to progress to educational routes post-16, but you must take a resit alongside your next phase of study, and will not be allowed to continue if your resit grade is still 3 or below. If you miss out by just one grade at A-level, your chosen  university may reject you. Although marking meets the best international standards, grading still contains much individual unfairness. That means many students may miss out on their preferred university, be forced to wait a year to try again, or decide not to enter higher education at all.

We know this mainly because of the efforts of Dennis Sherwood, who started writing about problems with grading five years ago. Sherwood’s analyses attracted media attention but often his findings were rejected by Ofqual, for example in Camilla Turner’s Daily Telegraph report of 25 August 2018, when an Ofqual spokesman was quoted as saying: ‘Mr Sherwood’s research is “entirely without merit” and has drawn “incorrect conclusions’ (Turner, 2018).

Ofqual tried to shut down Sherwood’s commentaries, and complained to the Independent Press Standards Organisation (IPSO) about a Sunday Times article headlined ‘Revealed – A-level results are 48% wrong’ published on 11 August 2019. IPSO’s finding upheld the complaint, but only on the narrow grounds that the newspaper had not made it sufficiently clear that the use of the word ‘wrong’ was the newspaper’s, and not Ofqual’s, characterisation of the research. However the IPSO ruling said:

“It was not significantly misleading to report that 48% of grades could be “wrong”, in circumstances where the research indicated that, in 48% of cases, a senior examiner could have awarded a different grade to that awarded by the examiner who had marked the paper. The complainant had accepted that different grades could be awarded as a result of inconsistencies in marking, but disagreed with the characterisation of the research which had been adopted by the publication.”

Sherwood’s argument has never been refuted. Ofqual, with its statutory responsibility to maintain public confidence in qualifications, was trying to ignore or attack stories that ‘one grade in four is wrong’. That tactic might have succeeded, were it not for Covid. The story of the infamous examinations algorithm, ultimately abandoned, need not be repeated here. However it showed, first, that few parents and indeed teachers understood how the grading system worked. Secondly, Ofqual’s defence of the flawed 2020 algorithm was so focused on the collective unfairness of grade inflation between one year and the next that they failed to recognise that their ‘solution’ moved grading from a national competition to an intensely local one. That made individual unfairnesses very visible, there was a public outcry and the algorithm was abandoned. Individual unfairness in grading persists – but has reverted to its former obscurity.

Dennis Sherwood accordingly wrote a book, Missing the Mark, which I reviewed for HEPI, setting out his arguments in detail. It seemed to be persuading more in the educational media to give his arguments the space they deserved. He was no longer entirely alone, with a small group (including me) finding his arguments convincing. Support from various media, notably the HEPI blog, gave him space to make his argument. However, as in the case of Mr Bates and the Post Office, there were still just a few individuals ranged against the forces of Ofqual and (some of) the educational establishment.

On 8 June 2023 I wrote ‘If A-level grades are unreliable, what should admission officers do? for HEPI, arguing that universities should recognise the limited reliability of A-level grades by giving candidates the benefit of the doubt, uplifting all achieved results by one grade. That blog was perhaps provocative but it did at least recognise the problem and suggest a short-term fix. My 2020 explanation about the algorithm had become the most-read HEPI blog ever, and I was invited, as I had been every year since 2020, to contribute a further blog to HEPI, to be published near to A-level results day. My follow-up to the June blog advised students and parents how to respond if they had fallen short of an offer they had accepted. I submitted it to HEPI but it was not accepted. HEPI did however publish a blog by one of its trustees, Mary Curnock Cook, on 14 August, the Monday before results day on Thursday.

Curnock Cook is the widely-respected former head of UCAS. She began:

In this blog, I want to provide some context and challenge to two erroneous statements that are made about exam grades:

  • That ‘one in four exam grades is wrong’
  • That grades are only reliable to ‘within one grade either way’

She asserted that the statement ‘one in four exam grades is wrong’ was a ‘gross misunderstanding’, but then said:

“In many subjects there will be several marks either side of the definitive mark that are equally legitimate. They reflect the reality that even the most expert and experienced examiners in a subject will not always agree on the precise number of marks that an essay or longer answer is worth. But those different marks are not ‘wrong’.”

In other words, as admitted by Ofqual, more than one grade could be a ‘legitimate’ assessment of the outcome for an individual. Huy Duong, another critic of the 2020 algorithm, had been widely quoted in the media in 2020 after he predicted the exact outcomes of the algorithm a week before the publication of results. He commented on Curnock Cook’s blog:

”… a lot of this is simply playing with words … whichever definitions of ‘wrong’ and ‘rights’ the establishment chooses to use, it is irrefutable that students are subjected to a grade lottery … If, as the author and the establishment contend, for a given script, both “Pass” and “Fail” are equally legitimate, then for the student’s certificate to state only either “Pass” or “Fail”, that certificate is stating a half truth.”

Curnock Cook then addressed the supposedly ‘erroneous’ statement that “grades are only reliable to ‘within one grade either way” – the statement made by Glenys Stacey as Chief Regulator – saying:

“Some commentators have chosen to weaponise this statement in a way that shows poor understanding of the concepts underpinning reliable and valid assessment and risks doing immense damage to students and to public confidence in our exam system.” 

How it is that Sherwood’s analysis shows ‘poor understanding’ is not explained. On the contrary, he seems to have a clear understanding of what Ofqual themselves have admitted. Curnock Cook said the claim about reliability had been taken out of context, but the context is not international tests of collective grading reliability, but the way universities and individual students actually use the grades.

Curnock Cook’s blog was welcomed by influential commentators like Jonathan Simons of Public First, a government favourite for research and PR, and some educationists such as Geoff Barton of the Association of School and College Leaders. She said that talking about unreliable grades “risks doing immense damage to students and to public confidence in our exam system”. Indeed it does, but the risk lies not in pointing out that the emperor has no clothes. The real risk is in not changing the system which remains unfair to so many individuals. The emperor still has no clothes, and it is time to redress things.

Most people who suffer injustice in grading do not even know it has happened. For individuals who do know, most will find that using official procedures to complain or appeal is expensive, and unlikely to change the outcome. In his campaign to illuminate the problem Mr Sherwood, like Mr Bates, met denial, opposition and the use of official processes to discourage the media from continuing to cover the story. People in the organisations concerned know how the system actually works, but they don’t want it to be widely known, for the sake of public confidence in the system. Groupthink puts collective inter-cohort ‘fairness’ ahead of fairness to every individual in every cohort. There was even, in 2020, blind faith in a computer system which was later proved to be faulty.

Public confidence in the qualifications and examinations system is of course absolutely vital. But the need for public confidence does not mean that individual unfairness on a large scale should be tolerated and ignored. There are several possible solutions to the problems of grading unreliability, and many would have little direct cost. HE institutions would have to take even greater care in using grades, as part of their wider assessment of the potential and abilities of candidates for their courses. That is a small price to pay for maintaining public confidence in a national system which everyone could be proud of for its fairness as well as its international standing.

This editorial draws on my article first published in The Oxford Magazine No 458, ‘Maintaining public confidence in an unfair system – the case of school examination grades’, and uses some parts of the text with permission.

Rob Cuthbert is Emeritus Professor of Higher Education Management, University of the West of England and Joint Managing Partner, Practical Academics rob.cuthbert@btinternet.com. Twitter @RobCuthbert

Ian Mc Nay


Leave a comment

Lies, damned lies and spin; never mind the statistics

By Ian McNay

Press reports, 31 January, on UCAS statistics on the 15 January deadline showed remarkable unanimity around telling, shall we say…not the whole truth:

–         Girls lead the way as degree applications hit record levels – Times

–         Record numbers of 18-year-olds apply to university – Telegraph

–         University applications hit record high – Guardian

The Telegraph had a second story claiming the number of applicants aged 20 and over had increased by 5%.

All this gave comfort to [English] ministers who claim that high fees have had no long term effect on applications. So, let us look at the longer term and compare the cycle for 2014 entry Continue reading