All doctoral students in the UK are expected to navigate periodic, typically annual, progress reviews as part of their studies (QAA, 2020). Depending on the stage, and the individual institutional regulations, these often play a role in determining confirmation of doctoral status and/or continuation of studies. Given that there were just over 100,000 doctoral students registered in the UK in 2021 (HESA, 2022), it could therefore be argued that the progress review is a relatively prominent, and potentially high stakes, example of higher education assessment. However, despite this potential significance, guidance relating to doctoral progress reviews is fairly sparse, institutional processes and terminology reflect considerable variations in approach, empirical research to inform design is extremely limited (Dowle, 2023) and perhaps most importantly, the purpose of these reviews is often unclear or contested.
At the heart of this lack of clarity appears to be a tension surrounding the frequent positioning of progress reviews as primarily institutional tools for managing key performance indicators relating to continuation and completion, as opposed to primarily pedagogical tools for supporting individual students learning (Smith McGloin, 2021). Interestingly however, there is currently very little research regarding effectiveness or practice in relation to either of these aspects. Yet, there is growing evidence to support an argument that this lack of clarity regarding purpose may frequently represent a key limitation in terms of engagement and value (Smith McGloin, 2021, Sillence, 2023; Dowle, 2023). As Bartlett and Eacersall (2019) highlight, the common question is ‘why do I have to do this?’
As a relatively new doctoral supervisor and examiner, with a research interest in doctoral pedagogy, in the context of these tensions, I sought to use a pedagogical lens to explore a small group of doctoral students’ experiences of navigating their progress review. My intention for this blog is to share some learning from this work, with a more detailed recent paper reporting on the study also available here (Clark, 2023).
Methods and Approach
This research took place in one post-1992 UK university, where progress assessment consisted of submission of a written report, followed by an oral examination or review (depending on the stage). These progress assessments are undertaken by academic staff with appropriate expertise, who are independent of the supervision team. This was a small-scale study, involving six doctoral students, who were all studying within the humanities or social sciences. Students were interviewed using a semi-structured narrative ‘event-focused’ (Jackman et al, 2022) approach, to generate a rich narrative relating to their experience of navigating through the progress review as a learning event.
In line with the pedagogical focus, the concept of ‘assessment for learning’ was adopted as a theoretical framework (Wiliam, 2011). Narratives were then analysed using an iterative ‘visit and revisit’ (Srivastava and Hopwood, 2009) approach. This involved initially developing short vignettes to consider students’ individual experiences before moving between the research question, data and theoretical framework to consider key themes and ideas.
Findings
The study identified that the students understood their doctoral progress reviews as having significant potential for supporting their learning and development, but that specific aspects of the process were understood to be particularly important. Three key understandings arose from this: firstly, that the oral ‘dialogic’ component of the assessment was seen as most valuable in developing thinking, secondly, that progress reviews offered the potential to reframe and disrupt existing thinking relating to their studies, and finally, that progress reviews have the potential to play an important role in developing a sense of autonomy, permission and motivation.
In terms of design and practice, the value of the dialogic aspect of the assessment was seen as being in its potential to extend thinking through the assessor, as a methodological and disciplinary ‘expert’, introducing invitational, coaching format, questions to provoke reflection and provide opportunities to justify and explore research decisions. When this approach was taken, students recalled moments where they were able to make ‘breakthroughs’ in their thinking or where they later realised that the discussion was significant in shaping their future research decisions. Alongside this, a respectful and supportive approach was viewed as important in enhancing psychological safety and creating a sense of ownership and permission in relation to their work:
“I think having that almost like mentoring, which is like a mini mentoring or mini coaching session, in these examination spots is just really helpful”
“I’m pootling along and it’s going okay and now this bombshell’s just dropped, but it was helpful because, yeah, absolutely it completely shifted it.”
“It’s my study… as long as I can justify academically and back it up. Why I’ve chosen to do what I’ve done then that’s okay.”
Implications
Clearly this is a small-scale study, with a relatively narrow disciplinary focus, however its value is intended to lie in its potential to provoke consideration of progress reviews as tools for teaching, learning and researcher development, rather than to assert any generalisable understanding for practice.
This consideration may include questions which are relevant for research leaders, supervisors and assessors/examiners, and for doctoral students. Most notably: is there a shared understanding of the purpose of doctoral progress reviews and why we ‘have’ to do it? And how does this purpose inform design, practice and related training within our institutions?
Within this study it was evident that in this context the role of dialogic assessment was significant, and given the additional resource required to protect or introduce such an approach, this may be an aspect which warrants further exploration and investigation to support decision making. In addition, it also framed the perceived value of the careful construction of questions, which invite and encourage reflection and learning, as opposed to seeking solely to ‘test’ this.
Dr Timothy Clark is Director of Research and Enterprise for the School of Education at the University of the West of England, Bristol. His research focuses on aspects of doctoral pedagogy and researcher development.
University discourse, policy, and practice has focused increasingly on access, widening participation and inclusion over the course of the last thirty years (Heath et al, 2013). In particular, understanding access, participation and inclusion for those who align with the different protected characteristics (as defined by the Equality Act 2010) has been of interest to academic research, given various political movements to widen access to higher education. There is a wealth of research in the space of equity, equality, and inclusion which has started to prise open the daily lived experiences of those who hold one or more of the protected characteristics as being part of their identity. Both in the tradition of UK academia, but also from research conducted in the US, we – as a research community – have begun to recognise the institutional and systemic structures which lead to sexism, microaggressions, blatant overt racism, disabilities and health inequalities, issues of access, pastoral burden and caring responsibilities. These facets can lead to extreme workloads, extreme discomfort, bullying and sometimes harassment routinely endured by members of both the academic community and the student body. Of course, research which seeks to make inequalities more transparent has also focused on social class background, which does not feature as one of the nine characteristics outlined by the Equality Act 2010. Here, research has predominantly focused on the experiences of working-class students, academics (and on one occasion, parents) but as yet, in the UK, the remit of who is included here is limited (Crew, 2020; 2021a; 2021b).
There are groups which exist outside the current research narrative which are less considered within the wider body of experiential evidence within the academy (Moreau & Wheeler, 2023; Caldwell, 2022). The ambition to promote access to these voices formed the basis of the rationale for my doctoral thesis research ‘Doing the heavy lifting: the experiences of working-class professional services and administrative staff in Russell Group universities’, completed in 2023. The study featured 13 participants who self-identified as working-class and worked in professional services and administrative roles in UK Russell Group universities. Using a novel approach it combined narrative inquiry (to understand historical personal biography and context) with more traditional semi-structured interviews, to understand the phenomena of existing in contemporary university spaces (Clandinin & Connelly, 2000).
As I discussed both in a presentation at SRHE’s annual conference and within my doctoral thesis research, there are distinct limitations within the academic research body which have isolated the experiences of students and academics with particular protected characteristics, often at the expense of intersectionality or of the representation of other stakeholders who have critical value within university spaces. I addressed the ways in which administrative staff and professional services staff are included within academic research, as a representation of their human capital, roles and responsibilities, the ‘minions of management’ that Dopson & McNay discuss leads to an absence of voice and authority. These accounts focus on the actions performed within the university space rather than the experience these individuals have of that space, and how these experiences reflect the wider institutional culture at play (Caldwell, 2022). This understanding of other people within higher education research as being inextricably connected with role rather than identity and experience is something which was also exemplified by Marie-Pierre Moreau and Lucie Wheeler (2023) in their recent SRHE conference presentation on the current status of academic research literature with ancillary workers in higher education in the UK. Finding little UK-based research, Moreau & Wheeler concluded that the everyday experiences of ancillary workers had thus far, to their knowledge, failed to have been included in the wider narrative about institutional culture and lived experience in UK HEIs.
In a previous blog post for SRHE, Michael Shattock discussed the centralisation of UK higher education away from regional responsibility and governance. Similarly, the degree to which the internal systems of university administration is centralised, or not, has the potential to facilitate or negate healthy working relationships and partnerships, fostered by governance structures. It is particularly pertinent that the brokers of the relationships which are formed from levels of centralisation are the professional services and administrative staff who facilitate the function and process of legislation, administration and research management and the teaching, research, and technical expertise of those working on academic contractual pathways. And yet, like the ancillary workers who provide critical support to the daily function of the university in the most literal form, the experiential perspectives of these huge groups of university employees are left largely outside of the scope of academic research.
Organisational culture literature dictates that culture is predominantly dictated by three elements: assumptions, values and artefacts (Schein, 2004). Where assumptions are a mental model used by managers to make sense of the environment, values are the socially constructed principles that guide behaviour; these are reflected through speech, approaches and spoken goals. Artefacts are the ‘visible and tangible layer’, in the case of the university, the statues and buildings (Harris, 1998; Joseph & Kibera, 2019). In understanding the possibilities for development and promotion, career trajectories, workload, working environments and relationships between people in higher education it might be possible to make some small-scale assumptions about how much these institutions are indeed changing towards becoming more inclusive or how far removing cultural icons of oppressions, such as statues, is a purely performative act.
By collecting first-hand experiential evidence around the assumptions and values of an institution, the nature of organisational culture might be possible to discern (Harris, 1998). I fail fundamentally to understand how research culture initiatives, which, in their broadest sense tackle the measurement and progression of positive research cultures in universities in the UK, can make any progress on the status and environment of our institutions without having legitimate, robust, empirical evidence driving policy and practice. And that empirical evidence needs to include the perspectives, insights, and opinions of everyone who is a direct stakeholder within the organisation. By omitting large swathes of those who directly affect and are directly affected by that organisation we omit the opportunity to make credible, inclusive, necessary progress both in policy, but also in the implementation of practice. The absence of these voices is an academic failure which, in its current form, fails to address the full spectrum of the political economy of UK universities. It is only in doing more work in this area that progress in equalities agendas can fully be realised.
Dr Jess Pilgrim-Brown is a sociologist and researcher in education. She focuses on issues relating to social class, gender and wider social inequalities. Her thesis research ‘Doing the heavy lifting, the experiences of working-class professional services and administrative staff in Russell Group universities’ was the first of its kind in the UK. Her research interests span sociological theory, innovative methods in qualitative research designs and research ethics. She is a current Research Associate at the University of Bristol and Postdoctoral Researcher at the University of Oxford.
We were very proud to have our paper accepted at SRHE’s 2023 Research Conference. This was particularly because we value our collaboration, which was born via SRHE, and our paper was about that journey. This blogpost shares highlights from our SRHE 2023 conference paper which outlined: (i) our collaboration to publication research story; (ii) the importance of SRHE in our collaborative journey; which amongst many positives for nearly a decade, links directly to (iii) our award-winning journal paper that presented a conceptual framing of employability in different global contexts.
Our collaboration to publication research story
This recent piece of work started with a joint seminar at the 2019 SRHE Conference at Celtic Manor. We gathered a group of employability researchers to explore different aspects of interrogating methodologies and approaches on employability from different disciplinary perspectives and country contexts. It was a lively session, inspiring much debate including questions from the audience, and making us realise that it was time to re-evaluate yet again the conceptions of what employability meant. We were encouraged to suggest a follow up session, and by the time we left the room we had agreed to investigate the production of a special issue of Higher Education Quarterly (HEQ).
We discussed the initial steps in our collaboration to publication journey. Helen shared her recollections and how she persuaded Labake, as an early career researcher, that her career would be strongly enhanced by taking the lead in the endeavour. Hence, throughout the Covid-19 lockdown and beyond, Labake pitched our idea, had it accepted by the editors of HEQ, and then worked to make sure that all the contributors met deadlines, and delivered papers of the quality needed for publication.
In July 2021, our special issue entitled ‘Interrogating theoretical and empirical approaches to employability in different global contexts’, appeared online and on paper in HEQ. We were ecstatic and amazed at the success of the publication, particularly when we discovered in 2023 that our introduction paper, Fakunle & Higson (2021) and Labake’s substantial paper (Fakunle, 2021), had been Wiley’s top cited and most downloaded articles respectively (in 2021/22). Additionally, both papers and two other papers from the Special Issue were cited in Tight’s (2023) recent review and synthesis of the debate and continuing discussion around how employability is viewed in relation to the core purpose of higher education. This affirms the intellectual quality and care with which the work had been carried out. We talked about the impact for our career trajectory, and its significance as Helen entered her third decade of researching in employability, with the first 20 years charted in Higson (2023).
The importance of SRHE to this work
This second part of our paper concentrated on the role of SRHE in the triumphs mentioned above. As we chart this contribution, we acknowledge that it is a story that many other HE researchers will recognise. For both of us, the SRHE has played a major and significant part in our research journeys, both individually, but in this context particularly together.
The story begins when Helen was appointed Co-Network Convenor of the SRHE Employability and Enterprise special interest group. This involved running a number of very successful research days in London, and an eventful one held in 2014 in Edinburgh, which saved Scottish colleagues some journey time. The well attended event attracted participants not only from Scotland, but also from Northern Ireland and from England, as far south at Bournemouth. The session was held at the Dovecote Studios, where we first met when Labake, then a PhD student, was working on the employability of international students. This shows the importance of these events, and the vital role of supervisors and network convenors to encourage PhD students to attend these events, which are always collegiate and inspire confidence in early career researchers. This SRHE event at Edinburgh was the starting point for the mentoring relationship and follow-up conversations at subsequent SRHE Conferences. We provide below a brief snapshot/testimonial about the efficacy of SRHE network events.
Helen’s reflections
As network convenor, I was always keen to create a community of practice for employability researchers at the Conference. On the first night, before most of the conference had started, I always hosted a table in the restaurant. This allowed lone researchers, new researchers, and first time conference goers to join our group in an informal setting. This always ensured that there were friendly attenders at conference presentations, and often led to future collaborations. A number of my best collaborators and now friendships (including with Labake), stemmed from those friendly dinners, at which we always ran out of chairs.
Labake’s reflections
I kept in touch with Helen during and after completing my PhD and my academic roles at the University of Edinburgh. I enjoyed attending several Employability events as a valued opportunity to network with colleagues from the UK and abroad and explore collaborative opportunities. This formed the basis for asking Helen to join me in contributing to the Conference seminar in 2019, and our continuing mutually beneficial collaborations. I am especially proud that one of my PhD students was accepted to present her research at the 2023 SRHE conference and is able to avail themselves of the networking opportunities!
Our groundbreaking employability framework
Our presentation culminated with the discussion about the employability framework in our award-winning journal article published here. The framework proposes a conceptualisation of employability in 3 dimensions: outcomes, process and conceptual. The outcomes approach is centred on economic parameters based on individual competence and employment rates. We point to the dominance of the outcomes approach in the multiplicity of definitions and understandings of employability. The process approach captures the role of higher education institutions in providing and assessing employability development opportunities. We differentiated between different conceptual approaches of employability dimensions such as the dominant human capital theory (underpinning the outcomes approach), critical realism, capability and positional conflict. The framework provides conceptual clarity that addresses contesting positionalities and differing positions on what employability is, and the relevance beyond dominant outcomes approach and western-focused context.
Conclusion
We are grateful and mindful that SRHE worked the magic for us, bringing together an experienced researcher and ECR with impactful outcomes. Our story is one of many. Hopefully, our story will inspire other ECRs to reach out and make the best of the opportunities that SRHE provides. We also want to highlight how the input and support from more experienced researchers, and collaboration can make a lot of difference in academic career trajectory!
Dr Omolabake Fakunle is Chancellor’s Fellow, and Director of Equality, Diversity and Inclusion at the Moray House School of Education and Sport, University of Edinburgh. She is Affiliate Faculty, Centre for Higher Education Internationalisation (CHEI), Università Cattolica del Sacro Cuore, Milano. Her award-winning research, teaching and consultancy includes inclusivity in internationalisation, employability, and decoloniality. She was a member of the inaugural Scottish Funding Council’s Tertiary Quality Framework Expert Advisory Group, and current member of the SRHE Governance and Appointments Committee.
Professor Helen Higson OBE DL was Deputy Vice-Chancellor at Aston University. She is now Professor of Higher Education Learning and Management in Aston Business School. Helen is a Principal Fellow of Advance HE and a National Teaching Fellow. Her recent research, policy and consultancy work includes intercultural training, developing employability and skills development, facilitating a coaching
[1] The ITV programme ‘Mr Bates v The Post Office’ was shown on British TV during the first week of January 2024 and has generated in the UK a media firestorm and a swift government response. Those, probably mostly outside the UK, who are unfamiliar with the story might like to read this explainer from Private Eyebefore reading this editorial. Or just Google it.
by Rob Cuthbert
Mr Sherwood, you’re the only one who’s been reporting these problems …
We have complete confidence that our system is robust.
This is a story of injustice on a massive scale, over a long period. The story of someone affronted by the unfairness who refused to give up, even though the authorities lined up to oppose him and try to make him go away. A story which has not yet attracted the attention it seems to deserve, given the way it affects the lives of tens of thousands of people who put their faith in a flawed system.
Every year a new group of tens of thousands of people are subject to the same repeated injustice. Most of them have no idea that they might have been unfairly treated. If they try to use official procedures for complaint and recompense most of them will fail. The authorities’ repeated mantra is that the system is ‘the best and fairest way’.
It could be, but it isn’t. And one person’s attempts to make things better have been met with denial, opposition, obfuscation, and the use of official processes to discourage media attention, by a public agency which is “independent of government”.
The reports showed varying reliability in the grades awarded by examiners, compared with the ‘true’ or ‘definitive’ grade awarded by a senior examiner. Dennis Sherwood, an independent analyst and consultant, interpreted Ofqual’s measurements of grade reliability as a consequence of what he termed ‘fuzziness’. Fuzziness is the range around a senior examiner’s ‘definitive’ mark that contains the ‘legitimate’ marks given by an ordinary examiner. The 2018 report found that grades for, say, English and History are much less reliable than those for Maths and Physics. In Sherwood’s terms, the ‘fuzziness’ of the marks associated with English and History is greater than for Maths and Physics.
Problems arise when a marking range straddles a grade boundary. For example, if a script is legitimately marked in a range from 38-42, but a grade boundary is set at 40, then more than one grade could result from that one script, depending on who marks it and how. Ofqual have admitted that this is the case:
“…more than one grade could well be a legitimate reflection of a student’s performance and they would both be a sound estimate of that student’s ability at that point in time based on the available evidence from the assessment they have undertaken.” (Ofqual, 2019).
The 2016 report says: “… the wider the grade boundary locations, the greater the probability of candidates receiving the definitive grade.” GCSEs have nine grades plus unclassified, and A-levels have six plus unclassified, meaning grade widths are inevitably narrower than, for example, university degree classifications with just four plus fail. With comparatively narrow grade widths more candidates will be close to a boundary. In other words, and however good the marking is, grading for many candidates will not always give a ’true’ or ‘definitive’ grade.
“It’s really important that people don’t put too much weight on any individual grade. … I know, unfortunately, that a lot of weight is placed on particular GCSEs for progression, maths and English being the obvious ones. In maths that is less problematic because the assessment in maths is generally highly reliable. In English that is problematic. This is not a failure of our GCSE system. This is the reality of assessment. It is the same around the world. There is no easy fix, I am afraid. It is how we use the grades that needs to change rather than creating a system of lengthy assessments.” (emphasis added).
Dame Glenys Stacey, Ofqual’s Chief Regulator until 2016, was reappointed as Acting Chief Regulator after the departure of Sally Collier in the aftermath of the 2020 results, and she said in 2020 (House of Commons Education Committee, 2020a: Q1059):
“It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think Michelle mentioned, that we have faith in them, but they are reliable to one grade either way.” (emphasis added)
According to Ofqual’s own research, we have a national system of grading that is only 95% reliable – and then only if you accept that grades are reliable within plus or minus a grade. The problem is that most people use grades more precisely than that. If you don’t get a grade 4 or above in GCSE English or Mathematics, you may be allowed to progress to educational routes post-16, but you must take a resit alongside your next phase of study, and will not be allowed to continue if your resit grade is still 3 or below. If you miss out by just one grade at A-level, your chosen university may reject you. Although marking meets the best international standards, grading still contains much individual unfairness. That means many students may miss out on their preferred university, be forced to wait a year to try again, or decide not to enter higher education at all.
We know this mainly because of the efforts of Dennis Sherwood, who started writing about problems with grading five years ago. Sherwood’s analyses attracted media attention but often his findings were rejected by Ofqual, for example in Camilla Turner’s Daily Telegraph report of 25 August 2018, when an Ofqual spokesman was quoted as saying: ‘Mr Sherwood’s research is “entirely without merit” and has drawn “incorrect conclusions”’ (Turner, 2018).
Ofqual tried to shut down Sherwood’s commentaries, and complained to the Independent Press Standards Organisation (IPSO) about a Sunday Times article headlined ‘Revealed – A-level results are 48% wrong’ published on 11 August 2019. IPSO’s finding upheld the complaint, but only on the narrow grounds that the newspaper had not made it sufficiently clear that the use of the word ‘wrong’ was the newspaper’s, and not Ofqual’s, characterisation of the research. However the IPSO ruling said:
“It was not significantly misleading to report that 48% of grades could be “wrong”, in circumstances where the research indicated that, in 48% of cases, a senior examiner could have awarded a different grade to that awarded by the examiner who had marked the paper. The complainant had accepted that different grades could be awarded as a result of inconsistencies in marking, but disagreed with the characterisation of the research which had been adopted by the publication.”
Sherwood’s argument has never been refuted. Ofqual, with its statutory responsibility to maintain public confidence in qualifications, was trying to ignore or attack stories that ‘one grade in four is wrong’. That tactic might have succeeded, were it not for Covid. The story of the infamous examinations algorithm, ultimately abandoned, need not be repeated here. However it showed, first, that few parents and indeed teachers understood how the grading system worked. Secondly, Ofqual’s defence of the flawed 2020 algorithm was so focused on the collective unfairness of grade inflation between one year and the next that they failed to recognise that their ‘solution’ moved grading from a national competition to an intensely local one. That made individual unfairnesses very visible, there was a public outcry and the algorithm was abandoned. Individual unfairness in grading persists – but has reverted to its former obscurity.
Dennis Sherwood accordingly wrote a book, Missing the Mark,which I reviewed for HEPI, setting out his arguments in detail. It seemed to be persuading more in the educational media to give his arguments the space they deserved. He was no longer entirely alone, with a small group (including me) finding his arguments convincing. Support from various media, notably the HEPI blog, gave him space to make his argument. However, as in the case of Mr Bates and the Post Office, there were still just a few individuals ranged against the forces of Ofqual and (some of) the educational establishment.
On 8 June 2023 I wrote ‘If A-level grades are unreliable, what should admission officers do? for HEPI, arguing that universities should recognise the limited reliability of A-level grades by giving candidates the benefit of the doubt, uplifting all achieved results by one grade. That blog was perhaps provocative but it did at least recognise the problem and suggest a short-term fix. My 2020 explanation about the algorithm had become the most-read HEPI blog ever, and I was invited, as I had been every year since 2020, to contribute a further blog to HEPI, to be published near to A-level results day. My follow-up to the June blog advised students and parents how to respond if they had fallen short of an offer they had accepted. I submitted it to HEPI but it was not accepted. HEPI did however publish a blog by one of its trustees, Mary Curnock Cook, on 14 August, the Monday before results day on Thursday.
Curnock Cook is the widely-respected former head of UCAS. She began:
“In this blog, I want to provide some context and challenge to two erroneous statements that are made about exam grades:
That ‘one in four exam grades is wrong’
That grades are only reliable to ‘within one grade either way’”
She asserted that the statement ‘one in four exam grades is wrong’ was a ‘gross misunderstanding’, but then said:
“In many subjects there will be several marks either side of the definitive mark that are equally legitimate. They reflect the reality that even the most expert and experienced examiners in a subject will not always agree on the precise number of marks that an essay or longer answer is worth. But those different marks are not ‘wrong’.”
In other words, as admitted by Ofqual, more than one grade could be a ‘legitimate’ assessment of the outcome for an individual. Huy Duong, another critic of the 2020 algorithm, had been widely quoted in the media in 2020 after he predicted the exact outcomes of the algorithm a week before the publication of results. He commented on Curnock Cook’s blog:
”… a lot of this is simply playing with words … whichever definitions of ‘wrong’ and ‘rights’ the establishment chooses to use, it is irrefutable that students are subjected to a grade lottery … If, as the author and the establishment contend, for a given script, both “Pass” and “Fail” are equally legitimate, then for the student’s certificate to state only either “Pass” or “Fail”, that certificate is stating a half truth.”
Curnock Cook then addressed the supposedly ‘erroneous’ statement that “grades are only reliable to ‘within one grade either way” – the statement made by Glenys Stacey as Chief Regulator – saying:
“Some commentators have chosen to weaponise this statement in a way that shows poor understanding of the concepts underpinning reliable and valid assessment and risks doing immense damage to students and to public confidence in our exam system.”
How it is that Sherwood’s analysis shows ‘poor understanding’ is not explained. On the contrary, he seems to have a clear understanding of what Ofqual themselves have admitted. Curnock Cook said the claim about reliability had been taken out of context, but the context is not international tests of collective grading reliability, but the way universities and individual students actually use the grades.
Curnock Cook’s blog was welcomed by influential commentators like Jonathan Simons of Public First, a government favourite for research and PR, and some educationists such as Geoff Barton of the Association of School and College Leaders. She said that talking about unreliable grades “risks doing immense damage to students and to public confidence in our exam system”. Indeed it does, but the risk lies not in pointing out that the emperor has no clothes. The real risk is in not changing the system which remains unfair to so many individuals. The emperor still has no clothes, and it is time to redress things.
Most people who suffer injustice in grading do not even know it has happened. For individuals who do know, most will find that using official procedures to complain or appeal is expensive, and unlikely to change the outcome. In his campaign to illuminate the problem Mr Sherwood, like Mr Bates, met denial, opposition and the use of official processes to discourage the media from continuing to cover the story. People in the organisations concerned know how the system actually works, but they don’t want it to be widely known, for the sake of public confidence in the system. Groupthink puts collective inter-cohort ‘fairness’ ahead of fairness to every individual in every cohort. There was even, in 2020, blind faith in a computer system which was later proved to be faulty.
Public confidence in the qualifications and examinations system is of course absolutely vital. But the need for public confidence does not mean that individual unfairness on a large scale should be tolerated and ignored. There are several possible solutions to the problems of grading unreliability, and many would have little direct cost. HE institutions would have to take even greater care in using grades, as part of their wider assessment of the potential and abilities of candidates for their courses. That is a small price to pay for maintaining public confidence in a national system which everyone could be proud of for its fairness as well as its international standing.
Rob Cuthbert is Emeritus Professor of Higher Education Management, University of the West of England and Joint Managing Partner, Practical Academics rob.cuthbert@btinternet.com. Twitter @RobCuthbert
What has this got to do with an academic conference on higher education?
The power of media, narrative story and the broader humanities have the capacity to convey stories through genres such as drama and comedy in compelling and accessible ways. My own work is concerned with discourse and narratives on the idea and purpose of a university and its role in society. I contributed to two presentations at SRHE 2023 which both involved an analysis of narratives – the first being political party manifestoes from 1945 to 2019 and the second an analysis of Knowledge Exchange Framework policy. Both of these presentations and my wider interests look at discourse and narratives as data in higher education policy and practice.
The telling of the compelling Post Office scandal story in an accessible format has reached millions of screens, sparking conversation in workplaces and around dinner tables. This surge in public feeling has kicked off further investigations into the miscarriage of justice which involves a complex network of state and private actors over many years. This shows how narratives can reach many diverse audiences to begin to unravel the personal stories as well as the complexities involved. The SRHE conference theme for 2023 itself looked to unpick connections and complexity between Higher Education Research, Practice, and Policy.
Connected research, policy and practice was a key theme in both keynotes, the first online from Professor Nicola Dandridge and the second kicked off the in person 3 day event in Birmingham at Aston University – a panel discussion and plenary on re-shaping Tertiary Education with Professors Huw Morris, Ellen Hazelkorn, Chris Millward, and Andy Westwood, chaired by Professor Sir Peter Scott.
The complexity in making connections across research, policy and practice was clear as the speakers challenged researchers of higher education to come up with answers to the sector’s issues and challenges as well as re-shaping the sector into one which is tertiary rather than just higher. Browsing the conference programme at the sessions to come showed hugely diverse topics and methods used in higher education research. It certainly is complex to respond to the challenge of research providing the answers or even more challenging the answer.
The growing direction of travel towards tertiary is thankfully not a singular path. Like other potential futures, the panel showed a plurality of potential paths, all bound up with a plurality of perspectives, values and ambitions as well as the key aspect of funding. The panel on tertiary education came up with at least three perspectives on our tertiary futures, from conservative through to radically progressive.
Narrative might be something that can help to make sense of some of this complexity in both analysis but also in making a change at policy and practice levels.
The first of my own two presentations at the conference looked at political discourse of higher education in UK elections from 1945 to 2019. Debbie McVitty and I looked at the political narratives and discourses of Conservative, Labour and Liberal Democrat party political manifestoes to track how higher education was written about and in what context. Broadly, the Labour Party used ‘higher education’ more than the other two parties but all three had similar frequency when writing about the sector when it came to the word ‘university’. We observed spikes in frequency of ‘higher education’ and ‘university’ in 1966, 1987, 2001 and 2010. The first three elections were incoming second and third term governments which might hold some clues for 2024 in the UK. The context in which manifestoes talk about higher education has changed and broadened over the 74 year period. In 1945 and for the majority of the remainder of the 20th century, higher education and universities were mentioned in the context of education, health, science and innovation and youth. Progressively following the turn of the millennium in line with growth in student numbers, political parties began broadening the scope and influence of universities. We saw themes linked to universities in the context of lifelong learning, the economy, immigration, the European Union, public services, apprenticeships and equality. In short, as universities have grown in size and number, politics has looked to them do and achieve more for society and adds to the complex role of higher education in society. As we look ahead to 2024 and the biggest election year the world has ever seen it will interesting to see how universities are positioned politically in the UK and all over the world.
More positive narratives could be found in my second presentation with Vanessa Cui from Birmingham City University. We looked at the narratives of the Knowledge Exchange Framework (KEF) – a regulatory policy exercise from UKRI. Universities are required, in a similar way to teaching and research excellence frameworks to submit narrative statements alongside quantitative measures. We looked at these statements to see how universities told the story of knowledge exchange (often described as third mission) outside the more structured activities of teaching and research. We found a wide range of activity carried out by universities which contributed to both the local economy as well as public and community engagement. Characters in these narratives included students and graduates, university staff, local authorities and public services, publics, businesses and other education institutions. Activities ranged from collaborating with local people on research projects and providing learning opportunities to responding to and contributing to large scale events such as the Commonwealth Games and City of Culture organisation. Moreover, universities clearly played an important role during the Covid-19 pandemic, not just in developing vaccines but providing services and support in collaboration with many different organisations and communities.
For both of these projects we are using the Narrative Policy Framework (NPF) as a broad methodological framing for policy narratives and responses, assuming:
A constructing of social reality
A bounded relativity (beliefs, norms, ideas, strategies, context)
Narratives have generalizable structural elements
Policy narratives operate at three levels (macro, meso, micro)
Narratives play a central role in communicating information
In previous work I have analysed similar regulatory narrative responses using computational text analysis (corpus-assisted discourse analysis) as a method of analysing corpora running into the millions of words. This we combined with the NPF and plan to develop this methodological integration in further work.
Objective and positivist measures are a big part of much of the English regulatory landscape, TEF takes data from the national student survey and continuation, completion and progression indicators to evidence student experience and student outcomes. The REF, KEF and TEF ask for narrative statements alongside the numbers as evidence and to ultimately provide outcomes. Vanessa and I concluded with regard to KEF that universities have a narrative challenge in crafting texts which tell the story of the idea and purpose of their institutions to regulators but also then to students and publics.
Narratives play a key role in human communication. This echoes the importance of narrative and story outlined above and the impact of drama and stories to public consciousness. Narratives and storytelling also play a key role in marketing, from BT selling the gift of family communication to the addictive quality of R Whites Lemonade. The marketisation of a higher education in the neoliberal era has been widely researched and theorised. But in responding to the call from the keynotes and others working with the sector at SRHE 2023, to make the case for higher education and universities, maybe we need to adopt some of the narratives used in the big neoliberal marketised machine. Again, how does the university, tell its story and purpose to a wide range of stakeholders?
Researchers in higher education are analysing and crafting narratives in diverse and creative ways. Charlie Davis presented his work on academics of working-class heritage creating narratives through stories and comics. Social science fiction narratives can allow us to explore ideas and different conceptualisations and visions of the future. These approaches are drawing upon research data, literature and theories but in new and futures-orientated and playful ways. Justyna Bandola-Gill presented her study on narrative CVs – a relatively new approach to research funding whereby researchers craft their own story rather than a list of achievements. And Josh Patel got into the detail of the Robbins report pulling out the ambitious and verging on poetic narrative from the neoliberal economist Lionel Robbins’ vision of expanded public university education – Josh urged us all to go and read a very accessible and hopeful narrative from 1960s higher education policy.
Narratives are not going away. In the latest 2023 publication of TEF statements, institutions could submit up to 25 pages as part of their provider submission (up 10 pages from the previous round) and new to the latest set of statements are panel decision narratives and (optional) student submissions. In December 2023 this provided half a million words each from panels and students and 1.8 million words from providers. A by-product of such an exercise is a unique corpus of texts which provide an insight into how a range of institutions are responding to policy in describing their own practice in diverse ways. This provides a huge amount of learning for the sector.
Narratives play a central role in communicating information and constructing reality. From a research perspective we can analyse these texts as policy stories and wider discourse on what is constructed as a social reality. Narratives involve characters, context, morals of a story and plot lines. Rhetoric is the ancient art of persuasion. Aristotle broke this down into ethos (speaker’s status, character, credibility and authority), pathos (appealing to emotions, values and beliefs of the reader) and logos (logic, reasoning and argument).
As well as using these tools for analysis, universities and higher education researchers can use them to create narratives which surface the purpose and ideals of education to politicians, policy-makers, funders and publics. We may need them, as hostilities towards the university grow.
Maybe higher education needs to tell its stories and narratives to the wider world in equally accessible and creative ways.
Adam is a Senior Research Fellow in education systems and policy at the University of Birmingham. Adam’s work looks at universities as part of tertiary education systems and the role that they play as key sites of knowledge production and dissemination in wider society. This includes how technologies and media have and are shaping, knowledge production and access.
Whilst much has been said about the broadening education gap caused by covid-19 (Hayes, 2021; Crossfield et al, 2023; De Witte and Francois, 2023), fears have also been raised about its impact on the next-step plans of those from widening participation (WP) backgrounds (Nelsonet al, 2021; Co-op, 2021; Kingsley, 2021), including progression to university. A Sutton Trust (Montacute, 2020: 13) report observed how ‘recessions’, such as that triggered by the pandemic, ‘are known to have considerable impacts on educational aspirations and opportunities, with’, it is noted, ‘young people from disadvantaged backgrounds more likely to have their educational decisions influenced by labour market conditions’ than their more privileged peers (Raven, 2023: 101). However, these commentators and researchers were reporting in the midst of the pandemic when much was uncertain. In seeking to provide a more recent set of perspectives, I was commissioned by a Uni Connect partnership (of local universities, colleges and schools) to gather the insights of 14 teaching professionals based in schools and colleges located in the English West Midlands. The timing of this research coincided with the easing of lockdown restrictions (UK Parliament, 2021). All the institutions involved (which totalled 15, since one participant worked across two schools) had significant numbers of students from disadvantaged neighbourhoods. Whilst the semi-structured interviews with these teaching professions highlighted a range of challenges faced by their students, they also drew attention to the kinds of support that had the potential to overcome them. The results that this blog post reports and reflects upon are based on a recently published journal article (Raven, 2023).
Challenges
The central argument made in this article was that ‘the challenges wrought by the pandemic accentuated the engrained disadvantages those from WP backgrounds have long had to contend with.’ My interviewees talked about the impact covid had on their students’ mental health. Our ‘year 10[s] [aged 14-15] and upwards,’ one teaching professional observed, are ‘used to hearing from the media that they are so far behind and have missed so much. That is [affecting] plans for where they go’ next. Whilst the impact was evident amongst students who had struggled before the pandemic, those with ‘no previous mental health issues’ were also being affected, including in terms of diminished levels of confidence and resilience. ‘They are much quicker to give up if it is hard work’, one interviewee observed (Raven, 2023: 107).
The teaching professionals also discussed students who were now less certain about the possibility – and suitability – of HE. In part, this was because they had concerns over components of the curriculum that had not been covered. However, it also reflected the fact that lockdowns and restrictions on school visits, as well as who could visit their school and college, meant that they had ‘missed out on’ a range of outreach interventions (Raven, 2023: 101). In addition, the pandemic had reinforced students’ concerns about moving away from home and leaving the communities they were familiar with and felt secure in. It had also acerbated long-held anxieties over debt and the costs (and benefits) of a higher education, in part because of the challenging labour market conditions generated by the pandemic, including in witnessing parents who had been furloughed or lost their jobs (Raven, 2023).
Response
In seeking to address these challenges, interviewees identified the need to ensure ‘students were informed of their options.’ A process, it was argued, that should start early in their secondary school education (years 8 or 9, when they would be 12 to 13 years old), and place university in the context of the wider learner journey, including the post-16 transitions to college or sixth form. Suggested interventions included workshops aimed at exploring what HE is and what it could offer, along with sessions designed to equip these young people with the skills that would enable them to research ‘the range of [university] courses available.’ There was also a need to provide these young people, and their parents/guardians, with ‘more information about student finance,’ as well as the ‘placement opportunities’ offered by many HE courses (Raven, 2023: 109-111).
In addition, the teaching professionals highlighted the importance of providing young people with a more detailed understanding of higher apprenticeships. One interviewee argued that whilst their students may ‘know you can work and get paid for it,’ they have very limited understanding of ‘the ins and outs of a degree apprenticeship,’ and that these are ‘attached to a university. [More] needs to be done’, it was added, including providing opportunities for these young people to speak with current higher level apprentices who could tell them about their ‘real life experiences’ (Raven, 2023: 111).
Whilst the provision of information, advice and guidance (IAG) was important, interviewees emphasised the need for outreach. Campus visits could dispel ‘commonly held perceptions that HE will be classroom-based’, it was argued. More broadly, providing first-hand accounts of HE could be especially important ‘because a lot of our kids don’t have parents who have been to university. So, doing those [university] trips has a big impact on the [them], and [is] something that we have not been able to do recently’ (Raven, 2023: 112).
The teaching professionals also highlighted the part that undergraduates could play as role models. In particular, those who came from ‘relatable demographic and social backgrounds.’ Various ideas for how university students could be deployed were described. These include a ‘university pen pals’ scheme, that ‘would enable those involved to ask the students about their university, their course, or anything [they] want to know really.’ Mentoring opportunities were also discussed. These could enable students from under-represented backgrounds to ask their mentors about ‘their journeys’, including ‘what the start of university [was] like, [and] how they cope[d] financially’ (Raven, 2023: 113).
Conclusion
The findings from this study suggest that the pandemic has reinforced existing patterns of disadvantage. Moreover, they provide a corrective to views that IAG and outreach support can be offered (and prove effective) as one-off activities. Instead, they underpin the imperative for a ‘sustained and progressive programme’ of interventions that not only support the development of students but counter the detrimental impact of external events (Raven, 2023: 16). However, there remains a need for more research. Not least, in hearing from the students themselves and, ideally, by adopting a longitudinal approach that captures change in views and post-18 intentions over time. It is hoped that a new research project will afford such insights.
Acknowledgements
Thank you to the teaching professionals who took part in this study for their time, insights and expertise. Thank you also to members of the Higher Horizons+ Steering Group for supporting this study and especially to Ant Sutcliffe, the Network’s Head, Dr Hannah Merry, the Operations Manager, and Katie Coombe-Boxall, the Data and Research Manager, for all their guidance and patience.
Neil Raven is an educational consultant and researcher in widening access. Contact him at neil.d.raven@gmail.com.
Some of the reported benefits of using learning analytics data include enabling personalised learning and narrowing attainment gaps. Indeed, a quick dip into some of the recent TEF feedback summaries to higher education institutions seems to suggest that use of learning analytics is valued by TEF panels. But can we learn more from the data to influence teaching practice? Aside from the potential benefits for a more personalised learning experience, we think that it’s a good way of understanding the learning process more generally. Over the past few years, we’ve been analysing some of the data generated from Aston University.
Last minute cramming is not effective in improving attainment.
Yes, your parents were correct – it’s much better to work consistently! Early engagement with studies really appears to matter. In fact, the average attainment levels of those first-year students whose engagement remained at the lowest relative levels throughout the year was very similar to those whose early engagement was lowest in the first three weeks but became the very highest in the last three weeks. In contrast, those who started off enthusiastically, but then lost interest, were awarded higher average marks than any of the groups that started off slowly, regardless of how much or whether their engagement peaked later. The consistency of the data – in that those who started off with high engagement tended to finish with high engagement – was remarkable. Also noteworthy were the effects of early engagement on attainment. For the chart below, we divided students into activity quintiles based on only their first three weeks of engagement (Q5 being the highest engagement) and on end of year mark quintiles (Q5 being the highest attainment). The width of the lines connecting engagement quintile to mark quintile is indicative of the proportion of students linking the two measures. The results highlight how few students pass from higher activity quintiles to lower mark quintiles and vice versa.
Of course, these results come with the usual caveats that we cannot infer cause and effect (it could be that the lower engagers in the first three weeks were just low achieving students). However, for us, this highlights the importance of a good induction into academic life – possibly enhanced by some structured engagement exercises to help get first years into good habits (ie tell them how they should be engaging, and the different ways that they can, not just that they should be doing so). There were probably a fair few students represented in this figure that were not even sure what they were supposed to be doing with all their ‘spare’ time
Behaviour outweighs demographics when predicting attainment
The recent pandemic generated much discussion about digital poverty, suggesting that who we teach might be important – at the very least in terms of access to technology. Our recent evidence suggests that both how you teach and who you teach mattered. However, it is important to note that behaviour outweighed demographics in predicting attainment, albeit that in this case behaviour was probably also influenced by demographics. The gap between disadvantaged students’ attainment and their peers widened during online teaching and assessment conditions, and disadvantaged students were also less likely to obtain all 120 module credits on their first try. We also observed changes in their patterns of engagement, although less so for synchronously delivered teaching (as compared to recorded lectures). Students with the lowest engagement were the ones driving the widened gap; those who engaged well with synchronously provided teaching (even if online) fared much better.
So, we should stop teaching online and get people into the classroom early?
No – not necessarily. We don’t want to claim that all online teaching is bad – instead we need to understand what forms of online teaching work, what good looks like, and how our various teaching strategies affect different groups. Anecdotally, many students have appreciated the flexibility of online teaching, particularly where it has included facilities such as the ability to ask questions anonymously. And if you want to reuse those pre-recorded videos, there has been some interesting research from other research groups on ‘watch-parties’. With the cost-of-living crisis, many students will appreciate being able to log into a lecture from home rather than forking out a bus fare or missing out on some part time work. What is important is to understand what works – and for whom.
Professor Liz Moores is Deputy Dean in the College of Health and Life Sciences at Aston University and has research interests in the evaluation of higher education, particularly as applied to widening participation issues.
Dr Rob Summers is research manager at the Centre for Transforming Access and Student Outcomes (TASO). Before joining TASO, Rob worked in the student outreach team at Aston University managing a randomised controlled trial of two post-16 outreach programmes as part of the TASO MIOM (Multi-intervention, Outreach and Mentoring) project.
